APT Speed For Incremental Updates Gets a Massive Performance Boost

← Back to Stories (view on slashdot.org)

APT Speed For Incremental Updates Gets a Massive Performance Boost

Posted by timothy on Saturday December 26, 2015 @07:56AM from the just-the-bit-I-missed dept.

jones_supa writes: Developer Julian Andres Klode has this week made some improvements to significantly increase the speed of incremental updates with Debian GNU/Linux's APT update system. His optimizations have yielded the apt-get program to suddenly yield 10x performance when compared to the old code. These improvements also make APT with PDiff now faster than the default, non-incremental behavior. Beyond the improvements that landed this week, Julian is still exploring other areas for improving APT update performance. More details via his blog post.

162 comments

Min score:

Reason:

Sort:

not too late by Anonymous Coward · 2015-12-26 08:06 · Score: 1

The speed got a performance boost?
It's not too late to change the title.
1. Re:not too late by Anonymous Coward · 2015-12-26 08:58 · Score: 0
  
  Well it better, given how it's yielding twice.
Re:Debain is not Free by Anonymous Coward · 2015-12-26 08:08 · Score: 0

The future is imaginary. It cannot exist in the present.
Julian, if you are reading this by Anonymous Coward · 2015-12-26 08:09 · Score: 0

Thank you! /Happy Debian user
Re: Debain is not Free by Anonymous Coward · 2015-12-26 08:13 · Score: 0

Debian _is_ free but ubuntu isn't:
https://mjg59.dreamwidth.org/37113.html
In fact, this is yet another reason to use the gpl, or at least the lgpl.
Ubuntu by Anonymous Coward · 2015-12-26 08:18 · Score: 0

Does anyone know if this will hit Ubuntu before the next LTS (16.04) is released in April? Would love to have this, but being this late I assume it might not be possible to get it into Ubuntu in time. It's not optimal to have to wait two years for a new feature when a new release is coming up.
1. Re:Ubuntu by jaklode · 2015-12-27 00:06 · Score: 1
  
  Yes, of course it wil, it is currently in xenial proposed waiting for the autopkgtest to run.
Now only... by Anonymous Coward · 2015-12-26 08:18 · Score: 0, Offtopic

If they'd get rid of systemd I'd be a happy camper.
1. Re:Now only... by Anonymous Coward · 2015-12-26 08:30 · Score: 0
  
  Toodles.
2. Re:Now only... by kthreadd · 2015-12-26 08:32 · Score: 2
  
  I'm missing something here. With all the flap about systemd, why the rush of all the distros to adopt it?
  There has hardly been a "rush" to speak of. Fedora was first and switched in 2011, that's almost five years ago and we still have distros that haven't switched yet. This is one of the absolute slowest tech migration ever in the Linux community.
  
  I'm on Mint, but even that is slated to go to systemd at the next major release. Binary logs, etc.? No thanks.
  If you by "binary logs" mean regular text logs with a bit of metadata attached so that you can actually find stuff then yes, that's a good thing.
3. Re:Now only... by Anonymous Coward · 2015-12-26 08:33 · Score: 0
  
  > Binary logs, etc.? No thanks.
  A simple configuration chance will allow you to output to syslog again, not to mention you can always do journalctl | grep[awk|cut] or even perl sorcery.
  People say it's the guys who deal with init.d scripts that are driving the push forward.
4. Re: Now only... by Anonymous Coward · 2015-12-26 08:50 · Score: 0
  
  Dropping stderr and most syslog messages makes it very hard to troubleshoot. That's the problem most people have with it.
5. Re:Now only... by gmack · 2015-12-26 08:51 · Score: 4, Interesting
  
  Mainly because that flap involved very few of the actual technical people and people got upset long after the decision was already made. It also seems to be the case that some trolls went off on a misinformation campaign complete with fake bug reports. Quite frankly, I was terrified of systemd from what I was reading here, but then I actually read up on the subject and realized most of what people have been saying about it is false.
  Now that I've actually tried it, my desktops boot faster and I have had a much easier time customizing the boot sequence of some of the servers I maintain.
6. Re: Now only... by Anonymous Coward · 2015-12-26 08:53 · Score: 0
  
  Those are out of date concepts. systemd was correct in finally killing them off.
7. Re:Now only... by Anonymous Coward · 2015-12-26 08:56 · Score: 0
  
  > Binary logs, etc.? No thanks.
  A simple configuration chance will allow you to output to syslog again, not to mention you can always do journalctl | grep[awk|cut] or even perl sorcery.
  People say it's the guys who deal with init.d scripts that are driving the push forward.
  Nah, its people that need adult supervision so they don't eat their own poopoo.
8. Re:Now only... by Anonymous Coward · 2015-12-26 08:56 · Score: 0
  
  Thank you for your unrequested advertisement, dear systemd/RedHat employee.
9. Re:Now only... by Anonymous Coward · 2015-12-26 08:58 · Score: 0
  
  I've been mostly indifferent to systemd, beyond the "registry==bad" memories. But thanks to systemd insisting to rfkill the wifi on my beaglebone...
10. Re:Now only... by ArchieBunker · 2015-12-26 08:58 · Score: 1
  
  These logs need a completely different set of tools to manipulate the text.
  
  --
  Only the State obtains its revenue by coercion. - Murray Rothbard
11. Re:Now only... by Anonymous Coward · 2015-12-26 09:00 · Score: 0
  
  No one says they play well with less. You use journalctl to read them, or any other program that can read the well-defined on-disk format. I don't know if such programs exist, but there is no reason why less couldn't be modified to support them.
12. Re:Now only... by Anonymous Coward · 2015-12-26 09:01 · Score: 0
  
  Nice damage control there, fanboy.
  >Now that I've actually tried it, my desktops boot faster and I have had a much easier time customizing the boot sequence of some of the servers I maintain.
  Thing is, all of this was possible before systemd. It brought literally nothing besides breakage, infiltration, and bloat. What do the systemd people have to say to this? Dismissal of insserv (i.e. the sysvinit dependency-ordered parallel boot setup) as a "hack" with no further qualification.
13. Re: Now only... by Anonymous Coward · 2015-12-26 09:03 · Score: 0
  
  Found the bearded old UNIX guy.
14. Re:Now only... by kthreadd · 2015-12-26 09:10 · Score: 3, Interesting
  
  Of course. It's not just plain text appended to a set of files so you would have to use journalctl to access them. Nothing wrong with that. Now this gives you a number of benefits. Since journald actually knows about the data that it stores it can make intelligent decisions. Everyone who has ever had to clean up a filled up /var/log partition knows how broken logrotate is. What if the journal could actually know about this and rotate based on size? It also makes it possible to actually search for things without having to take different date and log formats into account, and actually know that it came from the correct program. I know, crazy.
15. Re:Now only... by wjcofkc · 2015-12-26 09:14 · Score: 4, Insightful
  
  What's wrong with binary logs?
  
  Text is a terrible format for efficient storage of and access to structured data
  Access to binary logs is O(1) instead of O(n)
  journalctl outputs a pixel-perfect copy of what /var/log/messages was
  You can query more effectively and precisely than with awk, sed and grep
  You can still use awk, sed and grep if you want
  You can run syslogd in parallel and have your text file as well
  The binary format is well documented
  Traditional logs are binary as well as soon as they are rotated and compressed
  
  For fucks sake already, can we not have a single Linux related discussion that has nothing to do with systemd without it spiraling into a systemd flame fest? Systemd is not the devil. All I read here from detractors are people who are regurgitating bullshit they overheard while riding the bandwagon they blindly jumped on without actually having a single clue what they are talking about. Talk about the blind leading the blind. Meanwhile anyone with a clue who tries to chime in with a voice of reason is simply drowned out. Does using the word binary in sentence where you also refer to logs make you feel like some kind of super hacker? Sometimes I really think that's what all this never ending bullshit is about.
  
  --
  Brought to you by Carl's Junior.
16. Re: Now only... by Anonymous Coward · 2015-12-26 09:31 · Score: 0
  
  If you do things the right way the you don't have to troubleshoot.
17. Re:Now only... by Anonymous Coward · 2015-12-26 09:39 · Score: 0
  
  Yes, because it is really a good ideia to create more deviations from the standard unix just for the sake of it.
18. Re:Now only... by Anonymous Coward · 2015-12-26 09:42 · Score: 0
  
  It's perfectly OK for those people to not like systemd. That's fine. Not liking systemd is not the problem. The problem is not that those people just don't like it. It's that they hate it and make it their mission in life to harass the systemd developers and the people that support them. It's perfectly fine to not like systemd or the way it was designed, but that's where it ends. If you have strong feeling against systemd and your distro decides to adopt it then that's their decision, you either deal with it or you walk the other way.
19. Re: Now only... by Anonymous Coward · 2015-12-26 09:48 · Score: 0
  
  If I wanted another Windows with "new" concepts, I would have chosen microsoft.
20. Re:Now only... by Anonymous Coward · 2015-12-26 09:51 · Score: 0
  
  Many of us are running farm of thousands of servers doing real work and do not give a fuck about using experimental software to shave a few milliseconds from the boot time.
21. Re:Now only... by Anonymous Coward · 2015-12-26 10:12 · Score: 0
  
  We've been down that road for many years now. Buckle up and enjoy the ride.
22. Re:Now only... by Anonymous Coward · 2015-12-26 10:24 · Score: 1
  
  What's wrong with binary logs?
  Nothing, if they're well-designed and ACID compliant, which journald is not:
  
  The only way to deal with journal corruptions, currently, is to ignore them: when a corruption is detected, journald will rename the file to .journal~, and journalctl will try to do its best reading it. Actually fixing journal corruptions is a hard job, and it seems unlikely that it will be implemented in the near future. [...]
  Lennart Poettering 2014-06-25 09:51:01 UTC
  Yupp, journal corruptions result in rotation, and when reading we try to make the best of it. they are nothing we really need to fix hence.
  * https://bugs.freedesktop.org/show_bug.cgi?id=64116
  I think if they had used SQLite or OpenLDAP's LMDB, then it would be less of a big deal. From Howard Chu of OpenLDAP:
  * https://www.linkedin.com/pulse/20140924071300-170035-why-you-cannot-trust-lennart-poettering-systemd
23. Re:Now only... by Anonymous Coward · 2015-12-26 11:52 · Score: 0
  
  I have no idea why people insist on boot times. You are aware that a machine spends little time booting and a hell of a lot more running things? Except for Windows. Windows might actually be better with a boost from systemd.
24. Re:Now only... by Anonymous Coward · 2015-12-26 12:38 · Score: 1
  
  "Everyone who has ever had to clean up a filled up /var/log partition knows how broken logrotate is."
  Eh?
  I've been using happily using logrotate for more than a decade on all of my Linux systems. Logrotate is dead-simple and reliable. (And it handles logs /anywhere/, not just in /var/log.)
  I've seen systemd proponents use fabricated slander against logrotate to try make journalctl look good. Thing is, experienced sysadmins see through the slander. :)
  "It also makes it possible to actually search for things without having to take different date and log formats into account, and actually know that it came from the correct program."
  A properly configured syslogd handles this too. I've never used a Linux system on which syslogd was not properly configured, out of the box.
  It's a pity that bogus revisionist arguments carry any weight with the tech crowd, but I guess that is why old hands get frustrated by the newcomers to the field.
25. Re:Now only... by Anonymous Coward · 2015-12-26 13:26 · Score: 0
  
  Yes, because for a logging system having log entries and index consistent is *clearly* more important than writing out log entries...
26. Re:Now only... by Kjella · 2015-12-26 13:51 · Score: 3, Interesting
  
  What's wrong with binary logs?
  Nothing, if they're well-designed and ACID compliant, which journald is not:
  It's rare that I choose to defend systemd, but ACID compliance does not mean you never have to deal with a corrupt database. It is a software technique to make sure transactions complete in an atomic, consistent, isolated and durable way but it still presumes a "perfect" system and if a bit flip happens in memory or on disk outside the programming logic then ACID will fail. That is why you have ECC, RAID1/5/6 and ZFS, but even they fail and sometimes you have genuinely "impossible" results like you've added $2+$2 to the account and the result is $5 (bit flip from 0x0100 to 0x0101). If you're using plain text and UTF-8 this can happen there as well, there are combinations that are simply illegal to use. You expect the parser to ignore the "impossible" and carry on, apparently that's what journald is doing too.
  
  --
  Live today, because you never know what tomorrow brings
27. Re: Now only... by Anonymous Coward · 2015-12-26 13:53 · Score: 0
  
  Fuck you. The problem is that my debian system is hosed thanks to sysremd. It biots for all of one second, then prints an error. Then I reboot, use grub to select the non-default sysvinit option, and the system boots properly.
  TLDR: systemd propaganda, meet user experience.
28. Re: Now only... by Anonymous Coward · 2015-12-26 14:39 · Score: 0
  
  None of that is dropped. Why do you continue to spread this misinformation?
29. Re: Now only... by Anonymous Coward · 2015-12-26 15:13 · Score: 0
  
  Good luck debugging why your system crashed because it's a "feature" that JournelD can be corrupted by a crash. Then you lose your entire log file, not even partial data.
30. Re:Now only... by ArchieBunker · 2015-12-26 15:27 · Score: 1
  
  What are you even talking about "rotate based on size"?
  
  --
  Only the State obtains its revenue by coercion. - Murray Rothbard
31. Re: Now only... by Anonymous Coward · 2015-12-26 15:29 · Score: 0
  
  Fuck you. What was the error?
32. Re: Now only... by Anonymous Coward · 2015-12-26 15:30 · Score: 0
  
  Yes, but when you run millions of machines to let customers run thousands of interacting containers your needs change.
  Turns out people with customers and money are now driving Linux development.
33. Re: Now only... by Anonymous Coward · 2015-12-26 16:00 · Score: 0
  
  I see that you have never had /var/log fill up before.
34. Re:Now only... by nadaou · 2015-12-26 16:39 · Score: 1, Interesting
  
  [I apologize to the masses for responding to yet another offtopic systemd propagandafest thread.]
  But could I grab any old 32 or 64 bit ubuntu live install usb stick from the last 5 years to access an unbootable Debian system? (from there intstalling raid admin tools is a one line apt-get)
  Or before I can view the logs do I need to prepare and maintain a custom rescue boot disc for every debian version, with the same 32/64 bitness, same lib6c and other .so library versions, and same exact version of systemd? (some of the workstations track Debian/testing)
  All before I can set up some export pipes before I can run grep, sort, uniq, and sed to isolate the problem? Do I need to master a new SQL variant? (Honest question)
  Or will my life be made more difficult under critical server-is-down this is costing us 10s of thousands of dollars a minute pressure conditions due to artificial barriers which didn't used to be there?
  K.I.S.S.
  
  --
  ~.~
  I'm a peripheral visionary.
35. Re:Now only... by Anonymous Coward · 2015-12-26 17:06 · Score: 0
  
  That's fine. You'll still benefit from the real advantages related to administration/management and process control features.
  The bit where it's easy to configure, developed and tested by the biggest enterprise Linux vendor, instead of a toy distro like Ubuntu, should fill you with confidence. Even Debian's picked it up.
36. Re:Now only... by Anonymous Coward · 2015-12-26 18:06 · Score: 0
  
  I run Windows and I reboot on average once every two months or so because Windows Update wants to replace a file that's in use. Hibernating and waking up is much faster than shutting the computer down and booting it again, and has the added benefit of keeping your documents and browser windows open.
37. Re:Now only... by hey! · 2015-12-26 18:32 · Score: 2
  
  I have no idea why people insist on boot times. You are aware that a machine spends little time booting and a hell of a lot more running things?
  Sure, except when you do reboot it might be important to do it quickly some of the time.
  It's absolutely true that it's more important to optimize the common case over the uncommon case, but that doesn't mean the uncommon case is necessarily insignificant.
  
  --
  Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
38. Re: Now only... by Anonymous Coward · 2015-12-26 18:50 · Score: 0
  
  The days of having a dozen different partitions for things have long passed. The whole /bin /usr/bin came about from someone trying to save 500k of disk space and somehow it carried over decades later.
39. Re: Now only... by Anonymous Coward · 2015-12-26 20:04 · Score: 0
  
  I don't believe you. Your unsubstantiated anecdote weighs nothing on the scale of reason.
40. Re:Now only... by driblio · 2015-12-26 20:34 · Score: 2
  
  Do try harder.
  http://www.freedesktop.org/wiki/Software/systemd/journal-files/
41. Re: Now only... by Anonymous Coward · 2015-12-26 20:38 · Score: 0
  
  Not sure what's that supposed to mean. You only end up filling up the entire / partition instead. That's not great either.
42. Re:Now only... by Anonymous Coward · 2015-12-26 20:47 · Score: 0
  
  Ignorant systemd proponents who've never used logrotate don't understand that logrotate can rotate based on logfile size.
  Poettering could have saved us all a lot of trouble if he'd gone into politics rather than programming.
43. Re:Now only... by driblio · 2015-12-26 20:48 · Score: 1
  
  So if this is actually a mission critical set up, 1) you will already be exporting logs to an actual log server which is still up, 2) you can turn on plain text logs to support your offline troubleshooting, but still have the superior logging with meta data while online and 3) if you're fixing production with bootable CD you really need to work on your version control and release procedures.
  So I'm guessing you've never managed a server making/losing thousands dollars and/or actually used or configured journald properly. I don't like systemd (as an init system), and there's no reason logging should be so tied up with it. But journald is fine.
44. Re: Now only... by Anonymous Coward · 2015-12-26 20:58 · Score: 0
  
  Why for flying fuck are you booting your computer? That's once in a year, one in a quarter at worst task. Desktops are supposed to be used, not labored over. That's why I deleted all Linux from all my systems in 2007 and got Macs and few BSD systems. When I get home and want to watch a video or listen to music I just grab the remote and I'm in business. No need to compile codecs, graphic card drivers, X11/xfree, kernel and god knows what else every day just so I can play this particular file and what not.
45. Re:Now only... by Anonymous Coward · 2015-12-26 21:27 · Score: 0
  
  [I apologize to the masses for responding to yet another offtopic systemd propagandafest thread.]
  Oh, fuck off, will you? That intro made the rest of your post completely uninteresting. It might have contained valid questions, but I can't be bothered looking at them after your intro. Is it some kind of condition you systemd-haters have that is forcing you to be arrogant asses at every single turn or is it something else?
46. Re:Now only... by thegarbz · 2015-12-26 22:01 · Score: 1
  
  If you're that attached to your text logs just set journald to output them in the classic way.
47. Re:Now only... by thegarbz · 2015-12-26 22:04 · Score: 1
  
  logrotate runs periodically. It doesn't continuously poll for size making it trivial for an "event" to fill up a log file. I've seen a system with logrotate that worked perfectly for years keeping logs under control go down during brute force attack (why there was no brute force prevention is a different discussion) all because the log directory (which shared a partition with more critical parts of the system but that's a different discussion) filled up the entire partition.
48. Re: Now only... by thegarbz · 2015-12-26 22:09 · Score: 1
  
  Except stderr and syslog messages are not dropped. Better still they are recorded which is a step up from the old way and you can set them to not only replicate the old behaviour but keep logging the new way at the same time.
  Talking out of one's ass and not reading the fucking manual is the problem most people have with systemd.
49. Re:Now only... by Anonymous Coward · 2015-12-27 05:45 · Score: 0
  
  Nothing is wrong with binary logs as long as they can be accessed from a dumb rescue boot system (eg. flash-drive/DVD/floppy/whatever).
  Instead systemd has many dependencies that all have to be met to even view the log. Including specific versions of libraries and binaries that must match the systemd used on the system you're trying to recover. It's fucked up.
50. Re:Now only... by Anonymous Coward · 2015-12-27 10:21 · Score: 0
  
  "OMG, please take a bunch of time out to tiresomely teach me what the thousand things wrong with binary logs are again. Why is everybody so mean to systemd?! Everybody saying anything bad about systemd are bandwagoning idiots who have no rational reasons to avoid systemd!"
  Systemd shills are past hope to reason with. I just can't understand why they can't see that all the complaints about systemd are rooted in the despair of having freedom to choose alternatives taken away and having software forced on people. It doesn't matter that your religion makes you believe systemd is "The Way", stripping people's alternatives and calling them names for having different preferences and reasons for wanting the freedom to choose is fucked.
  Systemd proponents are advocates against freedom. Systemd proponents are the same as anyone who thinks nothing of stomping freedom in order to force what they believe is best for everyone even when others don't agree. It's unfortunate so many people who hate freedom so much like that creep into positions of power and then use that power to force people to bend to their will, pulling them away from things they've grown to love.
51. Re:Now only... by Anonymous Coward · 2015-12-27 10:26 · Score: 0
  
  yes, which is why we use plain text for these kinds of mission critical things, because in 7-bit ascii, one bit of corruption corrupts one character, and it UTF-8, each character sychronizes the stream, so you can corrupt at most two characters.
  furthermore, everyone knows how to read and write text, and UTF-8 is backwards compatible with text in the sense that things that read and write text can also read and write UTF-8 if they don't do anything with the high-order bit.
  the fact that text is slower to parse than binary formats is far less relevant today than it was 40 years ago.
52. Re:Now only... by nadaou · 2015-12-27 16:28 · Score: 1
  
  The story was about apt, not systemd. Therefore I was preemptively modding myself offtopic exactly so people like you who would be uninterested could quickly move on and not waste your time on it. Due to your wildly emotive response I guess I failed in that attempt.
  
  --
  ~.~
  I'm a peripheral visionary.
53. Re:Now only... by Anonymous Coward · 2015-12-28 04:02 · Score: 0
  
  >"OMG, please take a bunch of time out to tiresomely teach me what the thousand things wrong with binary logs are again.
  Basic Alinsky tactic. "If they reply to all letters, we'll send 30.000 of them."
54. Re:Now only... by Anonymous Coward · 2015-12-28 13:18 · Score: 0
  
  Of course. It's not just plain text appended to a set of files so you would have to use journalctl to access them. Nothing wrong with that.
  There is everything wrong with that. The non-standard journalctl log file format means that hundreds of standard system tools expecting text can't be used with the logs. Sure I can convert the log[s] to text but that obviates one supposed advantage of journalctl - efficiency.
  
  Now this gives you a number of benefits.
  Pretty much none actually. There's some theoretical benefits endlessly spammed by journalctl propagandists but in practice they don't exist.
  
  Since journald actually knows about the data that it stores it can make intelligent decisions.
  Intelligent decisions? Don't make me laugh. An intelligent decision is one where the system administrator can make smart choices based on their needs without being second guessed by really dumb software. e.g. The so-called journalctl "rotation" doesn't actually do anything useful.
  
  Everyone who has ever had to clean up a filled up /var/log partition knows how broken logrotate is.
  You're bullshitting. That is not a problem in practice and you know it. In any case journalctl doesn't fix or even address this problem. And why are you using a partition, which is nothing more than an inflexible and hard to manage directory?
  
  What if the journal could actually know about this and rotate based on size?
  Easily possible with traditional tools. But guess what? Most administrators don't bother. Journalctl just makes the process opaque and non-standard. The journalctl man page doesn't even document it. Good luck importing, exporting, organizing and processing logs.
  
  It also makes it possible to actually search for things without having to take different date and log formats into account, and actually know that it came from the correct program.
  You're bullshitting again. Not a practical problem - normally you're only searching for the specific subsystem with an issue anyway and using a familiar editor or tool to do it. Being hamstrung by a tool with basic, unrealistic search options just makes the whole process painful.
  
  I know, crazy.
  You're the crazy one for thinking that the minor, theoretical annoyances you've mentioned outweigh the major, practical journalctl problems including yet another log format (binary this time), non-standard compression, a non-standard pager, default, silent suppression of log contents (default -l - that's been fixed now but it demonstrates the incompetence and hubris of the developers that they even put it in to start with; systemctl status still does it and in addition just to add to the confusion displays historical log messages as current).
  To give the developers some credit this mess is mostly documented but you know what? Documentation is just a bandaid when the software itself is such a disorganized, unsystematic, thoughtless, sprawling mess. Some key documentation is still missing too.
  Some of the problems that systemctl/journalctl attempt to address are real (hot swapping, fast boot, various log files). Unfortunately however in this case the cure is worse than the disease. And the fact that the developers frequently lie about the so-called advantages of journalctl etc. and so-called disadvantages of alternatives doesn't help either.
Bad code is everywhere by mveloso · 2015-12-26 08:50 · Score: 3, Funny

Wow, reading one byte at a time unbuffered? Who does that in real life? It's been well-known for like 30 years that buffered reading is an order of magnitude faster than byte-at-a-time - which matches the above result. The standard C library does buffered reads, unless you turn them off explicitly.
Did someone really turn that off explicitly? Why?
Jesus, someone should check the XML parsers. Maybe the same guy wrote an XML parser and it's doing byte reads.
1. Re:Bad code is everywhere by Anonymous Coward · 2015-12-26 09:32 · Score: 0
  
  Wow, reading one byte at a time unbuffered?
  Instead of bitching about what someone else has done, it would be far better if you actually contributed something yourself. Talk is cheap, particularly in online forums.
2. Re:Bad code is everywhere by guruevi · 2015-12-26 09:39 · Score: 5, Insightful
  
  a) Back in the day we did because memory was expensive and these things were to run on 386'es and other platforms that might not have the room for a sizable buffer and memory/bus/CPU bus were all equally fast. You only need a buffer if your machine is busy doing other things)
  b) It might be a benefit on certain platforms but in certain situations it feels (without looking at the rest of the code) like the code might introduce a buffer overflow issue (he explicitly removes the zero-buffer option if the file read returns a null pointer as it's buffer).
  c) Ask the original developer or do a blame-search for that code before 'fixing' things.
  
  --
  Custom electronics and digital signage for your business: www.evcircuits.com
3. Re:Bad code is everywhere by jaklode · 2015-12-26 10:04 · Score: 5, Informative
  
  It's not using standard I/O function, but pure syscalls, which are obviously unbuffered. And the same code paths are also used for other stuff that maps files. Performance critical code implemented a buffer on top of that, and the ReadLine() function experiencing the main issue was only added as a convenience function and not used for anything critical until a few months ago (and we forgot that it was not optimized). Anyway, we implement buffering for ReadLine() now. I'll try to make it generic for both reads (all reads) and writes, but so far have not succeeded, probably because some code depends on unbuffered reads or writes.
4. Re:Bad code is everywhere by mveloso · 2015-12-26 10:53 · Score: 1
  
  I was doing buffered reads 20 years ago. You only need 1k of buffer to get a substantial performance improvement...and that was with floppies and tape.
  I mean, don't they teach this stuff in school? The disk travels at x RPM, so every byte you read means you have to wait for the sector to come around again. It doesn't really matter what x is (unless it's an SSD), because it's slow. It's like forever slow. You might as well get coffee and go to the bathroom waiting.
  This is like I/O 101.
5. Re:Bad code is everywhere by Anonymous Coward · 2015-12-26 11:13 · Score: 0
  
  I'm just a poor Java developer, and am thus lucky to have a huge standard library at my disposal. It wouldn't even cross my mind to reimplement such a basic function as ReadLine(), or even buffered IO, because it's obviously already done and battle-tested in the standard Java library.
  Why does a program like apt feels the need to reimplement something as basic as buffered IO and ReadLine rather than use a standard library function to do that? Is the standard C library so poor than it doesn't have a ReadLine function?
  I'm sure people writing apt are all smart people and everything, but seeing that, from my point of view, is really scary, and looks like reinventing the wheel.
6. Re:Bad code is everywhere by bioteq · 2015-12-26 11:26 · Score: 1
  
  The standard library doesn't come with a ReadLine function. It has the ability to read I/O, and to buffer it, but it is up to the implementation to do it how they see fit. In the case of APT (I cannot say I am surprised. I would really love to hear the reason they made it unbuffered to begin with,) since it is in C and a core function, it will be using ONLY the C library. So they have to implement the ReadLine() themselves.
  C was designed to be barebones. It's why you can easily shoot yourself in the foot (like in this case) and it's the reason I prefer C over any language. It's not that the library is poor; it's not -- it's actually quite well done. It just does not have a lot of the built in cruft that some languages hang on to, such as Java. It IS pretty standard practice to implement ReadLine() in a specific way; again, I would love to know the real reason. I am better it was debug code that 'worked' and no one thought to look at the most basic of I/O as a performance issue (they were looking at parsing, formats, network latency, etc)
7. Re: Bad code is everywhere by Entrope · 2015-12-26 11:47 · Score: 1
  
  The buffering in question here is user-space buffering. The lousy unbuffered code still had the benefit of the kernel's page cache, but had to make a system call for each read. If it went to disk for each read, the performance hit would have been many orders of magnitude, not just one.
8. Re: Bad code is everywhere by Entrope · 2015-12-26 11:57 · Score: 1
  
  C has had a buffered line reading function approximately forever. It is called fgets().
9. Re:Bad code is everywhere by Anonymous Coward · 2015-12-26 12:05 · Score: 0
  
  I can imagine some reasons why the standard C library, which is low-level and used everywhere, doesn't have a standard way to read lines.
  But that Linux, or Debian, doesn't have a core shared library somewhere that is able to read lines is, quite frankly, astonishing to me. I guess there must be 1000 different implementations of it. Is that really true? Is that the level of engineering of a Linux distro these days?
10. Re: Bad code is everywhere by bioteq · 2015-12-26 12:15 · Score: 1
  
  Yes, we have fgets(), but it's not exclusively a ReadLine() function, so to speak. Sane people would write a ReadLine() function using fgets() wrapped in some nice loops to read in buffered input. APT -- apparently not so much. I can only assume they were using getc or some such. I may dig in to apt-get source here in a few to see exactly what they were doing.
  I've been looking for a project to dedicate some spare time to and, seeing someone earlier in the discussion saying it only has four people working on it, I may give it a bit of my time.
11. Re:Bad code is everywhere by bioteq · 2015-12-26 12:16 · Score: 1
  
  I agree, 100% and, quite frankly, yes. That is the level of some engineering in linux.
  I am die hard linux; I love the idea, and I use tons of distros every day. I love the freedom. But they reinvent the wheel so often, that some times people forget the wheel is meant to be round.
12. Re:Bad code is everywhere by Anonymous Coward · 2015-12-26 12:33 · Score: 0
  
  Quite refreshing to see a Linux fan with common sense!
  FWIW, Java has a buffered readLine function. Quite surprisingly, it's in the class java.io.BufferedReader, and is named readLine().
  I see Java being bashed here again and again, but quite frankly, I'll take its verbosity and reusability any time over the horrendous naming of C functions.
13. Re:Bad code is everywhere by swillden · 2015-12-26 12:58 · Score: 1
  
  But that Linux, or Debian, doesn't have a core shared library somewhere that is able to read lines is, quite frankly, astonishing to me.
  The question is whether there was such a library when apt was written, in the early 90s. And whether it was guaranteed to be present on a base install, or whether it was something that apt was going to be installing. If not, then apt couldn't use a shared library version so it would have to statically link it, which may have argued for implementing something very simple instead.
  Remember that when apt was written installation was generally done from floppy disks so a little inefficiency in the parsing code wouldn't have been noticed.
  
  --
  Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
14. Re:Bad code is everywhere by bioteq · 2015-12-26 13:25 · Score: 1
  
  I hate to admit it, but a lot of "Linux Fans" lack common sense; they simply go where their friends/peers go. That whole hipster thing. Meh. I see things for their value, and their cons. I use windows AND linux, and I like both.
  I am NOT a fan of Java. But I see it's worth quite well. It has its use. A tool for every job. I, personally, prefer C, but that's because I've always used C (it's what I started on all those years ago) and it's what i always come back to. It's like that abusive Ex that you just cannot quit, but the sex is good, so you stay.
  C function names....I could write a book on that rant. Truly. C is horrid when it comes to standards, conventions, etc with naming. Hell, even with arguments. LibC itself is a pain for those who aren't intimately familiar with it.
15. Re:Bad code is everywhere by Anonymous Coward · 2015-12-26 14:02 · Score: 0
  
  Yes, let me completely DROP the project I am working on, and go and fix some incompetent programmers' bugs (yes, plural).
  Please use our wonderful open source software! And remember, if you want something done right, you can always do it yourself!
16. Re:Bad code is everywhere by Anonymous Coward · 2015-12-26 14:04 · Score: 0
  
  No, C is not "horrid when it comes to standards, conventions, etc with naming", the Indian Hill C Style and Coding Standards was at rev 6.0 over 25 years ago: https://www.cs.arizona.edu/~mc...
  You're just inexperienced and trying to sound important on an anonymous message board.
17. Re:Bad code is everywhere by bioteq · 2015-12-26 14:17 · Score: 1
  
  You're fucking kidding me, right? Go away, Troll.
  Just because there is a 'coding standard' does not mean it has good naming standards. I'm not -downing- C -- I've already stated several times that I -love- C and prefer it over anything else. I've been writing C applications since 1990. Both professionally ( as in, for PAY on major projects ) and on my own projects. I think I have earned a right to not be 'inexperienced.'
18. Re: Bad code is everywhere by mrprogrammerman · 2015-12-26 16:09 · Score: 1
  
  Exactly. It's the system call that kills the performance. Any decent kernel cache will have the data cached and even read-ahead in the background.
19. Re:Bad code is everywhere by Anonymous Coward · 2015-12-26 17:06 · Score: 0
  
  C is so good at what it does that no other programming language has honestly tried to fill the niche since. We're probably spoiled.
20. Re:Bad code is everywhere by jabuzz · 2015-12-26 22:05 · Score: 1
  
  Don't know put Office 2012 on the Mac does single byte writes when you save a file. Causes a file save over SMB to take 15 minutes. Do a "save as" and the same file takes a few seconds. Of course saving to local disk and you don't notice the difference.
  So if Microsoft with a team of paid full time programmers can make such stupid mistakes a why do you expect better of a little used piece of code (because for a large section of the Debian user base bandwidth long ago ceased to be an issue) that is maintained in peoples free time to be better?
21. Re:Bad code is everywhere by tshawkins · 2015-12-27 01:52 · Score: 1
  
  Go is targeted at this appliction area.
22. Re: Bad code is everywhere by Anonymous Coward · 2015-12-27 05:35 · Score: 0
  
  This code was from 4 months ago. Read the article. It had nothing to do with old practices, etc.
23. Re: Bad code is everywhere by jaklode · 2015-12-29 22:33 · Score: 1
  
  We were practically calling read(fd, buffer, 1). We're not using standard C library I/O functions, but raw syscalls. Our ReadLine() function does the same as fgets(), BTW.
Yum by Anonymous Coward · 2015-12-26 08:57 · Score: 0

Now, if only someone would do something about how balls-slow updating systems with yum seems to have gotten over the last so many years.
1. Re:Yum by kthreadd · 2015-12-26 09:11 · Score: 2
  
  Someone did, the result is called dnf and has replaced yum in Fedora.
2. Re:Yum by bioteq · 2015-12-26 11:28 · Score: 1
  
  The production server that I work with runs CentOS and its YUM updates are fairly quick. Granted, it has a super, super fat pipe and some pretty nice hardware. But on a production system, I'm not looking for speed in updating, I am looking for consistency and "don't fuck my shit up, yo!"
3. Re:Yum by Anonymous Coward · 2015-12-26 23:56 · Score: 0
  
  What does a RH rip-off have to do with the old Debian apt-get (most use aptitude)?
  The speed difference here won't be noticed. Network IO is the single slow point. Anything CPU bound in 2015 isn't noticeable when it comes to updating.
4. Re:Yum by bioteq · 2015-12-27 05:10 · Score: 1
  
  It has nothing to do with apt. I was just replying to the original A/C who said YUM was slow.
Many thanks by NotInHere · 2015-12-26 09:00 · Score: 3, Informative

Really looking forward to have apt-get speeds that can be compared with pacman. Julian Andres Klode, if you read this, please continue the great work!
1. Re:Many thanks by jaklode · 2015-12-26 10:13 · Score: 5, Funny
  
  I'm reading everything :)
2. Re:Many thanks by Anonymous Coward · 2015-12-26 17:28 · Score: 1
  
  can you make portage faster? ;)
3. Re:Many thanks by Anonymous Coward · 2015-12-26 23:04 · Score: 0
  
  Easy, rewrite it in a proper language instead of python.
4. Re:Many thanks by Anonymous Coward · 2015-12-28 23:14 · Score: 0
  
  Thanks for working on this. Thanks for reporting about. Sorry about the people who think they know that always using buffered IO would be the right solution, which just means they never found that for a system it can be faster to avoid that.
  This incremental update slowness, if you can fight it successfully will pave the way for apt-get to be even faster in that path, making it possible for it to become the default, and that would really be saving for everybody. So please continue doing this.
  Also, annoying how people talk of production as if the code changed here wasn't in production and be very hard to change, as it's used in so many ways and as if the people who wrote it, had no reasons ever.
  Don't be detracted.
XML is broken by design performance-wise... by ffkom · 2015-12-26 09:21 · Score: 3, Informative

... it's not even a prefix-code "per document", so only using buffered reads when parsing XML wouldn't even allow you to avoid reading into the data following the end of the XML document you actually want to parse.
If something is XML based and time criticial, I wouldn't bother to optimize the XML parsing, but rather exchange XML for a non braindead format to start with.
1. Re:XML is broken by design performance-wise... by Anonymous Coward · 2015-12-26 12:57 · Score: 0
  
  There are tricks to doing XML well but it depends on a reasonable overall system design. However, XML is really bad for ultra-high performance given that it does not have element sizes encoded into it. (Thus requiring some form of parsing to find element ends and thus no reasonable way to skip over an element without actually parsing them (especially since each one can nest) And don't talk about entities and their complex relationship to the parsing state.
2. Re:XML is broken by design performance-wise... by hey! · 2015-12-26 19:06 · Score: 2
  
  There's nothing braindead about XML, it's just not the right tool for many jobs it is used for. It's over-engineering a solution to a simple problem because everyone else is doing it that way that's braindead.
  When XML became hot around the end of the 90s, people did what they always do with a hot technology; they used it whether or not it made sense just to have it on their resume. It never made sense for things like an over-the-network data serialization format; or as a configuration file format where some kind of key/value file format would do; but it was all over the place in those kinds of roles. What it's really good at is encoding complex documents re-using multiple heterogeneous standards (e.g. a text document with SVG diagrams, MathML expressions, and Dublin Core metadata). It's a comprehensive and practical tool to use on those kinds of problems.
  
  --
  Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
3. Re: XML is broken by design performance-wise... by Anonymous Coward · 2015-12-27 04:20 · Score: 0
  
  And people that did PKI already did ASN.1 I was having a really hard time explaining to all the Javarasts why we should not put OCSP responses in XML but those Javarasts never listen
How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-26 09:24 · Score: 4, Interesting

Holy fuck, I just read the blog article and this is what it said:

The reason for this is that our I/O is unbuffered, and we were reading one byte at a time in order to read lines.

Writes are still unbuffered, and account for about 75% to 80% of our runtime.
How the fuck did that all happen to begin with?! Who the fuck wrote the code like that initially?! How fucking long has this been the case?!
I understand that bugs will happen. I really do. But this is like a total breakdown in process. Why the fuck weren't these problems detected sooner, like when the code was committed and reviewed? The code was reviewed, right?!
We aren't talking about some minor software project here. This is apt, for fuck's sake! This is one of the core pieces of Debian's (and Ubuntu's, and many other distros') basic infrastructure. This is the kind of shit that has to be done properly, yet it clearly wasn't in this case.
The Debian project needs to address how this shit even happened to begin with. This is fucking unbelievable. The entire Debian community deserves a full explanation as to how this debacle happened.
I thank God every day that I moved all of my servers from Debian to OpenBSD after the systemd decision. I know that the OpenBSD devs take code reviews and code quality extremely seriously, not just with the core OS itself, but even with software written by others. The OpenBSD project will even create and maintain their own custom forks of third party software if the original developers can't get their shit together!
1. Re:How the fuck did this slowness even happen?! by zopper · 2015-12-26 10:17 · Score: 3, Interesting
  
  IMO, this originates in early days of APT. And because it worked, nobody wanted to touch it later when the speed become an issue, because nobody wanted to risk breaking it, because, as you said, it is a critical part of the infrastructure.
  And now, though, when you explained us how BSD is better, go back and pretend there are no such old lines hacked together long time ago, with nobody dusting them in years. (By the way, how long there was, for example, HeartBleed in BSD? Until discovered and fixed also on every other platform? Wow, they do really great job with the reviews!)
2. Re:How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-26 10:19 · Score: 0
  
  Somebody should mod up the parent comment. Those are good questions to be asking! In one of the comments the blog author says, "We kind of forgot that we did not efficiently implement ReadLine() when we started using it for rred. The code in question was only in APT 1.1 releases and pre-releases uploaded during and after DebConf, so for about 4 months." So there is no good reason for this to have happened. One or more people were apparently sloppy and bad code got through. The Debian project is really going through some rough times these days. The systemd transition was an absolute nightmare, and it drove away some of Debian's most experienced and knowledgeable users. Now we have really bad code like this getting into APT. Debian used to be known for upholding extremely high standards but those standards do appear to be slipping now. Debian is a far cry from the respected Linux distribution project it once was.
3. Re:How the fuck did this slowness even happen?! by jaklode · 2015-12-26 11:08 · Score: 5, Informative
  
  Gosh it was slow code. Not so much bad code. There are 3 people working on APT in their free time. Instead of complaining, do something about it. You can join #debian-apt on OFTC, view the commits live, and check them. What matters more is that there are almost no functional regressions thanks to our test suite. Checking for performance regressions is an entirely different beast altogether.
4. Re: How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-26 11:26 · Score: 0
  
  How much money or time have you invested in it? If not, why do you think others would?
5. Re:How the fuck did this slowness even happen?! by MobileTatsu-NJG · 2015-12-26 12:02 · Score: 0
  
  I understand that bugs will happen. I really do. But this is like a total breakdown in process. Why the fuck weren't these problems detected sooner, like when the code was committed and reviewed? The code was reviewed, right?!
  That's what's great about Open Source and all those millions of eyeballs!
  
  --
  
  "I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)
6. Re:How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-26 13:36 · Score: 0
  
  LOL, you're totally wrong!
  "The code in question was only in APT 1.1 releases and pre-releases uploaded during and after DebConf, so for about 4 months."
  LOL, "this originates in early days of APT"! LOL! NOPE!
  LOL, "because it worked, nobody wanted to touch it later when the speed become an issue"! LOL! NOPE!
  LOL, "because nobody wanted to risk breaking it"! LOL! NOPE!
  It's hilarious how fucking WRONG you are! LOL!
7. Re: How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-26 14:56 · Score: 0
  
  Why so upset that performance was not all it could be if you used a functionality in apt (pdiff) that you probably never ever used?
  Did you put Microsoft in front of a firing squad when they accidentally debug compiled NTFS all those years?
8. Re:How the fuck did this slowness even happen?! by Kjella · 2015-12-26 16:42 · Score: 2
  
  The reason is that while apt is a critical piece of infrastructure, incremental apt really isn't. The use case is that you're supposed to download a smaller diff, apply the diff and install the updated version either faster or cheaper if you're on a metered line. If you don't have a compatible earlier version, you have to use the traditional package anyway. So you can always fall back to that, it's only nice-to-have. The people on broadband don't care, the ones on really slow lines probably want their updates offline by some other channel. But for those who do care, usually around 56k dial-up speed or so it has mostly been solved by making modular packages that depend on a main package so you only update the parts that are needed. Incremental packages would be even more fine grained, but is of relatively little importance, which is why few people care and code probably hasn't been reviewed much for performance.
  
  --
  Live today, because you never know what tomorrow brings
9. Re:How the fuck did this slowness even happen?! by jtayon · 2015-12-26 19:52 · Score: 0
  
  I have been working with a debian developer. They do like crappy code as long as "it does the job".
  We were reading UDP datagram (SIP stuff) 1 byte at a time because the debian developer thought expect was bad and was unable to figure any decent way of parsing newlines.
  We were having a best case of 15ms for parsing 1500 bytes, and a worst case of 20 seconds (yes do 1500 read(buff, 1) that generates 1500 system calls to be fast, context switch are SOOOOO cheap) . I was doing VoIP... in 10 seconds you normally expect your communication to be working. He really was pissed when I began showing the distribution graph of how long parsing 1.5Kb would take. I still wonder if I was fired for this graph.
  Anyway, it was the last push that helped me move to a BSD system, and I am glad about it, now I am looking for a job.
  Never again I work with debian developers. They are the worst free softwares fucktarded I ever had to deal with, full of principles and ideology and totally clueless about computer science or anything close to business.
10. Re: How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-26 19:58 · Score: 0
  
  I miss the days when user comments on slashdot used to teach me something. These days they look like ten year olds arguing in between classes, with the occasional retard as above discovering the joys of dick measuring.
11. Re:How the fuck did this slowness even happen?! by TheRaven64 · 2015-12-26 20:53 · Score: 1
  
  Indeed. I actually read TFA, expecting to see some neat algorithmic improvements in how they did incremental updates. Instead, the post read like a first-year undergraduate student assignment writeup ('we tried doing the really naive thing, it turned out to be really slow, so we did the obvious thing').
  
  --
  I am TheRaven on Soylent News
12. Re: How the fuck did this slowness even happen?! by gl4ss · 2015-12-26 21:15 · Score: 1
  
  well the lol seems right and the well written wrong.
  in early days you would certainly have cared. you know, installing debian in a day or 4 days would have mattered.
  
  --
  world was created 5 seconds before this post as it is.
13. Re:How the fuck did this slowness even happen?! by ZosX · 2015-12-26 23:51 · Score: 1
  
  That's truly shocking. Even a moron like me knows that is horribly awful design that sounds like it came straight from writing code on a COCO3.
  
  --
  zosxavius photography
14. Re:How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-26 23:53 · Score: 0
  
  What a drama girl you are. APT updates are blazingly fast, and on a Stable Debs, you're hardly getting them. Give it a rest with your OpenBSD shilling, no one bothers with that out of date shit in the real world.
15. Re:How the fuck did this slowness even happen?! by jaklode · 2015-12-27 00:16 · Score: 2
  
  Buffered reading my seem obvious too you, but the code in question had some users outside the class that worked directly on the file descriptor, and thus will not work correctly with buffered reading. Also, it's used in other code that implements its own buffering or does not need one, because it simply copies page-size blocks from one file to another. As another example, I have not figured out how to do go completely buffered without messing everything up yet: test suite fails all over the place if I switch from syscalls read,write,etc. to fread(), fwrite(), etc. Write buffering is especially weird: Callers now need to really make sure to actually flush the contents to the disk (or close the file), otherwise things will fail. As we cannot expect all callers to do that, we probably need to turn this into an optional flag.
16. Re:How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-27 00:53 · Score: 0
  
  you could also have added:
  LOL, "HeartBleed in BSD? Until discovered and fixed also on every other platform?" LOL! NOPE!
  for good measure, because you know, HeartBleed had nothing to do with BSD or OpenBSD whatsoever. It was a failure in OpenSSL which, contrary to what its name seems to indicate, is not developed by the OpenBSD project, never was. Reaction to this from OpenBSD folks: the code of OpenSSL is such a pile of crap that it cannot fixed through the normal "pull request to upstream" way, let's fork it instead. LibreSSL was born.
17. Re: How the fuck did this slowness even happen?! by Noah+Haders · 2015-12-27 01:39 · Score: 1
  
  Nobody knows what systemd really means, or where it came from.
18. Re: How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-27 05:22 · Score: 0
  
  http://www.openbsd.org/users.html
  Yea no one uses openbsd. You are a moron. Ever heard of Adobe? Yea they use openbsd. How about John Hopkins university, you know the school responsible for churning out top doctors, yes they use openbsd in many departments.
  It's not shilling if he's stating facts about a system.
19. Re: How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-27 05:40 · Score: 0
  
  You make me feel bad for Theo.
20. Re:How the fuck did this slowness even happen?! by TheRaven64 · 2015-12-27 05:52 · Score: 1
  
  Which sounds like it's really poorly structured code, with the kind of layering that would make me fail a student project, and bounce back in code review anything that was intended to go anywhere near production.
  
  --
  I am TheRaven on Soylent News
21. Re:How the fuck did this slowness even happen?! by DrJimbo · 2015-12-27 10:41 · Score: 1, Interesting
  
  Gosh it was slow code. Not so much bad code.
  I fail to see the distinction. This is painfully bad code. I, for one, would not enjoy working with people who think this is not really bad code.
  
  --
  We don't see the world as it is, we see it as we are.
  -- Anais Nin
22. Re:How the fuck did this slowness even happen?! by Anonymous Coward · 2015-12-28 22:54 · Score: 0
  
  Fortunately they don't have to enjoy working with you.
23. Re:How the fuck did this slowness even happen?! by jaklode · 2015-12-29 22:44 · Score: 1
  
  This is basically right. The reads were always unbuffered, we just added a ReadLine() in 2011 - that was actually more efficient and read entire blocks at once and then seeked back if it read to much - but this broke pipes, so it was switched to read a single byte at a time. Anyway, the code was a real disaster due to it's historic growth of adding various compression formats - but it is now nicely refactored, so we actually know what's going on, and can easily add new formats, buffers or anything we like.
APT! APT! by Anonymous Coward · 2015-12-26 09:25 · Score: 0, Funny

Apt appters apdate with APT, not luddite source!
APTERS APTERS APTERS
Does DNF still use the same RPM handling code? by ffkom · 2015-12-26 09:28 · Score: 2

If yes, I would wonder how much speed could it gain over yum... after all, whenever I used yum to update some OS, the vast majority of time was spent with the stuff "rpm" would do, not with the database maintainance yum adds.
1. Re:Does DNF still use the same RPM handling code? by Anonymous Coward · 2015-12-26 11:30 · Score: 0
  
  DNF is WORSE than yum. It is written in Python and runs slowly due to all the interpreter overhead, rather than being compiled to machine code. Additionally, the memory use is much higher in my tests as well.
LOL by Anonymous Coward · 2015-12-26 09:43 · Score: 0

"LTS" ...can't believe anyone actually falls for that one.
only two enhancements remaining by Anonymous Coward · 2015-12-26 09:47 · Score: 1

That's good.
Now it reamins 2 enhancement I'd really like to see :
low free space on hdd management.
parallel installation
low free space : walk the dependencies to find apt with no dependencies, install, clean [repeat]
parallel installation : right now apt download all files then install them it will be better to (download/install) each apt
1. Re:only two enhancements remaining by Anonymous Coward · 2015-12-26 11:11 · Score: 1
  
  low free space : walk the dependencies to find apt with no dependencies, install, clean [repeat]
  "sudo apt-get autoremove" will remove all packages that were not explicitly installed and are not depended on by any explicitly installed packages.
  
  parallel installation : right now apt download all files then install them it will be better to (download/install) each apt
  This would probably not actually cause a significant improvement in performance. If you're pointed at a fast mirror, downloading eight packages at once that max out your bandwidth will not be any faster than one package that maxes out your bandwidth. Similarly, when actually installing them, you'd need to determine what order they could be installed in to ensure that dependencies are safely installed before their dependants, so you wouldn't be able to parallelize the install very much; furthermore, one package that's maxing your disk's write speed will be just as fast as eight that are maxing it out. The installation process isn't particularly CPU intensive, so spreading it across multiple cores wouldn't really help.
2. Re:only two enhancements remaining by jaklode · 2015-12-26 11:18 · Score: 5, Informative
  
  Well, we do download in parallel if you use httpredir.debian.org and httpredir.debian.org returns different mirrors for different packages (which it does not do all the time, but reasonably often). I don't like installing in parallel, or downloading and installing at the same time, as they just make the error handling harder, for modest speedup.
3. Re:only two enhancements remaining by Anonymous Coward · 2015-12-26 21:58 · Score: 0
  
  If you're pointed at a fast mirror, downloading eight packages at once that max out your bandwidth will not be any faster than one package that maxes out your bandwidth.
  It's pretty hard to max out your bandwidth with single stream when you have sufficiently fast connection due to various reasons. It's a lot easier to do it when you have multiple streams (and LACP, ECMP & similar things only work when you have multiple streams). Also, there's always some overhead associated with starting a new download which becomes very noticeable when you download lots of small file, especially if there's high RTT.
  Similarly, when actually installing them, you'd need to determine what order they could be installed in to ensure that dependencies are safely installed before their dependants
  You can calculate the install order while downloading the packages if it's somewhat slow operation. And while it's not uncommon that there's a large dependency chain, there's almost always quite a few packages which installation can be parallelized. Admittedly there might be some edge cases where installation might end up taking more time, but most of the time you would end up with a large speed boost.
4. Re: only two enhancements remaining by Anonymous Coward · 2015-12-27 04:43 · Score: 0
  
  You are wrong. Dependencies only matter for run time, not install time. As long as you install all needed packages before starting any of them you are in the clear. What apt and rpm are doing installing dependencies first is brain dead and shows 0 level of abstract thinking. I can remember how many times I had to force nodeps install rpms manually because of cyclic dependencies and I didn't break the systems even once.
5. Re: only two enhancements remaining by Anonymous Coward · 2015-12-27 07:15 · Score: 0
  
  only true for new installs. what about running programs/services?
6. Re: only two enhancements remaining by jaklode · 2015-12-29 22:39 · Score: 1
  
  There are multiple stages. A package can be unpacked (preinst script run and unpacked) or configured (postinst script run too). Dependencies only matter for configuration, that is, running the postinst script. There are special predepends for the preinst script, but they really should not be used unless absolutely necessary. The thing is that the postinst script might assume that any dependency is configured. There are also breaks (configure time) and conflicts (unpack time). APT also tries to respect dependencies where not needed, but it's not that strict about them.
Re:Debain is not Free by sunderland56 · 2015-12-26 10:34 · Score: 1

We need a FREE INTERNET for every human beign
How about we just start with a free spell checker?
meanwhile.... by Anonymous Coward · 2015-12-26 10:45 · Score: 1

in windows land, windows updates slow to a friggin crawl, can take hours, and with the new windows 10{dot}shit, lacks any sort of user control... the exact opposite of debian and apt.
Stop by Anonymous Coward · 2015-12-26 11:02 · Score: 0

Stop this now you dogs.
WHY? by Anonymous Coward · 2015-12-26 11:04 · Score: 0

Because the idea is to speed up the tunnel of love dawg!
1. Re:WHY? by rawtatoor · 2015-12-26 11:06 · Score: 1
  
  Cause getting over on stuff because peaple are stupid is over causa me dawg! Peaple are gonna die here! BE CAREFUL!
Is there journalctl for other operating systems? by tepples · 2015-12-26 11:08 · Score: 1

You use journalctl to read them, or any other program that can read the well-defined on-disk format.
Say you're trying to troubleshoot problems with a machine by booting to a different operating system in order to read the logs of the machine's primary operating system. This works for plain text logs because support for ASCII text is ubiquitous, modulo some line ending weirdness in Windows. Do such "other programs" exist for all PC operating systems that can read ext2/3/4 file systems?
Re: Debain is not Free by Anonymous Coward · 2015-12-26 13:39 · Score: 0

I live in Australia you insensitive yankee clod!
What will it take for rpm and deb to merge ? by sandGorgons · 2015-12-26 17:39 · Score: 0

What will it take - money? Linus Torvalds throwing a hissy fit ? Ubuntu Snappy/Click packages ?
Linux adoption can be 10x what it is today if the package management is unified. Its not just about the format - its the whole ecosystem. One Linux App Store, better tooling to build packages, only one package to publish. Any kind of performance improvment (like this one) benefits both sets of users.
I mean RPM and DEB each have their advantages.. but what will it take for a single format to emerge ?
1. Re:What will it take for rpm and deb to merge ? by thegarbz · 2015-12-26 22:13 · Score: 1
  
  but what will it take for a single format to emerge ?
  A use case?
2. Re:What will it take for rpm and deb to merge ? by kthreadd · 2015-12-27 00:15 · Score: 1
  
  It's not just the package format. You can't even take a deb reliably between Debian and Ubuntu, or even between different releases of Ubuntu. There are different versions of libraries, sometimes with different soname. Making one package that runs everywhere is almost impossible unless you bundle pretty much every dependency.
3. Re:What will it take for rpm and deb to merge ? by jaklode · 2015-12-29 22:40 · Score: 1
  
  Have you looked at Limba and/or xdg-app?
Re:Is there journalctl for other operating systems by thegarbz · 2015-12-26 22:06 · Score: 2

Is this a requirement for you? Set journald to output classic plain text logs as well. You get all the benefits of a nice utility to sort through your logs and the ASCII text files which you find so critical to your use-case.
Sloooooooow. by Anonymous Coward · 2015-12-27 00:19 · Score: 0

So, apt is reading every character, unbuffered. And that's making it run at 10% of the speed it should. So just wtf is Windows Update doing to take hours?
A lot faster! by diablobsb · 2015-12-27 03:10 · Score: 1

andre@atlas:~# time apt &> /dev/null
real 0m0.002s
user 0m0.001s
sys 0m0.000s
WOW fast as hell!

--
I for one, welcome our new hot grits... PROFIT!
MOD UP by ArchieBunker · 2015-12-27 04:39 · Score: 0

Re-inventing the wheel again (poorly) because Pottering can't be bothered to read a man page.

--
Only the State obtains its revenue by coercion. - Murray Rothbard
SGML was there before XML... by ffkom · 2015-12-27 09:16 · Score: 1

... and perfectly covered the use case of which you say XML is good for. FrameMaker was one practical evidence to this.