The solution is to use/dev/urandom, but only after verifying that the pool has some entropy. Ideally, it would be nice to have an API that allows you to find out how many total bits of entropy have been gathered by the system, regardless of how many remain in the pool at any given point in time. If the system has managed to accumulate a few hundred bits, just use/dev/urandom and get on with life. If it hasn't, use/dev/random and block.
The solution is to use/dev/urandom, but only after verifying that the pool has some entropy. Ideally, it would be nice to have an API that allows you to find out how many total bits of entropy have been gathered by the system, regardless of how many remain in the pool at any given point in time. If the system has managed to accumulate a few hundred bits, just use/dev/urandom and get on with life. If it hasn't, use/dev/random and block.
This isn't so much about entropy "drying up" a few days after the system has booted - this is more about generating random numbers just after a system has booted and before "enough" entropy was gathered in the first place. From your link:
Not everything is perfect [...] Linux's/dev/urandom happily gives you not-so-random numbers before the kernel even had the chance to gather entropy. When is that? At system start, booting the computer.
but also from your link
FreeBSD does the right thing[...]. At startup/dev/random blocks once until enough starting entropy has been gathered. Then it won't block ever again. [...] On Linux it isn't too bad, because Linux distributions save some random numbers when booting up the system (but after they have gathered some entropy, since the startup script doesn't run immediately after switching on the machine) into a seed file that is read next time the machine is booting. [...] And it doesn't help you the very first time a machine is running, but the Linux distributions usually do the same saving into a seed file when running the installer. So that's mostly okay. [...] Virtual machines are the other problem. Because people like to clone them, or rewind them to a previously saved check point, this seed file doesn't help you.
Virtualization is a strong candidate because everything can be so samey but it can happen on real hardware too - imagine a trying to generate randomness on a basic MIPS based home router with flash based disks, no hardware RNG, typically booting from a fixed extract RAM disk install and doesn't have hardware clock to save time when powered off but makes ssh certs early during its first boot...
So this is bad for Linux systems that make security related "things" that depend on randomness early in startup but there may be mild mitigations in real life. If the security material is regenerated at a later point after boot there may be enough entropy around. If the the system is rebooted but preserves entropy from the last boot this may be mitigated for random material generated in subsequent boots (so long as the material was generated after the randomness was reseeded). If entropy preservation never takes place then regeneration won't help early boot programs. If the material based on the randomness is never regenerated then again this doesn't help. If you take a VM image and the entropy seed isn't reset then you've stymied yourself as the system believe it has entropy that it really doesn't.
First up lkml.org is a third party site that hosts Linux kernel mailing list archives on a website. Regular Linux kernel mail isn't actually sent from it (I believe that's done by vger) so we're looking up the email reputation for the wrong IP...
Secondly UCEPROTECT is a very aggressive blacklist which states upfront they will block people who they believe are in the vicinity of people who the judge to be sending them spam. It's not the be and end all though and on one server I looked some time ago it's effectiveness was surpassed by other blacklists (here's someone else's old DNS blacklist comparison for 2014). In general I prefer more conservative tools like senderbase when trying to work out an IPs mail reputation.
Anecdotes 1 + 2 were running in native mode because it was initially a fresh install that was then switched sysvinit-core. Anecdote 3 I don't know (most likely compatability mode as it was several years ago). Even if all the anecdotes were only running in compatibility mode the results show systemd finishing quicker...
Does the above make things any better and how fast do you expect things to go when you're bottlenecked on I/O throughput? Is 15 seconds for a hard disk or 4(!) seconds for a RAM disk so bad?
The NFS mounts were always mounted at boot and not on demand both before and after systemd. The difference was that because there were multiple NFS mounts only proceeded in serial before the switch. After the switch the waiting of the mounts happened in parallel not just with each other but other services too. Yes it is entirely pathological but it really happened (and presumably other parallel inits would have solved it this way too).
For what it's worth I have an EeePC too. The default Xandros distro was a classic case of not running very much because it was highly custom (e.g. no printing, no mdns, kernel doesn't have to probe for all different kinds of hardware, hotplug of USB devices, io scheduler set to deadline etc) and thus was fast - I would always expect it to outperform a generic "full fat" Linux distribution. I'd expect your FreeBSD to beat my current XFCE Ubuntu setup too because I bet you can get start less. With lots of hand tweaks the boot speed to an XFCE desktop on my EeePC 900 is still 17 seconds...
Anecdote 1: I've just timed a Debian Jessie single CPU hard disk based VM install with BTRFS as the filesystem, a GNOME 3 desktop where the user is auto logged in boot and where an autostart script records the time. Here are my rough systemd and sysvinit results (times are from after the kernel core finished to when the GNOME script ran): sysvinit (apt-get install sysvinit-core) First boot: 20 seconds Second boot: 18 seconds Third boot: 19 seconds systemd (apt-get remove sysvinit-core) First boot: 15 seconds Second boot: 16 seconds Third boot: 15 seconds
sysvinit averages 19 seconds, systemd averages 15.33 seconds. In this case it does appear that systemd booted the system faster.
Anecdote 2: Same as above but where the VM's disk is sitting wholly in RAM. Time for sysvinit dropped to 5 seconds and the time for systemd dropped to 4 seconds.
My personal guess is that the more you are running, the slower the disk the more likely systemd is to benefit you. You don't say how you did your comparison though or what type your "disks" were. If your comparison was between different versions of Linux distro then it could simply be that the previous version did less (which is always the fastest way to boot)...
Another anecdote: a few years back I saw Slackware systems at a University converted over to systemd. Boot times (which involved waiting for multiple NFS mounts) went from over three minutes to down to less than a minute because more of the waiting was done in parallel.
While my interaction with storage is "how long did it take for this operation to finish" the answer to that can vary dramatically on a hard disk depending on type and frequency of I/O I was doing (whereas the difference between best and worst values narrows with flash).
Your table doesn't capture the difference in wait time between reading only reading a 1GByte ISO in big chunks that's been laid out sequentially and serving up multiple torrents from big files that are over different parts of the disk simultaneously. You aren't always in the worst case and if the data is already cached in RAM you may be totally unable to perceive a difference because the waiting was decoupled from the request.
Telling people that "wait time" describes what they're seeing feels like it's trying to push latency and throughput into one number when both need to be distributions describing particular patterns of (potentially parallel) I/O and that's not even getting into queue depth. I guess there are parallels to telling people to describe things in terms of load average too...
While PCs continue to ship with legacy BIOS support you should be able to continue booting DOS on bare metal PCs (as you did). However FreeDOS does not yet support UEFI so if/when UEFI only machines come out the grand parent's challenge will become more difficult.
If you have an appropriate Android device Google Now will (apparently) display information based on your current context (e.g. if your phone learns where work and home are it might display information about traffic jams on the route home around the time it believes you will be traveling). You need a logged in Google account to use this feature.
OK Google is a way of using your voice to interact with your device (or Chrome web browser). So if I have the appropriate phone and it's been set to listen I can say "OK Google" and it will activate an app/mode where it will accept further voice input. On the Android phone I saw (and in my Chrome web browser on OS X) I can then ask it "What's the weather like?" and it pops up some weather related information and speaks back "It's ten degrees in ". Sometimes when you ask it questions just does a web search other times (on the device) it would start applications (e.g. mail) and so on. You do not need to be logged into Google to use this feature.
...but not without a price. If you can mathematically construct your program then you can prove that it is free from defects providing enough assumptions hold (the specification is correct, the tools used to build it are correct, the proof of correctness is correct, you had enough money/time/skill to do the process etc). For the time being, it's not possible to formally most programs that have already been written in mainstream languages so other techniques like testing will remain useful tools.
As an Anonymous Coward pointed out, it can be hard to detect runtime allocations overruns at compilation time. For these something like Clang's AddressSanitizer (GCC has added it too will help but at a cost of both time (slow down factor of 2) and space which is why you're unlikely to find it enabled on your precompiled SSH server binary. It's true there are cheaper checks (such as GCC's FORTIFY_SOURCE) that are less thorough/specialized that are often enabled by distros.
Sometimes you don't even need to change the file format - optimization can be applied to already compressed gzip/deflate files (which PNG uses) which can be used to create a more optimal deflate/gzip file. See tools like DeflOpt and defluff (DeflOpt can sometimes make even zopfli encoded files smaller).
Any Turing complete language can mimic any other Turing complete language (but at a price) so if your language supports condition driven loops you effectively have GOTO and IF. However if we see GOTO as syntactic sugar (and thus an efficiency optimisation/control flow obfuscator) wouldn't the combination of continuations and exceptions get you what GOTO can achieve?
Whether you consider this issue is hype depends on your answer to "if a tree falls in a forest and there's no one to observe it..." thought experiment.
The author of LZ4 has a summary with regards to LZ4 (both LZO and LZ4 are based on the LZ77 compression and both contained the same flaw) - that the issue has not been demonstrated as being exploitable in currently deployed programs due to their configuration (a rather angrier redacted original reply was originally posted). So at present this issue is severe but of low importance. If a way is found to exploit this problem on currently deployed popular programs without changing their configuration then this issue will also be of high importance but since this issue has now been patched hopefully newly deployed systems wouldn't be vulnerable.
Moreflicks lets you see what's available on multiple streaming services based on various "best of" lists (e.g. it's unlikely Netflix will ever tie in to the IMDB top 250 but Moreflicks does) and even has support for countries like the UK. It's sad to see an ecosystem like this being removed without replacement...
Slashdot Mirror
The solution is to use /dev/urandom, but only after verifying that the pool has some entropy. Ideally, it would be nice to have an API that allows you to find out how many total bits of entropy have been gathered by the system, regardless of how many remain in the pool at any given point in time. If the system has managed to accumulate a few hundred bits, just use /dev/urandom and get on with life. If it hasn't, use /dev/random and block.
The solution is to use /dev/urandom, but only after verifying that the pool has some entropy. Ideally, it would be nice to have an API that allows you to find out how many total bits of entropy have been gathered by the system, regardless of how many remain in the pool at any given point in time. If the system has managed to accumulate a few hundred bits, just use /dev/urandom and get on with life. If it hasn't, use /dev/random and block.
You could build what you are asking for by using the new (since v3.17 kernel) getrandom() syscall. See the part about emulating getentropy for determining if you've ever had up to 256 bits entropy in its man page for implementing your API suggestion...
This isn't so much about entropy "drying up" a few days after the system has booted - this is more about generating random numbers just after a system has booted and before "enough" entropy was gathered in the first place. From your link:
Not everything is perfect /dev/urandom happily gives you not-so-random numbers before the kernel even had the chance to gather entropy. When is that? At system start, booting the computer.
[...]
Linux's
but also from your link
FreeBSD does the right thing[...]. At startup /dev/random blocks once until enough starting entropy has been gathered. Then it won't block ever again.
[...]
On Linux it isn't too bad, because Linux distributions save some random numbers when booting up the system (but after they have gathered some entropy, since the startup script doesn't run immediately after switching on the machine) into a seed file that is read next time the machine is booting.
[...]
And it doesn't help you the very first time a machine is running, but the Linux distributions usually do the same saving into a seed file when running the installer. So that's mostly okay.
[...]
Virtual machines are the other problem. Because people like to clone them, or rewind them to a previously saved check point, this seed file doesn't help you.
So not great but not (always) a disaster and modern Linux allows programs to counter this if they wish by using getrandom.
Virtualization is a strong candidate because everything can be so samey but it can happen on real hardware too - imagine a trying to generate randomness on a basic MIPS based home router with flash based disks, no hardware RNG, typically booting from a fixed extract RAM disk install and doesn't have hardware clock to save time when powered off but makes ssh certs early during its first boot...
For the interested: Understanding-And-Managing-Entropy-Usage Whitepaper Black Hat whitepaper.
So it seems this is the classic problem that (Linux) programmers are told to use /dev/urandom (which never blocks) and some programs are doing so at system startup thus there's the opportunity for there to be "insufficient" randomness because not enough entropy has been gathered at that point in time. In short: using /dev/urandom is OK but if you are using it for security purposes you should only do it after /dev/random would have stopped blocking for a given amount of data for the first time since system startup (but there's no easy way to determine this on Linux). Or is there? Since the v3.17 kernel there is the getrandom syscall which has the beahviour that if /dev/urandom has never been "initialised" it will block (or can be made to fail right away by using flags). More about the introduction of the Linux getrandom syscall can be read on the always good LWN. And yes the BSD's had defences against this type situation first :-)
So this is bad for Linux systems that make security related "things" that depend on randomness early in startup but there may be mild mitigations in real life. If the security material is regenerated at a later point after boot there may be enough entropy around. If the the system is rebooted but preserves entropy from the last boot this may be mitigated for random material generated in subsequent boots (so long as the material was generated after the randomness was reseeded). If entropy preservation never takes place then regeneration won't help early boot programs. If the material based on the randomness is never regenerated then again this doesn't help. If you take a VM image and the entropy seed isn't reset then you've stymied yourself as the system believe it has entropy that it really doesn't.
and the[n] bought the farm in 2014
Aren't most things impossible after you've "bought the farm"?!
[...]
she got into dairy farming
I should have known you meant it literally...
First up lkml.org is a third party site that hosts Linux kernel mailing list archives on a website. Regular Linux kernel mail isn't actually sent from it (I believe that's done by vger) so we're looking up the email reputation for the wrong IP...
Secondly UCEPROTECT is a very aggressive blacklist which states upfront they will block people who they believe are in the vicinity of people who the judge to be sending them spam. It's not the be and end all though and on one server I looked some time ago it's effectiveness was surpassed by other blacklists (here's someone else's old DNS blacklist comparison for 2014). In general I prefer more conservative tools like senderbase when trying to work out an IPs mail reputation.
For what it's worth I've also seen GMail incorrectly mark mails sent to the fio mailing list (which is also managed by vger) as spam and in that case it was purely down to mail being proxied through the list which was a place that didn't match the sender's DMARC records. Most of the time GMail was getting the marking of spam right though (even for mailing list mails)...
The article now contains an update stating that queued TRIM was not involved (that ordinary TRIM was to blame is also mentioned in the author's Hackers News comment) only ordinary TRIM. It also appears that the company's drives were enterprise Samsung PROs.
Anecdotes 1 + 2 were running in native mode because it was initially a fresh install that was then switched sysvinit-core. Anecdote 3 I don't know (most likely compatability mode as it was several years ago). Even if all the anecdotes were only running in compatibility mode the results show systemd finishing quicker...
Does the above make things any better and how fast do you expect things to go when you're bottlenecked on I/O throughput? Is 15 seconds for a hard disk or 4(!) seconds for a RAM disk so bad?
Interesting... Googling around backs up your memory - systemd's readahead implementation was removed in v217.
The NFS mounts were always mounted at boot and not on demand both before and after systemd. The difference was that because there were multiple NFS mounts only proceeded in serial before the switch. After the switch the waiting of the mounts happened in parallel not just with each other but other services too. Yes it is entirely pathological but it really happened (and presumably other parallel inits would have solved it this way too).
For what it's worth I have an EeePC too. The default Xandros distro was a classic case of not running very much because it was highly custom (e.g. no printing, no mdns, kernel doesn't have to probe for all different kinds of hardware, hotplug of USB devices, io scheduler set to deadline etc) and thus was fast - I would always expect it to outperform a generic "full fat" Linux distribution. I'd expect your FreeBSD to beat my current XFCE Ubuntu setup too because I bet you can get start less. With lots of hand tweaks the boot speed to an XFCE desktop on my EeePC 900 is still 17 seconds...
Anecdote 1: I've just timed a Debian Jessie single CPU hard disk based VM install with BTRFS as the filesystem, a GNOME 3 desktop where the user is auto logged in boot and where an autostart script records the time. Here are my rough systemd and sysvinit results (times are from after the kernel core finished to when the GNOME script ran):
sysvinit (apt-get install sysvinit-core)
First boot: 20 seconds
Second boot: 18 seconds
Third boot: 19 seconds
systemd (apt-get remove sysvinit-core)
First boot: 15 seconds
Second boot: 16 seconds
Third boot: 15 seconds
sysvinit averages 19 seconds, systemd averages 15.33 seconds. In this case it does appear that systemd booted the system faster.
Anecdote 2: Same as above but where the VM's disk is sitting wholly in RAM. Time for sysvinit dropped to 5 seconds and the time for systemd dropped to 4 seconds.
My personal guess is that the more you are running, the slower the disk the more likely systemd is to benefit you. You don't say how you did your comparison though or what type your "disks" were. If your comparison was between different versions of Linux distro then it could simply be that the previous version did less (which is always the fastest way to boot)...
Another anecdote: a few years back I saw Slackware systems at a University converted over to systemd. Boot times (which involved waiting for multiple NFS mounts) went from over three minutes to down to less than a minute because more of the waiting was done in parallel.
While my interaction with storage is "how long did it take for this operation to finish" the answer to that can vary dramatically on a hard disk depending on type and frequency of I/O I was doing (whereas the difference between best and worst values narrows with flash).
Your table doesn't capture the difference in wait time between reading only reading a 1GByte ISO in big chunks that's been laid out sequentially and serving up multiple torrents from big files that are over different parts of the disk simultaneously. You aren't always in the worst case and if the data is already cached in RAM you may be totally unable to perceive a difference because the waiting was decoupled from the request.
Telling people that "wait time" describes what they're seeing feels like it's trying to push latency and throughput into one number when both need to be distributions describing particular patterns of (potentially parallel) I/O and that's not even getting into queue depth. I guess there are parallels to telling people to describe things in terms of load average too...
While PCs continue to ship with legacy BIOS support you should be able to continue booting DOS on bare metal PCs (as you did). However FreeDOS does not yet support UEFI so if/when UEFI only machines come out the grand parent's challenge will become more difficult.
Supposedly Chromecast support will be in VLC 3.0.
If I believe the data is going to already be mostly sorted why wouldn't I just use insertion sort?
If you have an appropriate Android device Google Now will (apparently) display information based on your current context (e.g. if your phone learns where work and home are it might display information about traffic jams on the route home around the time it believes you will be traveling). You need a logged in Google account to use this feature.
OK Google is a way of using your voice to interact with your device (or Chrome web browser). So if I have the appropriate phone and it's been set to listen I can say "OK Google" and it will activate an app/mode where it will accept further voice input. On the Android phone I saw (and in my Chrome web browser on OS X) I can then ask it "What's the weather like?" and it pops up some weather related information and speaks back "It's ten degrees in ". Sometimes when you ask it questions just does a web search other times (on the device) it would start applications (e.g. mail) and so on. You do not need to be logged into Google to use this feature.
...but not without a price. If you can mathematically construct your program then you can prove that it is free from defects providing enough assumptions hold (the specification is correct, the tools used to build it are correct, the proof of correctness is correct, you had enough money/time/skill to do the process etc). For the time being, it's not possible to formally most programs that have already been written in mainstream languages so other techniques like testing will remain useful tools.
From Eric Blake's bug-bash post
If you see anything like the following:
you're still vulnerable. There may be other issues the above does not cover.
You can already ask some compilers to do what you are asking - it's just often not on in shipped builds.
At compilation time warnings can be generated for out of bounds accesses that can be determined statically. Clang has -fsanitize=bounds, GCC has -Warray-bounds.
As an Anonymous Coward pointed out, it can be hard to detect runtime allocations overruns at compilation time. For these something like Clang's AddressSanitizer (GCC has added it too will help but at a cost of both time (slow down factor of 2) and space which is why you're unlikely to find it enabled on your precompiled SSH server binary. It's true there are cheaper checks (such as GCC's FORTIFY_SOURCE) that are less thorough/specialized that are often enabled by distros.
My understanding is that Apple have done the work to make it viable to use LLVM for certain levels of Javascript JITing so it is now feasible to use LLVM to compile long running dynamic code. Said code needs to be long running to a) build up information about the instructions being run b) offset the overhead of compilation.
ACT has a JPEG recompression test which clearly shows a bunch of compressors making a JPEG smaller. Even better - there's a great paper by the author of packJPG talking about how to compress a JPEG losslessly using the technique teppples described...
Sometimes you don't even need to change the file format - optimization can be applied to already compressed gzip/deflate files (which PNG uses) which can be used to create a more optimal deflate/gzip file. See tools like DeflOpt and defluff (DeflOpt can sometimes make even zopfli encoded files smaller).
Any Turing complete language can mimic any other Turing complete language (but at a price) so if your language supports condition driven loops you effectively have GOTO and IF. However if we see GOTO as syntactic sugar (and thus an efficiency optimisation/control flow obfuscator) wouldn't the combination of continuations and exceptions get you what GOTO can achieve?
Whether you consider this issue is hype depends on your answer to "if a tree falls in a forest and there's no one to observe it..." thought experiment.
The author of LZ4 has a summary with regards to LZ4 (both LZO and LZ4 are based on the LZ77 compression and both contained the same flaw) - that the issue has not been demonstrated as being exploitable in currently deployed programs due to their configuration (a rather angrier redacted original reply was originally posted). So at present this issue is severe but of low importance. If a way is found to exploit this problem on currently deployed popular programs without changing their configuration then this issue will also be of high importance but since this issue has now been patched hopefully newly deployed systems wouldn't be vulnerable.
Moreflicks lets you see what's available on multiple streaming services based on various "best of" lists (e.g. it's unlikely Netflix will ever tie in to the IMDB top 250 but Moreflicks does) and even has support for countries like the UK. It's sad to see an ecosystem like this being removed without replacement...