Unless your graphics driver provides a full 3D stack (userspace GL libraries down to kernel drivers) you will be using Mesa on Linux. You are probably thinking of Mesa as purely a software renderer whereas it is also used as a frontend to open source 3D drivers and uses DRI to provide access to the hardware's acceleration.
This is one of those questions that's going to keep being asked... Perhaps one day I'll be fast enough to get a first post on this that people actually read...
A few people have volumes of Knuth's Art of Programming on their shelves (but it's harder to find people who have read all of them).
One of the consultants who taught at my University said that the Mythical Man Month and Peopleware were good. I've read these too and can also recommended them (although they are more about managing programmers rather than programming per se). The consultant also recommended Design Patterns (although he said not to read the book cover to cover but rather to just be aware of them so you could refer to them later).
I've heard the "Dragon Book" (Compilers: Principles, Techniques, and Tools I think is the 2nd edition) being talked of favourably.
Many people seem to recommend reading Godel, Escher, Bach (I'd say it's about mathematical thinking)...
I've noticed which book answers tend to fall a bunch of categories:
Books that talk about software engineering/management/teams.
Books that talk about programming languages.
Books that talk about Computer Science.
Books that improve your mathematical thinking.
Books that programmers like but aren't programming/maths at all.
If you're going to ask someone "which book?" try limit the categories they should give you an answer for...
Vendor A2 closed source mobile - Bad. Buggy, vendor knows there are issues but doesn't fix them, driver limits performance forcing others to implement workarounds.
Vendor Q closed source mobile - Bad. Buggy, vendor is unresponsive to bug reports.
Vendor P closed source mobile - Unknown. Driver does not publicly support high enough version of OpenGL ES.
NVIDIA definitely write their own OSX drivers. I'm pretty sure AMD/ATI and Intel write their own OSX drivers too but these days GPU drivers are usually delivered with operating system updates (in a similar way that you can get driver updates through Windows update). Given how squeezing out GPU hardware documentation for Linux has been tough I don't think NVIDIA/AMD would be keen to help someone else write drivers that unlocked full functionality...
It's not entirely clear what you mean when you say "root exploit" but one interpretation is an exploit that when run as a regular user gives you administrator/root permissions. There have definitely been recent XP privilege escalations exploits for XP recently (e.g. CVE-2013-5065 leverages a bug in NDProxy).
If you are no longer able to keep your OS regularly patched it's no longer safe and you are better off using something else for online activities. Save XP for those appliances that have to use it and can be stringently firewalled/quarantined.
The "Can you recover overwritten data?" question was answered a few years ago in the paper Overwriting Hard Drive Data: The Great Wiping Controversy. The conclusion was with an electron microscope you could get 1 bit back but the chance of recovering more than that is negligible (and that is in the new barely used drive scenario).
The last question was basically "Can I use Capsicum to create a program that in turn isolates other arbitrary programs in a meaningful way (e.g. in the style of sandboxie)"?
Hey TR, thanks for the comprehensive replies (to be honest I thought I'd asked so late no one would see them) - you elaborated on things that I did not glean from the presentation. Well done for splitting secomp and secomp-bpf up too. I have a few more questions:
Does Capsicum only work at the process level? I can't have a more privileged thread that is still uncontained (i.e. still able to perform a blocked syscall) while other threads are contained?
How do you envision codebases supporting Capsicum in a way that they leaves them still portable to platforms where Capsicum is not available? Is it going to be a case of #ifdefs all the way down?
Would it be possible to make a sandbox program that uses Capsicum to in turn sandbox another (Capsicum-unaware) program that it goes on to run or is it likely going to be too restrictive for the second program?
To answer my own question here's what OS X and Windows do with system wide wifi passwords:
OS X stores the wifi password in the (encrypted) System keychain. The System keychain (System.keychain) is stored in a known location on disk and the material to decrypt it (SystemKey) is also stored in a known location on disk. The permissions on SystemKey file are set to be readable by only root.
What Windows does varies depending on version. For XP the wifi password is converted into a key and this key is stored directly in the registry unencrypted. For Vista and later the wifi password is encrypted (not turned into a key) with the System's Master Key and saved into XML file inside a known path on disk. To reverse this process offline, you need the particular decrypted Master Key used to encrypt the wifi password. Due to the way that Window's DPAPI works there may be many multiple Master Key's, one of which was the one actually used to encrypt the wifi password. All System Master Key's live under a well known path on disk but are encrypted. To decrypt a System Master Key, data from the SYSTEM and SECURITY registry hives has to be used. Permissions on the aforementioned registry hives and Master Keys is tight so even a "regular" Administrator cannot directly access the underlying files while the system is running and some of the files are marked as hidden (but this is by the by for an offline attack).
The issue of passwords being stored unencrypted on media has come up before with Android email passwords, Pidgin passwords and so on. If your attacker can bypass filesystem permissions you are already in a world of pain. One way to mitigate this would be to use a password protected keychain/keyring but this only works if you don't automatically unlock it...
Say that I want my Windows machine to automatically log in as a user when I turn it on. Because of the way Windows works it needs to be able to unlock my account (almost certainly to be able to unlock credential stores that would be otherwise locked), which means that when I enable Windows auto-login my password is going to be saved into the registry in plain text.
Perhaps Mac OS X can magically do better? Well not really - OS X XOR's your password with a fixed key and saves into/etc/kcpassword. For an attacker this is not a big hurdle over what Windows does. Unless your password is available OS X would be unable to unlock your keychain and all sorts of things would have to start prompting you if they wished to work.
If the keys to reverse the encryption are stored alongside the encrypted object you have not gained any more security but are just obfuscating your data - an attacker can simply steal both at the same time, run the decryption algorithm and use the object. To be secure you need to have something your attacker doesn't have access to which is at odds with unattended operation. If you want to have something happen completely unattended (i.e. from power on) fashion you are going to need ALL the information available in a directly usable form at some point and it's going to have to be "unprotected". While saving things like hashes are bit better (as they don't reveal the underlying password which may have been reused elsewhere) someone can still steal the hash and use it as is for accessing that service and in many cases a hash is no good as challenge response is being used to prevent the whole secret from having to be passed.
I do have one question though - what do OS X and Windows when you save things like WiFi/802.11x passwords that are accessible to every user? To what extent do they try and protect their system "keychains" and wouldn't such protection be obfuscation?
GMail will be fetching the images by default but only after the user opens the mail. So it's an improvement because the user's browser and IP address will be hidden (as it will be Google's servers doing the fetching) and it's a step back because it is tracking images will work by default. If you want the old behaviour of not showing images you will need to opt into it so only those who explicitly don't want to be tracked will remain anonymous.
The Windows driver also had much larger spikes in the frame latency than the Intel Linux driver.
That sounds true looking at the summary statistics of the graph but the question is at what frequency and by how much when compared to Linux on the same frames? We know the Linux statistics have at least one peak (of 45) but none are visible on that graph because the Windows values have been drawn over the top of of the other values. It's a bad visualisation of the data and it's hard to learn all that much from it...
Do you remember the days when Linux didn't perform well in this area ?
In all honesty I don't (I remember tearing being more common which is a slightly different issue) which isn't to say it wasn't the case - I'd be grateful for sources which show what it used to be and what it is now with the same hardware. From what little I know, being able to measure OpenGL frame latency has only been possible for a few years on Linux...
The headline is bad/misleading - many of those benchmarks are showing a disparity of more than 10% between the drivers. Using the numbers from the Phoronix article, Linux results are the highest number from any Linux driver (there are many cases where the most recent driver was not the best) to try and prove headline:
Linux = [35.88, 140.90, 43.37, 23.5, 32.23, 19.17, 25.17, 16.68, 99.24, 63.94, 46.80, 29.46] Windows = [41.47, 162.88, 36.57, 27.0, 31.46, 19.37, 24.47, 16.85, 104.04, 65.15, 55.05, 36.63] for i in range(len(Linux)):
diff = abs(round((1 - Linux[i]/Windows[i])*100, 1))
"Windows win by %d.1%%" % (diff) if Linux[i] < Windows[i] else "Linux . win by %d.1%%" % (diff)
'Windows win by 13.1%' 'Windows win by 13.1%' 'Linux . win by 18.1%' 'Windows win by 13.1%' 'Linux . win by 2.1%' 'Windows win by 1.1%' 'Linux . win by 2.1%' 'Windows win by 1.1%' 'Windows win by 4.1%' 'Windows win by 1.1%' 'Windows win by 15.1%' 'Windows win by 19.1%'
So out of 12 results, 5 showed a 10%+ difference between Linux and Windows Intel drivers in favour of Windows and 1 showed a 10%+ difference in favour of Linux. The conclusion that the drivers are neck and neck does not follow from the premise for around 40% of the results and that's when being unfairly generous to Linux!
on both a real Pi and a whole machine Qemu pi on a 2.5Ghz Intel machine. The real Pi was 3-4 seconds faster. A Qemu Pi may be faster if you didn't emulate the whole machine and just translated the binary though...
Both upstart and systemd will support unmodified initscripts. It can make your boot slower but it preserves backwards compatibility so your wish has been already been granted (albeit in reverse)!
You can still use startx even under systemd and upstart (e.g. if you boot to runlevel 3 (which is also emulated)) assuming your setup has provided it.
I think I've only ever seen a boot that took ten minutes (or longer) in the case that fsck had to run to check a disk and I'm not counting that. I'm also not including time spent in the BIOS. I guess I should have said "a few minutes" to be clearer though.
These were good admins, there were a lot of NFS3/4 shares and it was part of a Kerberos domain but maybe your setup is better tuned. I don't think it was misconfiguration and seeing how things sped up so dramatically after the change... But if it's that hard to get it right with initscripts that's another nail in their coffin in my book.
Human readable initscripts can be far more complicated than systemd units but it depends massively on the task at hand. Some initscripts can be horrific - the Debian Apache initscript is not simple (but it does a lot so...).
How often do I reboot *nix boxes? Varies massively from "only when glibc and kernel changes are released" (so a few times a month tops) to a few times a day. In my previous example many of the machines were in computer labs so they would shut themselves down every day. When you are watching these machines boot up at start of the day you really notice these things. 40 machines just sitting there with text sporadically appearing just to get to a greeter which Windows 7 reached in less than half the time until the systemd switch wasting students tuition fees in practicals - bleh. I even notice boot times on my VM's at work these days - when I spin up 10 Linux VMs from scratch it's nice seeing them all ready in under 40 seconds. Even on my creaky EeePC it's nice being able to get to the desktop from cold in less than 20 seconds.
I'm happy work has been put into improving boot speed over the years - it really has become much better than it was back in 2001 even though more is being done. Stuff like systemd is even solving some long standing problems which used to require hacks like portreserve so it's not just speed.
At a former university the desktops used Slackware with initscripts all the way up until 2011. These desktops were beefy machines with decent chunks of RAM and SATA disks but would take minutes to finish booting. The administrators there switched the start-up scripts to systemd across a holiday and things became dramatically better - no boot took more than 30 seconds and most disk based boots were sub 15 seconds making the old system look laughable.
I'm glad that Slackware gave the admins the choice because the old way of doing it has had its day. Not everything from the 90s was better...
At a former university the Linux side of the desktops used Slackware with initscripts all the way up until 2011. These desktops were beefy machines with decent chunks of RAM and SATA disks but would take minutes to finish booting. The administrators there switched the start-up scripts to systemd over the holidays and things were so much better - no boot took more than 30 seconds to finish (most took less than 15 seconds) making the old system look laughable.
SysV initscripts have had their day and it's time to move on.
So it serves up numbers to humans - does this mean that only computer-hard captchas are going to help reading books?
Further it knows you're a computer/human already but gives a test to reaffirm this anyway? Seems wasteful but I guess it acts as a safety net and allows better classification in the future...
I can well believe there are differences in battery performance between the OSes but we need someone to sit down with programs performing exactly the same operations on base configurations of all the OSes and then report the results. Saying they vary is one thing but far more interesting is to know why. Is it the drivers, is it the scheduler, is it the kernel, is it a better userspace or is it some combination of all of them?
My understanding is that both Linux and Windows supported timer coalescing before OS X. Linux had a tickless kernel. OS X's XNU kernel is allegedly tickless but I can't find out when this change actually took place. As for Windows it's not clear - my understanding is that Windows 8 is tickless but I can't find a clear reference only one that says Windows 8 idles more than Windows 7 so perhaps it has dynamic ticks and hence can be tickless. That last link seems to suggest that Microsoft have put a large amount of effort into trying to make Windows more battery friendly...
One of the things I noticed on OS X 10.8 though is that when the battery is near to depletion it seems to force the CPU to run at a slower rate until the machine power goes out completely. I don't know the other OSes do this or whether it's a positive impact but it could impact on results that purely go on time rather than amount of work done.
Slashdot Mirror
Unless your graphics driver provides a full 3D stack (userspace GL libraries down to kernel drivers) you will be using Mesa on Linux. You are probably thinking of Mesa as purely a software renderer whereas it is also used as a frontend to open source 3D drivers and uses DRI to provide access to the hardware's acceleration.
I've yet to see binary any drivers use Mesa.
This is one of those questions that's going to keep being asked... Perhaps one day I'll be fast enough to get a first post on this that people actually read...
Link summary from last time:
General comments
I've noticed which book answers tend to fall a bunch of categories:
If you're going to ask someone "which book?" try limit the categories they should give you an answer for...
There's a comment at the bottom of the article by David Poole that links to a post talking about OpenGL driver quality on desktop Linux and mobile Linux. The summary from that blog post is:
NVIDIA definitely write their own OSX drivers. I'm pretty sure AMD/ATI and Intel write their own OSX drivers too but these days GPU drivers are usually delivered with operating system updates (in a similar way that you can get driver updates through Windows update). Given how squeezing out GPU hardware documentation for Linux has been tough I don't think NVIDIA/AMD would be keen to help someone else write drivers that unlocked full functionality...
It's not entirely clear what you mean when you say "root exploit" but one interpretation is an exploit that when run as a regular user gives you administrator/root permissions. There have definitely been recent XP privilege escalations exploits for XP recently (e.g. CVE-2013-5065 leverages a bug in NDProxy).
Perhaps you meant "remote exploit" but also last year there was CVE-2013-3175 malformed asynchronous RPC request so another machine can attack your XP machine over the network with no user intervention. See this table of 2013 Windows XP CVE entries for a list of what MS have been patching...
If you are no longer able to keep your OS regularly patched it's no longer safe and you are better off using something else for online activities. Save XP for those appliances that have to use it and can be stringently firewalled/quarantined.
The "Can you recover overwritten data?" question was answered a few years ago in the paper Overwriting Hard Drive Data: The Great Wiping Controversy. The conclusion was with an electron microscope you could get 1 bit back but the chance of recovering more than that is negligible (and that is in the new barely used drive scenario).
The last question was basically "Can I use Capsicum to create a program that in turn isolates other arbitrary programs in a meaningful way (e.g. in the style of sandboxie)"?
Hey TR, thanks for the comprehensive replies (to be honest I thought I'd asked so late no one would see them) - you elaborated on things that I did not glean from the presentation. Well done for splitting secomp and secomp-bpf up too. I have a few more questions:
To answer my own question here's what OS X and Windows do with system wide wifi passwords:
OS X stores the wifi password in the (encrypted) System keychain. The System keychain (System.keychain) is stored in a known location on disk and the material to decrypt it (SystemKey) is also stored in a known location on disk. The permissions on SystemKey file are set to be readable by only root.
What Windows does varies depending on version. For XP the wifi password is converted into a key and this key is stored directly in the registry unencrypted. For Vista and later the wifi password is encrypted (not turned into a key) with the System's Master Key and saved into XML file inside a known path on disk. To reverse this process offline, you need the particular decrypted Master Key used to encrypt the wifi password. Due to the way that Window's DPAPI works there may be many multiple Master Key's, one of which was the one actually used to encrypt the wifi password. All System Master Key's live under a well known path on disk but are encrypted. To decrypt a System Master Key, data from the SYSTEM and SECURITY registry hives has to be used. Permissions on the aforementioned registry hives and Master Keys is tight so even a "regular" Administrator cannot directly access the underlying files while the system is running and some of the files are marked as hidden (but this is by the by for an offline attack).
The issue of passwords being stored unencrypted on media has come up before with Android email passwords, Pidgin passwords and so on. If your attacker can bypass filesystem permissions you are already in a world of pain. One way to mitigate this would be to use a password protected keychain/keyring but this only works if you don't automatically unlock it...
Say that I want my Windows machine to automatically log in as a user when I turn it on. Because of the way Windows works it needs to be able to unlock my account (almost certainly to be able to unlock credential stores that would be otherwise locked), which means that when I enable Windows auto-login my password is going to be saved into the registry in plain text.
Perhaps Mac OS X can magically do better? Well not really - OS X XOR's your password with a fixed key and saves into /etc/kcpassword. For an attacker this is not a big hurdle over what Windows does. Unless your password is available OS X would be unable to unlock your keychain and all sorts of things would have to start prompting you if they wished to work.
If the keys to reverse the encryption are stored alongside the encrypted object you have not gained any more security but are just obfuscating your data - an attacker can simply steal both at the same time, run the decryption algorithm and use the object. To be secure you need to have something your attacker doesn't have access to which is at odds with unattended operation. If you want to have something happen completely unattended (i.e. from power on) fashion you are going to need ALL the information available in a directly usable form at some point and it's going to have to be "unprotected". While saving things like hashes are bit better (as they don't reveal the underlying password which may have been reused elsewhere) someone can still steal the hash and use it as is for accessing that service and in many cases a hash is no good as challenge response is being used to prevent the whole secret from having to be passed.
I do have one question though - what do OS X and Windows when you save things like WiFi/802.11x passwords that are accessible to every user? To what extent do they try and protect their system "keychains" and wouldn't such protection be obfuscation?
GMail will be fetching the images by default but only after the user opens the mail. So it's an improvement because the user's browser and IP address will be hidden (as it will be Google's servers doing the fetching) and it's a step back because it is tracking images will work by default. If you want the old behaviour of not showing images you will need to opt into it so only those who explicitly don't want to be tracked will remain anonymous.
Sources: Wired, Ars Technica
The Windows driver also had much larger spikes in the frame latency than the Intel Linux driver.
That sounds true looking at the summary statistics of the graph but the question is at what frequency and by how much when compared to Linux on the same frames? We know the Linux statistics have at least one peak (of 45) but none are visible on that graph because the Windows values have been drawn over the top of of the other values. It's a bad visualisation of the data and it's hard to learn all that much from it...
Do you remember the days when Linux didn't perform well in this area ?
In all honesty I don't (I remember tearing being more common which is a slightly different issue) which isn't to say it wasn't the case - I'd be grateful for sources which show what it used to be and what it is now with the same hardware. From what little I know, being able to measure OpenGL frame latency has only been possible for a few years on Linux...
The headline is bad/misleading - many of those benchmarks are showing a disparity of more than 10% between the drivers. Using the numbers from the Phoronix article, Linux results are the highest number from any Linux driver (there are many cases where the most recent driver was not the best) to try and prove headline:
Linux = [35.88, 140.90, 43.37, 23.5, 32.23, 19.17, 25.17, 16.68, 99.24, 63.94, 46.80, 29.46]
Windows = [41.47, 162.88, 36.57, 27.0, 31.46, 19.37, 24.47, 16.85, 104.04, 65.15, 55.05, 36.63]
for i in range(len(Linux)):
diff = abs(round((1 - Linux[i]/Windows[i])*100, 1))
"Windows win by %d.1%%" % (diff) if Linux[i] < Windows[i] else "Linux . win by %d.1%%" % (diff)
'Windows win by 13.1%'
'Windows win by 13.1%'
'Linux . win by 18.1%'
'Windows win by 13.1%'
'Linux . win by 2.1%'
'Windows win by 1.1%'
'Linux . win by 2.1%'
'Windows win by 1.1%'
'Windows win by 4.1%'
'Windows win by 1.1%'
'Windows win by 15.1%'
'Windows win by 19.1%'
So out of 12 results, 5 showed a 10%+ difference between Linux and Windows Intel drivers in favour of Windows and 1 showed a 10%+ difference in favour of Linux. The conclusion that the drivers are neck and neck does not follow from the premise for around 40% of the results and that's when being unfairly generous to Linux!
I was about dispute an earlier post but then ran
time factor 18014398777917439
on both a real Pi and a whole machine Qemu pi on a 2.5Ghz Intel machine. The real Pi was 3-4 seconds faster. A Qemu Pi may be faster if you didn't emulate the whole machine and just translated the binary though...
Both upstart and systemd will support unmodified initscripts. It can make your boot slower but it preserves backwards compatibility so your wish has been already been granted (albeit in reverse)!
You can still use startx even under systemd and upstart (e.g. if you boot to runlevel 3 (which is also emulated)) assuming your setup has provided it.
I think I've only ever seen a boot that took ten minutes (or longer) in the case that fsck had to run to check a disk and I'm not counting that. I'm also not including time spent in the BIOS. I guess I should have said "a few minutes" to be clearer though.
These were good admins, there were a lot of NFS3/4 shares and it was part of a Kerberos domain but maybe your setup is better tuned. I don't think it was misconfiguration and seeing how things sped up so dramatically after the change... But if it's that hard to get it right with initscripts that's another nail in their coffin in my book.
Human readable initscripts can be far more complicated than systemd units but it depends massively on the task at hand. Some initscripts can be horrific - the Debian Apache initscript is not simple (but it does a lot so...).
How often do I reboot *nix boxes? Varies massively from "only when glibc and kernel changes are released" (so a few times a month tops) to a few times a day. In my previous example many of the machines were in computer labs so they would shut themselves down every day. When you are watching these machines boot up at start of the day you really notice these things. 40 machines just sitting there with text sporadically appearing just to get to a greeter which Windows 7 reached in less than half the time until the systemd switch wasting students tuition fees in practicals - bleh. I even notice boot times on my VM's at work these days - when I spin up 10 Linux VMs from scratch it's nice seeing them all ready in under 40 seconds. Even on my creaky EeePC it's nice being able to get to the desktop from cold in less than 20 seconds.
I'm happy work has been put into improving boot speed over the years - it really has become much better than it was back in 2001 even though more is being done. Stuff like systemd is even solving some long standing problems which used to require hacks like portreserve so it's not just speed.
At a former university the desktops used Slackware with initscripts all the way up until 2011. These desktops were beefy machines with decent chunks of RAM and SATA disks but would take minutes to finish booting. The administrators there switched the start-up scripts to systemd across a holiday and things became dramatically better - no boot took more than 30 seconds and most disk based boots were sub 15 seconds making the old system look laughable.
I'm glad that Slackware gave the admins the choice because the old way of doing it has had its day. Not everything from the 90s was better...
At a former university the Linux side of the desktops used Slackware with initscripts all the way up until 2011. These desktops were beefy machines with decent chunks of RAM and SATA disks but would take minutes to finish booting. The administrators there switched the start-up scripts to systemd over the holidays and things were so much better - no boot took more than 30 seconds to finish (most took less than 15 seconds) making the old system look laughable.
SysV initscripts have had their day and it's time to move on.
...and don't people sometimes malfunction while behind the wheel too?
Obligitory XKCD 538 link.
So it serves up numbers to humans - does this mean that only computer-hard captchas are going to help reading books?
Further it knows you're a computer/human already but gives a test to reaffirm this anyway? Seems wasteful but I guess it acts as a safety net and allows better classification in the future...
I can well believe there are differences in battery performance between the OSes but we need someone to sit down with programs performing exactly the same operations on base configurations of all the OSes and then report the results. Saying they vary is one thing but far more interesting is to know why. Is it the drivers, is it the scheduler, is it the kernel, is it a better userspace or is it some combination of all of them?
My understanding is that both Linux and Windows supported timer coalescing before OS X. Linux had a tickless kernel. OS X's XNU kernel is allegedly tickless but I can't find out when this change actually took place. As for Windows it's not clear - my understanding is that Windows 8 is tickless but I can't find a clear reference only one that says Windows 8 idles more than Windows 7 so perhaps it has dynamic ticks and hence can be tickless. That last link seems to suggest that Microsoft have put a large amount of effort into trying to make Windows more battery friendly...
In addition to the above, Windows has a huge number of energy saving features: Idle detection that can control things like what processes are allowed to start, Windows 8 store apps use a "only focussed app runs" model unless it's a background task, USB suspend (Windows 7), adjustable tick rate (Windows 2000) (Windows seems to suffer from programs that push for higher resolution ticks though). It would be nice to know whether all these things are having an impact.
One of the things I noticed on OS X 10.8 though is that when the battery is near to depletion it seems to force the CPU to run at a slower rate until the machine power goes out completely. I don't know the other OSes do this or whether it's a positive impact but it could impact on results that purely go on time rather than amount of work done.
Additionally isn't Google a Linux vendor these days? Seems a bit disingenuous to still say
no Commercial Unix or Linux vendor has ever given our project [openssh] a cent