Debian Working on Reproducible Builds To Make Binaries Trustable

An anonymous reader writes: Debian's Jérémy Bobbio, also known as Lunar, spoke at the Chaos Communication Camp about the distribution's efforts to reassert trustworthiness for open source binaries after it was brought into question by various intelligence agencies. Debian is "working to bring reproducible builds to all of its more than 22,000 software packages," and is pushing the rest of the community to do the same. Lunar said, "The idea is to get reasonable confidence that a given binary was indeed produced by the source. We want anyone to be able to produce identical binaries from a given source (PDF)."

Here is Lunar's overview of how this works: "First you need to get the build to output the same bytes for a given version. But others also must to be able to set up a close enough build environment with similar enough software to perform the build. And for them to set it up, this environment needs to be specified somehow. Finally, you need to think about how rebuilds are performed and how the results are checked."

Comments

Seems like a little random build size

[Revek]

Would make it harder for them to exploit.

Re:Seems like a little random build size

[Desler]

But that isn't the point of this. It's how to verify that your binary doesn't have tampered with source code.

Re:Seems like a little random build size

[cold fjord]

That's a tricky problem.

Countering "Trusting Trust"

Re:Seems like a little random build size

[Desler]

Yes it is difficult. That's why they are trying to solve the problem.

Re:Seems like a little random build size

[MyAlternateID]

But that isn't the point of this. It's how to verify that your binary doesn't have tampered with source code.

I care about this, too. That's one reason I run a source-based distribution. It's not the only reason. It's not even the main reason. But it's one reason.

Anyone who really needs this kind of assurance was probably also building from source. You can do it once on-site, then make your own binary packages and push those to all of your other machines so it's really not bad. I think a much more insidious threat comes from malicious yet innocent-looking source, like what you find in the Underhanded C Contests.

It doesn't do much good to have a reproducible build of a program when it contains an innocent-looking yet malicious piece of code. Just consider Heartbleed. Whether Heartbleed was intentional or not, it proves that people can run vulnerable code for a very long time before it's found out, and that was a program intended to be secure.

Re:Seems like a little random build size

[Bing Tsher E]

I wonder, though, if an always-connected build machine could have compromised object files pushed onto it mid-build. It's a theoretical risk, but not one that couldn't be accomplished by a determined foe. The build process is well characterized for many projects, knowing when to push a rogue object file into the build directory wouldn't be that difficult.

It means the penetrating entity would need to already have access to your system, but 'object pushing' would be a useful technique for escalating security breaches.

Re:trust already lost in Debian

[iggymanz]

you'd have to how the Debian internal politics have changed, there are articles about that and also about how it relates to systemd. even slashdot had one

This seems like a job for Virtual Box

[goombah99]

Wouldn't virtual box be able to do this? Then one only has the trouble of validating the virtual box. While that is actually probably harder to do you don't have to do this as often and one could use some open source virtual box equivalent and compile that from source.

On the otherhand I don't quite understand why, if one can compile the source, one needs to worry about untrusted binaries. Perhaps the intent here is for some master agency to watch for tinkered binaries or to post it's own Checksums apart from Debian. Then everyone has two sources for validated checksums.

Re:This seems like a job for Virtual Box

[wolrahnaes]

On the otherhand I don't quite understand why, if one can compile the source, one needs to worry about untrusted binaries. Perhaps the intent here is for some master agency to watch for tinkered binaries or to post it's own Checksums apart from Debian. Then everyone has two sources for validated checksums.

Almost right, except without the master agency. This isn't for the incredibly paranoid types who would already be compiling from source. This is for the rest of us, the lazy people who would rather "apt-get install foo" and just assume the distro's doing things right. If the builds are reproducible then eventually someone's going to verify them. If no variations are discovered, the rest of us lazy masses can be a lot more confident that we're not running anything unexpected.

Re:This seems like a job for Virtual Box

[Anonymous Coward]

From the article the issue was that the cia had found a way to own the *compiler* binaries and each program it compiled would have a vulnerability added at build time.

Build timestamps mess this up

[savuporo]

Unless you freeze system time for the full duration of the build, every piece of code that builds in __TIME__ or __DATE__ macros, will screw with this. Other environment macros injected into the build ( like git revision etc ) as well.

Re:Build timestamps mess this up

[wolrahnaes]

Pages 6 and 7 of the PDF linked cover time-related issues and basically agree, anything that builds time/date in to the binary is a problem that needs to be fixed.

Git revision on the other hand is a recommended solution, since it points at a specific state of the code and will always be the same if the code is unchanged.

Re:Build timestamps mess this up

[Ungrounded Lightning]

Unless you freeze system time for the full duration of the build, every piece of code that builds in __TIME__ or __DATE__ macros, will screw with this.

And I have run into this at my current employment, when checking that I had successfully selected the correct version of an archived source and reproduced the binary build. The source apparently had an instance of __DATE__ in it, and would compile differently on different days.

But the datestamps - at least in the tools I was using - were always the same length (so they didn't move other stuff around as they changed) and easily recognized. I could easily verify that the only differences were the datestamp.

Seems to me that it would be trivial to do a diff-like utility (or add yet another option to diff) that would determine whether files were identical EXCEPT for instances of __TIME__ and __DATE__ expansions - and report both the otherwise-match and the number and values of the date/time stamps.

Re: Build timestamps mess this up

[savuporo]

__FILE__ has a similar effect, as it refers to absolute path. Absolute build paths would also be have to be created deterministically, no temp directories or anything like that. And then, make sure everything that gets linked in statically, including the toolchain bits follow the same rules.

Re:Build timestamps mess this up

[Eunuchswear]

Why would a lot of code need to be "fixed" just because someone anally retentive wants deterministic builds?

Uh, because some anally retentive person wants deterministic builds? Did you RTFS?

Re:Build timestamps mess this up

[savuporo]

The other thing is to hack your build of the toolchain, so that __TIME__ and __DATE__ and __FILE__ could be stubbed and/or overridden by command line. Havent looked at GCC or clang codebases, but i would think it wouldnt be too hard.

Re:Build timestamps mess this up

[Forever Wondering]

When I needed to compare binaries, I wrote a script that would clean the source of __DATE__/__TIME__ and RCS/CVS style $Id stuff.

While the $Id was okay for source comments, it was common practice to add something like "static char *rcs_id = "$Id";" to each .c file in such a way that you'd get a bunch of these in the final binary.

This script can be run recursively on a copy of the source tree. Or, it can be done "on the fly" by having the script masquerade as gcc. Then, do the build.

This works a bit better than a special binary diff util because the $Id strings are variable length and could change the offsets [slightly] within many generated instructions.

RCS/CVS also had $Log for comments which pulled in the entire commit log into a block comment. This made even a simple recursive diff on the source trees of two different versions be messy.

Re: Build timestamps mess this up

[wolrahnaes]

I know this is Slashdot and all, but you really should RTFA. Again that's covered and a variety of solutions are offered, but you are basically right in that doing this right requires that those things all be the same where they're used.

The tricky part here is determining in which cases those sorts of macros are actually required and thus must be worked around versus where they can be replaced with something else to achieve the same goal (replacing time/datestamped builds with git commit IDs for example) versus where they were just pointless (embedding the hostname of the build machine in to the binary for example).

Re:Build timestamps mess this up

[wolrahnaes]

Why would a lot of code need to be "fixed" just because someone anally retentive wants deterministic builds? If they truly care they can LD_PRELOAD fake date/time libs.

The reason for deterministic builds is to allow those of us who want to use binaries from our distros for convenience sake verify that the binary is actually built from the source it claims to be from. It only takes a few people actually doing it to confirm things are good for all of us.

Basically it lets the lazy masses gain the same level of confidence in what they're running as those who compile everything from source.

I thought this problem was solved a long time ago by the bitcoin developers w/ gitian.

Bitcoin solved it as far as they needed for their own purposes, this project aims to solve it sufficiently to be generally applicable across the entire Debian operating system. The goal is that the entire Debian binary package repository could be audited by anyone who cares to do it. Obviously if it's generic enough to cover all of Debian the same techniques should be usable pretty much across the entire computing world.

Awesome

[trawg]

I was thinking about this being a problem a while back - how to deal with building something from source and knowing I was getting the same output that the developers wanted me to have. Coincidentally about the same time, a href="http://developers.slashdot.org/story/13/06/20/1548228/are-you-sure-this-is-the-source-code">this article popped on Slashdot and introduced me to Ken Thompson's article Reflections on Trusting Trust - a great read and something that really opened my eyes (in that wide-open-because-of-terror kind of way).

Also from that thread came this email from one of the Tor developers talking about their deterministic build process to do the same thing.

I think this is a problem that would be really great to solve as soon as possible. I very much hope that once we start seeing more reproducible builds we don't suddenly find out that certain compilers have been compromised long ago.

Re:Awesome

[Forever Wondering]

The problem solution can get messy.

Most packaging systems have a control file for each package that specifies dependencies for other packages with versions, conflicts, etc. They specify deps for stuff they care about (e.g. gtk3 version X needs cairo version Y) but they don't always specify the version of gcc et. al. that they need, because that's not important from their perspective. That is, they're happy to build with gcc 4.0.0 or 4.1.0 or whatever. Sometimes the deps are specified as "I need package X with version >= 2.6"

To be reproducible, these packages would need to spell out exact versions for the build tools and possibly/probably the exact version for any given package it depends on. It's a bit inflexible to burden an individual package with specifying an exact version dependency [and can break a lot of stuff], so generating an "uber" version/dependency file during the "make world" seems like the sanest approach.

But, that may not work either. When distro major releases are done, they can do "make world" and get the exact versions for all packages. But, sometime later, a package may be updated to a new version. For example, it's a forward/backward compatible bug fix to a shared library, so application program packages that depend on it (via rev >= X) don't need to be rebuilt.

Also, if a given package has a dependency like "gcc >= 4.0.0" [because it needs a given feature that was added in 4.0.0] may work fine for a while. But, later the package control file may have to be changed to "gcc >= 4.0.0 && gcc " need to be rebuilt, just to verify that the stat.h change builds the same binary? Further, maybe some apps may be updated to use this new define. It would be easy for a programmer to see that merely adding a #define is harmless and backwards compatible. That is, the smart way is use the latest stat.h even for older programs as they will build the same binary regardless of which version is used. But, a validation would have a hard time determining this and would have to rely on package version numbers.

Existing package systems do have a fair number of extra fields to help with such a choice (e.g. "I'm version x.y.1 and I'm backwards compatible with x.y.0"). But, can/should a validation system rely on this? IMO, it must disregard this and still regen the package and do the compare to detect mistakes in the package control file. That is, the only proof is the final binary comparisons, regardless of what the control file says.

Now comes the fun part ...

Naively, the validation system could loop through all packages and rebuild a container, virtual machine, or even just a simple chroot target directory with the correct dependent package versions for the given package that you wish to validate. Most of the the time would be spent simply extracting the packages in order to get the canonical environment for the build/verify operation.

Far better would be for the validation system to build the container initially and just change dependent packages as needed. But, this would require the validator to do global analysis of all of the dependency graph and choose the best way. For example, you'd like to install gcc 5.1.0 and verify most packages, deferring those that need the newer 5.1.1 until the end. Flipping back and forth would be wasteful and it's desirable to avoid this "thrashing".

But, such global analysis requires a satisfiability solver of some sophistication (e.g. libsolv) or you end up with massive amounts of cycles consumed.

As to disk space ... I've been using fedora and do "reposync" so I have both binary and source rpms. For a given fedora version (e.g. fc21), this runs to ~200GB for everything, which is what you'd need on each system if you wanted to "crowd source" the validation effort.

Re:Soon to be assimilated

[Anonymous Coward]

At least systemd doesn't have maintainers that just randomly comment out lines of code in security software based on unverified, false positives from a static analyzer.

No but it does have a bunch of assholes determined to shove it down our throats no matter what, using all sorts of politics and other "not based on technical merit" tactics to make this happen. If I ever run systemd (not likely, but possible) it sure as hell won't be because somebody else decided I should. If I wanted somebody else to decide what was on my system, I would run a commercial OS and be done with it.

I find the Wiki page on systemd to be exquisitely ironic. In the first paragraph or two it says "The name systemd adheres to the Unix convention of making daemons easier to distinguish by having the letter d as the last letter of the filename." Well that's great! At least there is one single thing about systemd that can be said to follow the Unix way of doing things. And here I was thinking there were none at all.

Awesome - on trusting trust

[trawg]

I was thinking about this being a problem a while back - how to deal with building something from source and knowing I was getting the same output that the developers wanted me to have. Coincidentally about the same time, this article popped on Slashdot and introduced me to Ken Thompson's article Reflections on Trusting Trust - a great read and something that really opened my eyes (in that wide-open-because-of-terror kind of way).

Also from that thread came this email from one of the Tor developers talking about their deterministic build process to do the same thing.

I think this is a problem that would be really great to solve as soon as possible. I very much hope that once we start seeing more reproducible builds we don't suddenly find out that certain compilers have been compromised long ago.

Re:Awesome - on trusting trust

[Ungrounded Lightning]

The binaries emitted are more or less guaranteed to change every time the compiler is updated.

Sure.

So you compile it with the (perhaps out-of-date) toolset, getting a match to the distributed binaries, then again with the latest-and-greatest toolset if you want to use its output rather than the standard one (and take your own chances that your new compiler revision was compromised.)

The problem being addressed is making sure that your own build environment contains the same source and makes the same binaries as the distribution version, so you know that you have uncorrupted source and an uncorrupted toolset (or as close to it as you can reasonably expect to get - which is pretty close, given that you can apply the same techniques to the tools themselves). Then you can join, and take advantage of, the large "group of eyeballs" checking the source (and build system) for malware inclusions.

Re:trust already lost in Debian

[i.r.id10t]

So... what to do then? What other distribution works well, has the large amount of packages available, is freely available (not a big deal since RH isn't going to be systemd free), and pretty much Just Works?

I Don't Get It

[jmac_the_man]

What does this solution provide that checksums do not? If you trust Debian's repositories, and they publish checksums and build sizes, you can independently verify that the package you downloaded is the one they published. As an additional level of security, they can use signed binaries, where they encrypt the binaries with a private key, and then, if you trust Debian's repositories, you decrypt their binaries with their public key before you install them. IIRC, Debian already uses both of these.

I mean, clearly the Debian guys have thought of this, so what am I missing?

Re:I Don't Get It

[jopsen]

validation of source to binary step... This is so you don't have to trust their binary... but can read their source and trust that it is in fact the source for the binaries..

Yes, but can you trust your compiler tool chain?

[laing]

This only works if Debian can guarantee the integrity of the development tool chain. See this >30 year old talk/paper by Ken Thompson describing the problem. Once inserted, the malware is persistent and invisible. Re-compiling your compiler and applications from known-good versions doesn't help.

Re:trust already lost in Debian

[Bing Tsher E]

NetBSD has the pkgsrc collection, which is fairly large, and it is never going to be polluted by systemd.

Compromised hardware

[fabrica64]

What about compromised CPUs? If you are the NSA I think it's easier to build a backdoor into the CPU than try to keep up with ever changing software builds. Isn't it? CPUs are totally controlled by three or four U.S. companies, are closed source nobody has ever seen into it...

Re:Compromised hardware

[caseih]

A partial answer to this is to build your own CPU and system in software. Like Bochs. But you could build this virtual system on any number of other completely incompatible platforms for verification. Would be slow. But at least it would be consistent and verifiable. You couldn't use hardware virtualization for this. Would have to be completely implemented in software. And if different people implemented the same reference platform independently (using their own preferred language and programming techniques) that would add an additional layer of verification. Even the deepest NSA compromise would have a hard time completely influencing this.

Re:Compromised hardware

[dwheeler]

If you're worried about compromised CPUs being used to compile executables that are used by others, then reproduceable builds are a great countermeasure. Just use reproduceable builds on many different CPUs, and compare them to ensure they are the same (for a given version of source and tools). The more variations, the less likely that there is a subversion. If what you're compiling is itself a compiler, then use diverse double-compiling (DDC) on many CPUs.

If you're worried that an INDIVIDUAL may end up with a compromised CPU, then yes, it's much harder to counter attack. On some systems, you can isolate the system (no network traffic, etc.). That said, an adversary has to send packets to subvert a specific system, then every time they do the subversion they risk being detected, so it's far less likely to be used for bulk surveillance... it would more likely be one well-resourced organization (e.g., a government) working against another well-resourced organization.

Diverse double compiling (thanks dwheeler)

[tepples]

So long as two or more independently developed, self-hosting compilers for a language exist, with at least one as publicly available source code, a Ken Thompson attack on the public-source one is infeasible. David A. Wheeler proved it; here's the gist:

1. Use Visual C++, Intel C++, and Clang++ to compile g++. The binaries you get in this stage will differ, but if VC++, Intel C++, and Clang++ are uncompromised, they will have exactly the same behavior.
2. Use each of the three copies of g++ you compiled earlier to compile g++, disabling timestamps in the output. Because they all have the same behavior (the behavior of g++), they should all produce the same the output. Thus the binaries you get in this second stage will be identical unless one of the first compilers is compromised.

Re:Diverse double compiling (thanks dwheeler)

[Paul Jakma]

No he didn't prove it is infeasible. For one, that would require a method to prove that the compilers are indeed wholly independent, which hasn't been provided. Also, note that people in some sub-field of technology tend to move around. An engineer who has worked on one compiler is *more* likely to also work on another compiler at some stage than any random engineer. The DDC technique *assumes* that diverse compilers are independent - it takes it on trust. Wheeler's work if anything re-inforces the essence of Thompson's philosophical point, that we must either completely build and control every aspect of our system OR we must trust to at least some degree in someone else. Note also that someone can frustrate this technique by deliberately making their software not build reproducibly, for apparently innocent reasons (e.g. D Wheeler had such issues with using tcc for DDC). A fuller version of my critique of "Diverse Double-Compiling".

That sounds like I'm being very dismissive of DDC, but I'm not. It could be really useful, *if* it is feasible to actually regularly reproduce builds. Debian is working on this, and hopefully they'll get there - but it's not a trivial task either. However, DDC does not fully counter Thompson's attack - not in the normal absolute sense of the word "fully" at least.

Re:Diverse double compiling (thanks dwheeler)

[Paul Jakma]

And the end of that comment still sounds more dismissive than I wanted... Take 2:

I'm not being dismissive of DDC. Distros regularly attempting to get reproducible builds with diverse compilers will raise the bar and make attacks harder if it can be done, and additionally it will help catch bugs. However, DDC does not fully counter Thompson's attack, and it is good to remain aware of the assumptions it operates under.

I.e. could be a very nice step forward, though it is important to note the "fully countering" isn't quite "fully" and there are limitations.

Easy enough to handle trusting trust

[raymorris]

Since you mentioned Reflections on Trusting Trust, that issue is easy enough to avoid. There are some simpler and more clever methods, but consider this:

Use Borland 1.0 to compile llvm.
Use this new llvm binary to compile gcc.
Chain a few more in you want to.

You don't need to trust the first compiler. It could be trojaned so as to trojan new copies of itself. You'd only be concerned if you thought that Borland 1.0 was trojaned in such a way as to add a trojan to the code of a compiler that didn't yet exist, llvm, AND that trojan wasn't for Borland or llvm, but for the current version of gcc - another compiler quite different from anything that existed when Borland 1.0 was created.

The perpetrators would have to not only be astronomically clever, they'd also have to see into the future, twice, in order to build such a trojan.

Re:Easy enough to handle trusting trust

[Paul Jakma]

Why do you think a new trojan can not infect old binaries?

The Thompson attack is what we would recognise today as a class of virus. Indeed, as Thompson's point was a general one about the unavoidable need to trust others, if one did not build every component capable of basic logical manipulation oneself, to fully counter Thompson's attack you would have to be able to counter every possible kind of virus and rootkit - and not just of the software, but also of any other firmware and microcode that might handle or be involved in running your code. (Read his paper, he is clear he envisions his attack could be implemented in lots of ways and places in the abstract).

Re:Easy enough to handle trusting trust

[Paul Jakma]

Good thing there are no well-known, stable hooks in programmes to allow code to be run in a generic fashion, as part of, say, binary file formats. Oh wait...

Re:trust already lost in Debian

[Eunuchswear]

You do know that you sound like a crazy person, don't you?

Why are they not doing this already?

[McLae]

I was doing this 15 years ago. Your build document specifies the build computer (Brand, model, OS version, etc.), the tools needed to do the build (Compiler(s), Script to code translators, etc., with exact versions), and the source version, with instructions on how to pull that version from the repository. And all the steps in the build.

Our build package included copies of the OS install CD, install media for any tools needed, and the complete code set as text files.

You wipe the disk on the build machine, install the OS, install tools, run the build, and you get a known result.

This should be standard procedure for any software released to the public (Or purchaser). If not, you are open to chaos with releasing software. (Been there, as user. Not fun!)

Thomas, DeSoto, TX

Re:Why are they not doing this already?

[allo]

But i guess your binaries still had a different checksum. For example because of timestamps. So you need to analyse byte-by-byte, what are the differences and if they are unimportant. Now you get the same binaries and do not need to check anything further.

Re:trust already lost in Debian

[Barsteward]

well, vote with your feet and go off to devuan and stop polluting a non-systemd article with your immature and incorrect rant

Re: trust already lost in Debian

[Barsteward]

Again we see the immaturity of the anti-systemd kids. - just fixed that for you, didn't bother with the rest of your cut and paste post because it just shows just how little you know.

Re:The stupidest part of open source

[ledow]

I have to say - necessary compile environments really put me off coding projects.

Just having a Makefile is not sufficient. Even having all the right versions of prerequisite libraries isn't sufficient. Sometimes you have to patch and tweak and pass parameters and all kinds us to build the damn thing properly.

Lots of software is like this. Especially when you work on a non-standard platform, even ARM, or with certain libraries (ffmpeg! grr!).

Let's not even get into what happens when it compiles against some ancient version of an obscure library even if it barely uses it.

We really need a way to specify this kind of thing. There needs to be an XML format with namespaces that specify - down to the version, source file and original location - what the original prerequisites were for each particular build. Hopefully that's what they are working towards.

P.S. This is a MASSIVE killer for teaching programming students. Introduce them to C, everything is fine. Get them to compile against a particular library - even SDL which is quite portable - and you can be in for a world of hurt. Compile on something unusual like MinGW and you're back to the command line hand-compiling some of these libraries with dozens of switches and path-overrides just to compile a simple Hello World example.

Re: Soon to be assimilated

[Anonymous Coward]

The systemd guys are young and energetic.

and foolish.

Re: Soon to be assimilated

[mynamestolen]

I think you missed the sarcasm

already being done in gaming machine builds...

[chriskenrick]

I used to work for a manufacturer of poker(slot) machines and hybrid/table casino games, and this was a non negotiable requirement. A given set of source code had to produce exactly the same binary output, to the point where you'd get identical checksums when verifying it. Furthermore, external test labs responsible for certifying the software also needed to be able to build the software from source, and verify the binary in the same way.

The biggest headache in the process was anything that included a date time stamp in the object code or final binary .. from memory there was some part of the build process that would reset all those date stamps to a fixed value. A virtual machine as a build box was also the only reasonable solution for guaranteeing a very locked down, but easily cloneable build environment (before that we were resorting to supplying physical build machines to each test lab)

Tor Project Writeup on Deterministic builds

[whh3]

Part 1
Part 2

Re:Yes, but can you trust your compiler tool chain

[Henning Rogge]

This only works if Debian can guarantee the integrity of the development tool chain. See this >30 year old talk/paper by Ken Thompson describing the problem. Once inserted, the malware is persistent and invisible. Re-compiling your compiler and applications from known-good versions doesn't help.

The problem got a lot more complicated for the attacker today... Thompsons attack works well if there are only a few architectures and only a single compiler. But the attack complexity grows exponentially in the presence of multiple architectures (that can be used to cross-compile each other) and multiple compilers (that can compile each other). Now you need a compiler virus that not only compiles on all architectures well, it also needs to detect all kind of compilers that are there and works on all versions of them. The "reproducible build" system makes it even worse for the attacker, because its easier to to compare the results.

Re:The stupidest part of open source

[gl4ss]

gradle and some other stuff is basically like your suggestion.

specify repos, names and version(s)/ranges that you want and build.. in theory anyways.

Borland CDs are read only

[raymorris]

> Why do you think a new trojan can not infect old binaries?

CD, and floppies with the tab set, are read-only. Unless this virus changes the physical properties of aluminum, your old Borland CD isn't going to get infected.

Re:Borland CDs are read only

[Paul Jakma]

You can't run a compiler from read-only media though.

Re:Borland CDs are read only

[metrix007]

Of course you can, it just needs a writable working directory.

Re:Borland CDs are read only

[Paul Jakma]

Perhaps I wasn't being explicit enough.

The CDROM might be read-only, but the software has to be copied into memory by something in order to run. As per Thompson's original point, it isn't sufficient to protect one piece of the system. As he stated, his attack implies that *every* programme that is involved in the handling of software must either be validated to the same level as having written it yourself OR you must invest trust:

In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program ..

(emphasis mine).

Indeed, his point on trust extends beyond just programme-handling programmes to all logic (soft or hard) involved in the handling and the running of software. Thompson mentions microcode almost after the text above:

As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.

Since Thompson, we've had "Blue pill" rootkits that use x86 virtualisation features to effectively run themselves as microcode under the victim system (and unbeknownst to).

It's been necessary since Ken Thompson's bsd hack

[Tesseractic]

I applaud this initiative, and it may make me switch back to Debian as my OS of choice.

The trusting trust problem is a serious one, and if you can't rely on being able to build a
byte-for-byte identical unit from source, you can't really have any confidence that you're
running code that represents what the authors intended.

  - I used to be a perfectionist - now I am much better; I know how to compromise.

Borland predates Linux, ELF

[raymorris]

Are you under the impression that the DOS .exe files produced by Borland 1.0 are approximately compatible with Linux ELF files? Maybe you're thinking that because neither are Windows, Linux must be DOS? No, there's nothing "stable" between the two completely different formats. So the Borland compiler couldn't possibly include a trojan for an operating system that didn't yet exist, using an executable format that didn't yet exist.

Re:Borland predates Linux, ELF

[Paul Jakma]

I'm not familiar with DOS exe format. However, there must be some well-defined entry point.

Thompson's attack doesn't mean that any subversion of the Borland 1.0 compiler is limited to when the Borland 1.0 compiler was created. Thompson was making an extremely general point about security in programmable systems: You either build pretty much all of it yourself, or else you must invest trust in others.

not trusting is hard work

[raymorris]

Well, since you're not familiar with either format, let me give you an analogy. Go build a mold for making intake manifolds to fit all 2040 model year cars. That's essentially equalivent to what Borland would have had to do in order to include a Linux elf trojan in the 1980s.

The Thompson paper reminds us that the normal workflow involves trusting the toolchain. It in no way indicates that we can't choose a paranoid workflow instead. One type of paranoid workflow involves validating our modern tools by using much older and much different tools. Because attackers of the 1980s couldn't predict the future, they couldn't code surreptitious trojans for the 2015 toolchain.

Re:not trusting is hard work

[Paul Jakma]

Not sure what car manifolds have to do with it - argumentum ad vehiculum.

Again, you're assuming that an old toolchain can only have old attacks. That's a flawed assumption. A modern attacker can subvert your system so that old toolchains are subverted to apply further subversions.

Are there practical steps we can take to raise the bar and make such attacks much harder to execute. Sure. Can we guarantee our system is free of such subversions, without either trusting others to some degree or building the system entirely ourselves: no we can't. Which was Thompson's point.

explain how you rewrite the laws of physics

[raymorris]

> A modern attacker can subvert your system so that old toolchains are subverted to apply further subversions.

Explain please, how you imagine the silver in a pressed Borland Turbo CD or the DOS CD it runs on, is going to get new malware added to 20 years after it was pressed.

The stock Borland Turbo and DOS disks are read-only. That means they can't be changed. I'm not sure what part of read-only you don't understand.

Re:explain how you rewrite the laws of physics

[Paul Jakma]

The system is subverted, e.g. command.com has been modified, so that when Borland Turbo is loaded into memory it too is subverted. Alternatively, DOS 22h is replaced with a version that checks every disk write to see if it is the beginning of a DOS executable, and if so, subverts it. Alternatively, ... etc.

There are surely many ways. Otherwise, you are arguing that DOS is not vulnerable to a broad range of all-powerful subversions, which is patently untrue.

trolling are really that dense?

[raymorris]

> command.com has been modified

I'm not sure if you're just trolling or if you really, truly don't know what a CD-ROM is, what read-only means.

Before iphones - I mean before the very first iphone, and before Windows 7 or 8, you couldn't download apps. Instead, apps were made out of aluminum- metal. The metal was inside of some plastic. You had to physically walk into a store to buy your apps, and you'd walk out with these metal and plastic circles. Those circles had the apps. You couldn't change them. The operating system was the same way. To update your system, you'd throw away your old"metal circle and buy a new, different one.