Debian Working on Reproducible Builds To Make Binaries Trustable
An anonymous reader writes: Debian's Jérémy Bobbio, also known as Lunar, spoke at the Chaos Communication Camp about the distribution's efforts to reassert trustworthiness for open source binaries after it was brought into question by various intelligence agencies. Debian is "working to bring reproducible builds to all of its more than 22,000 software packages," and is pushing the rest of the community to do the same. Lunar said, "The idea is to get reasonable confidence that a given binary was indeed produced by the source. We want anyone to be able to produce identical binaries from a given source (PDF)."
Here is Lunar's overview of how this works: "First you need to get the build to output the same bytes for a given version. But others also must to be able to set up a close enough build environment with similar enough software to perform the build. And for them to set it up, this environment needs to be specified somehow. Finally, you need to think about how rebuilds are performed and how the results are checked."
Here is Lunar's overview of how this works: "First you need to get the build to output the same bytes for a given version. But others also must to be able to set up a close enough build environment with similar enough software to perform the build. And for them to set it up, this environment needs to be specified somehow. Finally, you need to think about how rebuilds are performed and how the results are checked."
I'm confused - you say this is a Debian initiative, but I don't see how this promotes Social Justice, Inclusiveness and Diversity.
Would make it harder for them to exploit.
Isn't deadbeef the name given to the roast beef sandwich between your mom's leg?
Gitian is a good start
Hello, apk! I'm still struck by the admiration and respect I feel for your great achievement: your host engine.
Can you perhaps provide a systemd module for it? It would be really great!
i'm a fan of equality, so i'm not a fan of feminism, but how does this have anything to do with feminism?
on what at the roots amounts to a misguided feminist agenda
ELI5?
you'd have to how the Debian internal politics have changed, there are articles about that and also about how it relates to systemd. even slashdot had one
Wouldn't virtual box be able to do this? Then one only has the trouble of validating the virtual box. While that is actually probably harder to do you don't have to do this as often and one could use some open source virtual box equivalent and compile that from source.
On the otherhand I don't quite understand why, if one can compile the source, one needs to worry about untrusted binaries. Perhaps the intent here is for some master agency to watch for tinkered binaries or to post it's own Checksums apart from Debian. Then everyone has two sources for validated checksums.
Some drink at the fountain of knowledge. Others just gargle.
Unless you freeze system time for the full duration of the build, every piece of code that builds in __TIME__ or __DATE__ macros, will screw with this. Other environment macros injected into the build ( like git revision etc ) as well.
http://validator.w3.org/check?uri=http%3A%2F%2Fwww.slashdot.org Errors found while checking this document as HTML5!
See subject: Via this method-> http://it.slashdot.org/comment... (Did ok here CODING FOR DEFCON in 2005 (my compressed/packed exe + sizecheck @ startup technique)).
* This can also be done via alternate methods like CRC checks too...
(I'm honestly surprised more app makers don't do this to be blunt about it... it works).
I don't compress the exes anymore (triggered false positives) such as in APK Hosts File Engine 9.0++ SR-2 32/64-bit -> http://start64.com/index.php?o... BUT I still perform that self-test of itself @ its initialization for the reasons noted in the 1st link above & below...
APK
P.S.=> I'm not sure what they're out to prove or do here, but this helps function that way for trustworthiness of an application & verifying it's the original build size by doing the above - which also functions as a "built-in" native to the app itself virus checker too (since 'traditional exe viruses' attached themselves to the tail of an executable & alter the jump tables to work, but ADDING SIZE TO DO SO which this method catches in that capacity)... apk
systemd
I don't compress the exes anymore (triggered false positives)
Ah, so Khyber was TELLING THE TRUTH about AV software detecting your shit having viruses. And from what I am reading, not once did they mention anything about it being a false positive. Of course, given risks on the internet now days, I wouldn't expect them to keep the file on their system if its suspect, so I don't blame them for not investigating further.
How about you eat crow and go apologize to them for your own incompetence and lack of foresight?
I was thinking about this being a problem a while back - how to deal with building something from source and knowing I was getting the same output that the developers wanted me to have. Coincidentally about the same time, a href="http://developers.slashdot.org/story/13/06/20/1548228/are-you-sure-this-is-the-source-code">this article popped on Slashdot and introduced me to Ken Thompson's article Reflections on Trusting Trust - a great read and something that really opened my eyes (in that wide-open-because-of-terror kind of way).
Also from that thread came this email from one of the Tor developers talking about their deterministic build process to do the same thing.
I think this is a problem that would be really great to solve as soon as possible. I very much hope that once we start seeing more reproducible builds we don't suddenly find out that certain compilers have been compromised long ago.
At least systemd doesn't have maintainers that just randomly comment out lines of code in security software based on unverified, false positives from a static analyzer.
No but it does have a bunch of assholes determined to shove it down our throats no matter what, using all sorts of politics and other "not based on technical merit" tactics to make this happen. If I ever run systemd (not likely, but possible) it sure as hell won't be because somebody else decided I should. If I wanted somebody else to decide what was on my system, I would run a commercial OS and be done with it.
I find the Wiki page on systemd to be exquisitely ironic. In the first paragraph or two it says "The name systemd adheres to the Unix convention of making daemons easier to distinguish by having the letter d as the last letter of the filename." Well that's great! At least there is one single thing about systemd that can be said to follow the Unix way of doing things. And here I was thinking there were none at all.
I was thinking about this being a problem a while back - how to deal with building something from source and knowing I was getting the same output that the developers wanted me to have. Coincidentally about the same time, this article popped on Slashdot and introduced me to Ken Thompson's article Reflections on Trusting Trust - a great read and something that really opened my eyes (in that wide-open-because-of-terror kind of way).
Also from that thread came this email from one of the Tor developers talking about their deterministic build process to do the same thing.
I think this is a problem that would be really great to solve as soon as possible. I very much hope that once we start seeing more reproducible builds we don't suddenly find out that certain compilers have been compromised long ago.
Bitcoin solved this by using gitian. It builds inside a VM.
https://gitian.org/
So... what to do then? What other distribution works well, has the large amount of packages available, is freely available (not a big deal since RH isn't going to be systemd free), and pretty much Just Works?
Don't blame me, I voted for Kodos
Compile your own binaries using ports on freebsd. :)
Oh, and no systemd there either.
if only...
I mean, clearly the Debian guys have thought of this, so what am I missing?
This only works if Debian can guarantee the integrity of the development tool chain. See this >30 year old talk/paper by Ken Thompson describing the problem. Once inserted, the malware is persistent and invisible. Re-compiling your compiler and applications from known-good versions doesn't help.
NetBSD has the pkgsrc collection, which is fairly large, and it is never going to be polluted by systemd.
So... what to do then? What other distribution works well, has the large amount of packages available, is freely available (not a big deal since RH isn't going to be systemd free), and pretty much Just Works?
OS X.
Post the source or it's a virus.
What about compromised CPUs? If you are the NSA I think it's easier to build a backdoor into the CPU than try to keep up with ever changing software builds. Isn't it? CPUs are totally controlled by three or four U.S. companies, are closed source nobody has ever seen into it...
https://www.reddit.com/r/badbi...
I bet all hardware is pwned - check it out, some serious shit!
See subject: 57 antivirus programs say so + MalwareBytes' hpHosts Admin (MalwareBytes employee) hosts & recommends it -> http://hosts-file.net/?s=Downl... & MalwareBytes = BEST antivirus per this VERY recent testing of them all http://www.av-test.org/en/news...
(MalwareBytes' hpHosts admin HAS the sourcecode & verified it clean as well - hence, why he + others in the security community HIGHLY RECOMMEND it as well as host it! You wouldn't BELIEVE what they put me through for all of this testing either... & I passed them all as clean/safe!)
&
It's GUARANTEED safe & clean per it being checked by 57 antivirus programs recently in BOTH its 64-bit model https://www.virustotal.com/en/...
+
In its 32-bit model also https://www.virustotal.com/en/...
APK
P.S.=> I don't owe ANYONE here a view @ my work after the above, that's certain - considering most here don't code (or they'd have work of their own out there, 99% here don't) & wouldn't understand it anyhow!
+
I personally consider much of the "Open SORES" movement largely a code thieving system!
Face it - "all those eyes on the code" don't stop ANDROID infestations galore daily almost
&
Their INFERIOR browser addons like "AlmostALLAdsBlocked" (paid off to NOT DO ITS JOB & crippled by default) http://www.businessinsider.com... are exactly that - inferior in not doing as much for security, speed, reliability, & anonymity from any SINGLE one of them vs. hosts, + they're less efficient as well in terms of resources consumed, by FAR, as well! apkb
So many times, I've downloaded the source of something, and been completely unable to build it, because the build environment/libraries/dependencies/whatever was not specified.
While there are occasionally step by step instructions of what to install, there most commonly isn't, and the "developer" doesn't have a fucking clue.
In many cases, the problem is solved by using a different version of the compiler - the precise version the dev uses - which is utter bullshit. It's like needing to use a certain brand of oven to bake your cake.
Debian, and all programs, should ALWAYS have been able to be compiled by anyone to achieve the same binary. Isn't that the purpose of make files?
From a technical standpoint, I understand why this occurs, but from a technical standpoint, it should have always been avoidable, and the ball was dropped - by practically everyone - a long time ago.
Alas, Debian once was the standard that other Linux distributions were measured by.
Fast, stable, dependable, reliable, secure, usable.
Not so today, since they decided to pour bucket loads of experimental garbage code into their base, that the distribution can give the Gnome Desktop user a more Windows like experience. ..
Note: Nobody uses Gnome anymore. Move on
To be fair, a Linux distribution with training wheels (systemd) could be helpful to those wishing to migrate away from M$ addiction, and have a system to learn the linux desktop on. But changing the entire Linux ecosystem for this experimental bloat was suicide.
Thankfully, FreeBSD has reached a maturity that Debian/Linux purists can migrate to, and can still have a safe and secure, modern, state of the art, desktop system, their way, without the squirrelly flashy wizbang bloated garbage that has now infested the Linux world.
"Suppose you were an idiot...and suppose you were a member of Congress...but I repeat myself." Mark Twain
Write Your own compiler?
AFAIR You could write a half decent LISP interpreter in a weekend, starting from virtually nothing. You are basically writing a basic -calculus to machine language code, and then it magically becomes self-hosting.
See this paper describing how to counter that problem. The trusting trust attack is not the end of software security.
They're forums sliders attempting to bury other conversations. It's what idiots like that do and why they do it. If you've never heard of forums sliding look it up. They do it above posts that are posted earlier than ones they want buried by making replies to the earlier posts (often off topic and stupid like you yourself state).
Again we see the immaturity of the systemd kids. For us adults, concepts like stderr, syslog, and exit statuses are concepts critical for managing a server. We understand and embrace the UNIX-way. If those kids spent as much time learning UNIX as they did posting childish rants, they'd why people don't like systemd.
So long as two or more independently developed, self-hosting compilers for a language exist, with at least one as publicly available source code, a Ken Thompson attack on the public-source one is infeasible. David A. Wheeler proved it; here's the gist:
Since you mentioned Reflections on Trusting Trust, that issue is easy enough to avoid. There are some simpler and more clever methods, but consider this:
Use Borland 1.0 to compile llvm.
Use this new llvm binary to compile gcc.
Chain a few more in you want to.
You don't need to trust the first compiler. It could be trojaned so as to trojan new copies of itself. You'd only be concerned if you thought that Borland 1.0 was trojaned in such a way as to add a trojan to the code of a compiler that didn't yet exist, llvm, AND that trojan wasn't for Borland or llvm, but for the current version of gcc - another compiler quite different from anything that existed when Borland 1.0 was created.
The perpetrators would have to not only be astronomically clever, they'd also have to see into the future, twice, in order to build such a trojan.
You do know that you sound like a crazy person, don't you?
Watch this Heartland Institute video
Our build package included copies of the OS install CD, install media for any tools needed, and the complete code set as text files.
You wipe the disk on the build machine, install the OS, install tools, run the build, and you get a known result.
This should be standard procedure for any software released to the public (Or purchaser). If not, you are open to chaos with releasing software. (Been there, as user. Not fun!)
Thomas, DeSoto, TX
The reproducibility problem was solved in the 1980's by the Domain Software Engineering Environment (DSEE) of Apollo Computer, Inc. Apollo produced both hardware and software ["not quite UNIX"] which competed with early UNIX graphical workstations. The source code control of DSEE was on a par with git. In addition to the version of the source, the output of a build also recorded the identity (name and version) of every tool and library which affected the output. Reproducing a build involved reproducing the entire build environment, including all tools and libraries; and it was possible to branch within that environment (creating a new version that contained a patch, for instance.) There was enough shared caching and symlinking to make it usable, although a fresh start from a not-recently-used root could take a while.
The biggest obstacle to people running your hosts program: you. If you were selling a commercial product, any competent marketing manager would tell you to stay behind the scenes and shut the fuck up while they attempt to be professional.
Bragging about your lack of posting limits? While complaining that others abuse things? Hah. But somehow in your deranged TimeCube mind you will convince yourself that you scored some great victory.
I don't take the advice of spammers. I don't use their programs. I don't take them seriously. You have a couple of sycophants but other than that, everyone else here thinks you're an obsessive nutjob. You never did answer the question: have you been diagnosed with any kind of mental or emotional illness? Answering that question entails saying "yes" or "no", in case you're unclear about how to directly answer an unambiguous question.
well, vote with your feet and go off to devuan and stop polluting a non-systemd article with your immature and incorrect rant
"The hands that help are better far than lips that pray." - Robert Ingersoll (1833-1899)
Again we see the immaturity of the anti-systemd kids. - just fixed that for you, didn't bother with the rest of your cut and paste post because it just shows just how little you know.
"The hands that help are better far than lips that pray." - Robert Ingersoll (1833-1899)
stops you from getting infected in the 1st place by blocking known sources of infestation...
You mean the Internet?
That when you consider compiler bootstrapping and generational compile vectors for inserting attacks "in the middle" I hope they take the step to create or audit a very clean bootstrap path to a known version of the compilers. Else no matter what is done, a series of "slow-change" modifications to the compilers over years could leave the system wide open to unrecoverable backdoors.
or uae freebsd or l4. the commies have discovered informatics. let them fsck wirh linux...
systemd is hihgly gay, and was pushed into debian thanks to debian woman and other non-sense, not-technical "arguments".
Until you've done better or prove him wrong, take your own advice, shut the fuck up and get on topic troll.
I've done better. I do better all the time, by not spamming some stupid program on a forum filled with people who don't give a shit. I do better all the time, by using multiple overlapping tools for security and not psychotically insisting that said stupid program is The One True Way. I do better all the time, by conducting my own arguments instead of having to hide while little dick-sucking sycophant myrmidons like you do it for me.
The systemd guys are young and energetic.
and foolish.
I think you missed the sarcasm
work in progress
You haven't done shit you lousy little useless bum. If you did you'd show it. You can't.
I used to work for a manufacturer of poker(slot) machines and hybrid/table casino games, and this was a non negotiable requirement. A given set of source code had to produce exactly the same binary output, to the point where you'd get identical checksums when verifying it. Furthermore, external test labs responsible for certifying the software also needed to be able to build the software from source, and verify the binary in the same way.
.. from memory there was some part of the build process that would reset all those date stamps to a fixed value. A virtual machine as a build box was also the only reasonable solution for guaranteeing a very locked down, but easily cloneable build environment (before that we were resorting to supplying physical build machines to each test lab)
The biggest headache in the process was anything that included a date time stamp in the object code or final binary
Part 1
Part 2
remove nospam. to email!
This only works if Debian can guarantee the integrity of the development tool chain. See this >30 year old talk/paper by Ken Thompson describing the problem. Once inserted, the malware is persistent and invisible. Re-compiling your compiler and applications from known-good versions doesn't help.
The problem got a lot more complicated for the attacker today... Thompsons attack works well if there are only a few architectures and only a single compiler. But the attack complexity grows exponentially in the presence of multiple architectures (that can be used to cross-compile each other) and multiple compilers (that can compile each other). Now you need a compiler virus that not only compiles on all architectures well, it also needs to detect all kind of compilers that are there and works on all versions of them. The "reproducible build" system makes it even worse for the attacker, because its easier to to compare the results.
Mod Parent Funny
See subject: NOD32, Comodo, McAfee, Sophos, EmsiSoft, ArcaVir, & Qihoo360 rescinded those false positives, & the link below is further proof of that as well!
I proved that they found my 32-bit COMPRESSED model of my program a "virus" yet NOT my 64-bit one - & the folks @ MalwareBytes' hpHosts (who helped me thru this no less & HOST + RECOMMEND my program -> http://hosts-file.net/?s=Downl... ) have its sourcecode WHICH IS THE EXACT SAME IN BOTH MODELS (except for resource strings that say "32-bit" vs. "64-bit" to differ both models in their GUIs) so they knew they couldn't say 1 is a virus, the other not, when they're exactly the same.
Face facts (since you downmodded the last time I posted this to try to "hide it" since you failed -> http://linux.slashdot.org/comm... )
* You FAIL, troll (Khyber posting by ac no doubt)...
(Some of the "rules" the fools @ these companies use aren't right, & I've proved that above on COMPRESSED EXECUTABLES ALONE - some do it if you use a WinRAR SFX installer, which is NOT A "VIRUS" EITHER... most everyone in the security community knows it!)
FUNNIEST PART IS HOW THE ANTIVIRUS MAKERS TRY TO SCREW EACH OTHER OVER THAT WAY -> http://www.theregister.co.uk/2...
As well as how SYMANTEC literally ADMITTED ANTIVIRUS IS NOT VERY EFFECTIVE AS A DEFENSE ANYMORE too -> http://dottech.org/157355/syma...
* Whereas my program (proven clean MANY times shown above AND below too), stops you from getting infected in the 1st place by blocking known sources of infestation...
APK
P.S.=> As to Khyber? He's an imbecile (I shot him down easily on many points regarding that & other technical matters)-> http://linux.slashdot.org/comm... which you downmodded to try "hide" too, Khyber posting as ac now... apk
"NOD32 detects a trojan in APK's HOSTS bullshit." - by Khyber (864651) on Saturday August 22, 2015 @01:02PM (#50370415)
VirusTotal & NOD32 SHOW IT COMPLETELY CLEAN IN ITS EXES
https://www.virustotal.com/en/...
AND
https://www.virustotal.com/en/...
There's only 2 exe's & 5 text files in it - The exe's are proven clean as shown above in the 2 links from VirusTotal, the installer's a SFX rar (keeps it 2mb smaller on download) - that's NO virus!
(Unless YOU know of a way that .txt files are "viruses")
---
"he's tying to get your fucking information." - by Khyber (864651) on Saturday August 22, 2015 @01:02PM (#50370415)
My program doesn't transmit outward ONLY intake of data from 10 reputable sources in the security community!
---
"APK is apparently too fucking stupid to do this at the ROUTER level where it's most effective" - by Khyber (864651) on Saturday August 22, 2015 @01:02PM (#50370415)
You believe in "eggshell security" which fails per -> http://www.theregister.co.uk/2...
A TRULY COMPETENT NETWORK ADMIN WOULD DO FAR MORE THAN MERE PERIMETER LEVEL SECURITY @ ROUTER LEVEL!
(Right down to the endpoints/network nodes level in PC workstations also using tools you already have in hosts + firewalls (vs. "piling on 'MOAR'" that's inefficient & not nearly as effective in slower usermode browser addons)).
---
"Windows 10 has hardcoded IPs and bypasses HOSTs." - by Khyber (864651) on Saturday August 22, 2015 @01:02PM (#50370415)
Windows ONLY bypasses hosts files for Windows update (Win8 & below) & for the tracking "telemetry" in Windows 10 - test it yourself.
---
"Browsers can bypass HOSTs as well." - by Khyber (864651) on Saturday August 22, 2015 @01:02PM (#50370415)
WTF? They'd be bypassing the IP stack itself, hosts are part of it - since that's impossible? You've proven yourself a moron, again.
APK
P.S.=> See subject + the above, & "EAT YOUR WORDS", Khyber... apk
> Why do you think a new trojan can not infect old binaries?
CD, and floppies with the tab set, are read-only. Unless this virus changes the physical properties of aluminum, your old Borland CD isn't going to get infected.
First you need to get the build to output the same bytes for a given version.
Admittedly I haven't even looked into the eccentricities of open source compilers, but... if you look at the Microsoft compilers, for example, they don't even produce the same output bytes from the same compiler for the same source code input. Their output is "close", but Microsoft's compilers include something akin to a timestamp in the executable headers that is different for every invocation.
The hardest part is keeping an identical build environment. Nixos is specially designed to do so: https://nixos.org/
I applaud this initiative, and it may make me switch back to Debian as my OS of choice.
The trusting trust problem is a serious one, and if you can't rely on being able to build a
byte-for-byte identical unit from source, you can't really have any confidence that you're
running code that represents what the authors intended.
- I used to be a perfectionist - now I am much better; I know how to compromise.
Are you under the impression that the DOS .exe files produced by Borland 1.0 are approximately compatible with Linux ELF files? Maybe you're thinking that because neither are Windows, Linux must be DOS? No, there's nothing "stable" between the two completely different formats. So the Borland compiler couldn't possibly include a trojan for an operating system that didn't yet exist, using an executable format that didn't yet exist.
Well, since you're not familiar with either format, let me give you an analogy. Go build a mold for making intake manifolds to fit all 2040 model year cars. That's essentially equalivent to what Borland would have had to do in order to include a Linux elf trojan in the 1980s.
The Thompson paper reminds us that the normal workflow involves trusting the toolchain. It in no way indicates that we can't choose a paranoid workflow instead. One type of paranoid workflow involves validating our modern tools by using much older and much different tools. Because attackers of the 1980s couldn't predict the future, they couldn't code surreptitious trojans for the 2015 toolchain.
> A modern attacker can subvert your system so that old toolchains are subverted to apply further subversions.
Explain please, how you imagine the silver in a pressed Borland Turbo CD or the DOS CD it runs on, is going to get new malware added to 20 years after it was pressed.
The stock Borland Turbo and DOS disks are read-only. That means they can't be changed. I'm not sure what part of read-only you don't understand.
> command.com has been modified
I'm not sure if you're just trolling or if you really, truly don't know what a CD-ROM is, what read-only means.
Before iphones - I mean before the very first iphone, and before Windows 7 or 8, you couldn't download apps. Instead, apps were made out of aluminum- metal. The metal was inside of some plastic. You had to physically walk into a store to buy your apps, and you'd walk out with these metal and plastic circles. Those circles had the apps. You couldn't change them. The operating system was the same way. To update your system, you'd throw away your old"metal circle and buy a new, different one.