Slashdot Mirror


Docker Image Insecurity

An anonymous reader writes Developer Jonathan Rudenberg has discovered and pointed out a glaring security hole in Docker's system. He says, "Recently while downloading an 'official' container image with Docker I saw this line: ubuntu:14.04: The image you are pulling has been verified

I assumed this referenced Docker's heavily promoted image signing system and didn't investigate further at the time. Later, while researching the cryptographic digest system that Docker tries to secure images with, I had the opportunity to explore further. What I found was a total systemic failure of all logic related to image security.

Docker's report that a downloaded image is 'verified' is based solely on the presence of a signed manifest, and Docker never verifies the image checksum from the manifest. An attacker could provide any image alongside a signed manifest. This opens the door to a number of serious vulnerabilities."
Docker's lead security engineer has responded here.

73 comments

  1. Read the update by jbolden · · Score: 5, Informative

    Read the update. Pretty much the Docker team is implementing a container verification system and working through the details of decentralized security. v1 is part of the mechanism being in place. It assumes that an upstream verification is in place which is at best-semi helpful. Everyone agrees that the current system does nothing and the message is highly misleading in that it might lead someone to believe that there is a security system in place when the plumbing isn't finished.

    So there is no argument here between the parties (what nothing to fight about on /.). Worth pointing out to the /. community however not to take that message seriously yet.

    1. Re:Read the update by postbigbang · · Score: 4, Informative

      Seems as though you're giving them a free ride for a rather poorly implemented message. And this is Slashdot, where we'll fight if we feel like it.

      Docker's been pretty loose and fast, and "not taking that message seriously yet" in a supposedly production environment seems a bit sophomoric.

      --
      ---- Teach Peace. It's Cheaper Than War.
    2. Re:Read the update by binarylarry · · Score: 1

      So step 1, collect underpants?

      --
      Mod me down, my New Earth Global Warmingist friends!
    3. Re:Read the update by Anonymous Coward · · Score: 0

      Let them fight!!!

    4. Re:Read the update by Anonymous Coward · · Score: 0

      Yey! We have just reinvented package managers. Fucking open source, nothing but spinning our wheels over and over.

    5. Re:Read the update by Todd+Knarr · · Score: 4, Insightful

      Upstream verification won't help. The client has to verify that the image it received is the same one the server verified, otherwise someone can hack a router to silently redirect the client to a malicious server and serve up whatever image they want alongside a copy of the signed manifest for the official image and you're fsckd. What they need is:

      1. The manifest has to be signed.
      2. The manifest has to contain a secure checksum (cryptographic hash) of the official image the server has.
      3. The client has to verify the signature of the manifest to confirm that the manifest hasn't been altered and comes from the official source.
      4. The client has to verify that the checksum of the image it received matches the checksum for the image in the manifest.
      5. Step 4 is apparently what's missing from the client.

    6. Re:Read the update by jbolden · · Score: 1

      Docker's been pretty loose and fast, and "not taking that message seriously yet" in a supposedly production environment seems a bit sophomoric.

      I agree it is a bad idea. And they agree it is a bad idea. Not sure what we can argue about if both sides agree they screwed up with this mesage.

    7. Re:Read the update by postbigbang · · Score: 4, Interesting

      It's nice to try to deflate this, but the blunder and the QA mistake remain. As I like to hesitate on the side of caution, I'd change this quickly. Just agreeing that one screwed up and not halting distribution for this head-desk sort of error -- in the face of the enormous security risk endowed -- isn't quite satisfactory.

      I'm here to punish no one, but in a crazy sort of way, I find this one to be a bit mind-boggling, to the tune that each and every appliance that wasn't independently MD5'd is now a freaking five star security risk. Chain of authorities are tremendously important, and reasonable people would believe, mistakenly, that all is fine, when none of it now is, because the chain of authorities chain has been broken, and for what I know, from its inception.

      So you're telling me to cool down, and I'm telling you that every single Docker implementation is now reasonably suspect, because of this go-lightly screw-up.

      --
      ---- Teach Peace. It's Cheaper Than War.
    8. Re:Read the update by jbolden · · Score: 1

      Yeah sorta of, except you can't have a single repository playing the role of the distribution. So you need a vastly more complex package manager than anything we invented before, hence the wheel spinning.

    9. Re:Read the update by jbolden · · Score: 2

      No Docker implementation is any worse than it was before. They went from no security to slightly better security that in practice in most install is unlikely to be useful but with a misleadingly reassuring message.

      There could very well be problems since people could be letting down their guard when they shouldn't. My point is that there isn't much debate since the Docker people explained what was going on, everyone agrees that is what is going on and the Docker people agree the message everything is OK shouldn't be in place before the plumbing is to make everything OK.

    10. Re:Read the update by postbigbang · · Score: 1

      And my point is that the chain of authorities is broken, and needs to be fixed, and anyone not doing MD5s are at risk. I would do these anyway. Lots of others don't.

      --
      ---- Teach Peace. It's Cheaper Than War.
    11. Re:Read the update by i.r.id10t · · Score: 1

      So it seems that simply fixing the output to indicate that the filename is in the signed manifest would solve this... when the sum in the signed manifest is checked against the actual file they can change the message back (I would assume that is the reason for such a reassuring message).

      --
      Don't blame me, I voted for Kodos
    12. Re:Read the update by jbolden · · Score: 2

      It is a bit more complex but yes. A much better message might be something like "plumbing 2 of 4 steps functional -- passed" or even "checksum passed: note if you don't know how the Docker checksum works you probably don't have enough auxiliary plumbing for it to be working for you, so please be cautious". which would make it clear that nothing is really being tested at this point for most users.

       

    13. Re:Read the update by jbolden · · Score: 2

      Yes. You have to run an out of band security system at this point for Docker same as before.

    14. Re:Read the update by Anonymous Coward · · Score: 0

      Huh? Sorry to sound harsh but you can agile grandma's kittypic website to your heart's content. But when it comes to software that has any dependency on security, there's no excuse for bugs like this to make it into production. This ADD code-slinging mindset has no place in professionally developed software.

      From post by dev:
      "However, as the libtrust and v2 image work, and subsequently, registry protocols are still in draft and security review, it is difficult for us to recommend that users yet attempt deploying these."

      Random poster:
      "And yet you deployed it to production in 1.3."

      Dev response:
      "This is a valid criticism. I'll admit this code did not get the proper review and audit that I personally expected before being deployed. At Docker, we consider that a process bug. Things happen, we learn, we adapt. We didn't pull the code, but neither did we advise anyone else yet deploy a v2 server. Again, we have an active audit against this code currently, and the spec is developing."

    15. Re:Read the update by ArsonSmith · · Score: 1

      for something that runs in a fenced network and only takes deploys and runs integration tests before being destroyed, I don't see how this is a huge problem?

      You mean people use docker to run production images?

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
    16. Re: Read the update by Anonymous Coward · · Score: 0

      How do you think app engines work ?

      Containerizing your app on their engines.

      A company that rhymes with a scgroogle recently dumped their container system for one that rhymes with tocker.

      Your guess is as good as mine, but is guess some are in production.

    17. Re:Read the update by jbolden · · Score: 2

      Yes. All the time. One of the core idea of modern PaaS systems is that from an OS standpoint what runs in dev runs in test and production no configuration during migration. Google for example last year was running over a billion containers in production.

    18. Re:Read the update by jbolden · · Score: 1

      DevOps style production has less bugs, better reliability and much higher satisfaction than classic approaches. Turns out, ADD works. You may not like it but the research is unequivocal for complex and changing systems.

    19. Re: Read the update by Anonymous Coward · · Score: 0

      Says who? Where's the evidence?

    20. Re: Read the update by Anonymous Coward · · Score: 0

      If you are still doing MD5 you are STILL at risk. They have created fake certs with forged MD5, waking a forged image is even simpler given the size. You need to use SHA1 at the very least, but that too is not very secure anymore. SHA2 is a must these days for anything security related.

    21. Re:Read the update by greg1104 · · Score: 1

      Read Before you initiate a “docker pull”. Note that it contains warnings about almost everything supposedly discovered by this researcher, and it came out before it. That's why no one in Docker land is even taking this seriously. People who are surprised by this aren't aware of what's going on with Docker by definition, because if they were they'd know this is old news.

      Docker 1.X is a first release with lots of known limitations--this is one of them--plus the inevitable security issues of any new software. No one with any sense is putting it into production yet.

    22. Re:Read the update by Anonymous Coward · · Score: 0

      I think it's negligent to release code that has security worse than posting an md5 of the image on a website.

      I think it's criminal to give users a fake warm-fuzzy by indicating that something meaningful has been "verified".

      Can you imagine a browser that accepted self-signed certs without any indication? MitMs become trivial and SSL becomes essentially useless. Can you imagine a Windows update that just had signed manifests and didn't checksum the update code? That's what docker is essentially doing... and the security guy does not plainly admit this...

    23. Re:Read the update by ArsonSmith · · Score: 1

      Docket is a container, but not all containers are docker. I'm pretty sure Google doest down load their images from docker hub.

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
    24. Re:Read the update by jbolden · · Score: 1

      Google uses multiple container systems and a lot of their infrastructure isn't Linux but Docker is a key component of their containerization strategy. If it were even 10+% Docker you talking well in excess of 100m Docker containers in production.

    25. Re: Read the update by jbolden · · Score: 1

      IBM / Rational, Forrester research, GE. I could keep going.

  2. Always audit the source yourself! by Anonymous Coward · · Score: 0

    In fact, build your own casino, with blackjack and hookers.

  3. Re:I wont read TFS by Anonymous Coward · · Score: 0

    But how does Bennet Hasselton verify Bennet Hasselton's hand?

  4. What is this new madness? by tgibson · · Score: 3, Insightful

    I'm about to leave for Sears, inseam and waist measurements in hand. And here I read that my image security is at risk. I better find a new brand of pants I guess.

    1. Re:What is this new madness? by Anonymous Coward · · Score: 0

      I'm about to leave for Sears, inseam and waist measurements in hand. And here I read that my image security is at risk. I better find a new brand of pants I guess.

      Keep your hands close to your inseam at all times.

    2. Re:What is this new madness? by Anonymous Coward · · Score: 0

      Keep your friends' hands closer..

    3. Re: What is this new madness? by Anonymous Coward · · Score: 0

      Yah what's with these open source names. Try searching for anything Chef related (i.e. the provisioning system). You've got chef, knife, recipes, the kitchen ... Google is often no help unless I qualify it further.

      Brew on OS X is just as bad

    4. Re: What is this new madness? by Anonymous Coward · · Score: 0

      It's not as bad as the Windows 8.1 update, which is not to be confused with the Windows 8.1 Update update.

  5. re by JohnVanVliet · · Score: 1

    just another example of the "bleep'ed ed bleep" that passes for a good idea

    it REALLY is time for a X30+ solar flare to kill the electricity for 10 years

    then MAYBE we will have had time to well THINK FIRST!!!
    and change the priories from
    new and "Bleeped up"
    to stable and SECURE

    --
    "I don't pitch OpenSUSE Linux to my friends, i let Microsoft do it for me
    1. Re:re by hawguy · · Score: 1

      just another example of the "bleep'ed ed bleep" that passes for a good idea

      it REALLY is time for a X30+ solar flare to kill the electricity for 10 years

      then MAYBE we will have had time to well THINK FIRST!!!
      and change the priories from
      new and "Bleeped up"
      to stable and SECURE

      If you're interested in stable and secure, you're already not using Docker, so problem solved.

      The problem with insisting that everything has to be well thought out and planned first is that gets in the way of innovation, and things slow way down while you do your planning. But while you're spending a couple years trying to plan out the project and account for every use case and vulnerability, by the time you've written the code, it's already out of date and not useful, so the planning has to start over again.

      If you want to apply NASA level of planning and diligence where a software project can take years (or even decades), you should feel free to use only tried-and-true solutions. Maybe an IBM mainframe will give you what you're looking for. But don't insist that the entire world needs to stop to meet your needs for stability and security.

    2. Re:re by ArsonSmith · · Score: 1

      Hmm, why not? If you're using docker in a production environment I'd hope you're building your own images, so where is the security concern? If you're just testing out software and testing builds then sure pull away from the docker hub, but if you're deploying to production you better have your own artifact repository storing your personal generated images.

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
  6. What? by ArchieBunker · · Score: 5, Insightful

    Don't tell us what the fuck a docker is or anything...

    --
    Only the State obtains its revenue by coercion. - Murray Rothbard
    1. Re:What? by Anonymous Coward · · Score: 0

      Docking is when you place the head of your penis inside the foreskin of another man's penis. Somehow, that makes more sense than this article.

    2. Re: What? by Anonymous Coward · · Score: 0

      Rocket is the container developed by a group building their OS for running containers.

      Rocket is also a clean focused code base that implements a design. As a reference spec, the source can be read in about an hour.

    3. Re:What? by Anonymous Coward · · Score: 0

      It is clearly the Unity thing, doh. Didn't you read the summary? It mentions Ubuntu and images, so that must be it! Clearly something to do with the promotional images that pop in the Unity "Docker". How is an unauthenticated image a security hole I can't tell. Do they exploit the PNG decompressor or something?

  7. Dockers is a brand of khaki garments by Anonymous Coward · · Score: 5, Funny

    Dockers is a brand of khaki garments and other accessories from Levi Strauss & Co.

    It's obviously a single pant-leg or shoe, hence, Docker, not Dockers.

    You're welcome.

    1. Re:Dockers is a brand of khaki garments by PPH · · Score: 2

      And I'd be pretty insecure if mine had a hole in them.

      Not even considering how my image would suffer.

      --
      Have gnu, will travel.
  8. Can it be enabled? by ThePhilips · · Score: 2

    Docker's report that a downloaded image is 'verified' is based solely on the presence of a signed manifest, and Docker never verifies the image checksum from the manifest.

    Can it be enabled? If yes, then I do not see a problem.

    Otherwise, the signing crap is just that: crap.

    It takes needlessly long time to verify the signature. (Because they are not slow! - they are so secure, so very much OMG secure.)

    It is a huge risk to reconfigure a production system to use unsigned data if emergency arises. (Think recovery from a local backup.)

    Developers forget to renew their certificates and suddenly, in the middle of a production, whole system goes down, because OMG the certificate has expired and data may be not secure!!!

    And then, in the end, the signing keys get leaked or stolen...

    --
    All hope abandon ye who enter here.
    1. Re:Can it be enabled? by Anonymous Coward · · Score: 0

      the "verification" message prints by default and it's seemingly impossible to trigger a failed verification message. so essentially:

      if (true) print("Verification succeeded");

    2. Re:Can it be enabled? by Anonymous Coward · · Score: 0

      Forget to renew the certs, keys leaked, etc. Lemme guess, just got your MCSE?

      Congrats!

  9. Love that response by Tailhook · · Score: 4, Insightful

    A summary of that wall-of-text "response" from the Docker "lead security engineer":

    "Bullshit, bullshit v1 bullshit. Bullshit discussions about bullshit CVE bullshit. (yes we know its broken) Bullshit v2 bullshit, next version bullshit Bullshit."

    If you can't dazzle them with your intelligence, baffle them with your bullshit.

    --
    Maw! Fire up the karma burner!
    1. Re:Love that response by Anonymous Coward · · Score: 0

      I agree, that response was terrible damage-control drivel. Maybe he should write his replies himself next time instead of asking the in-house PR dude to do it for him.

    2. Re:Love that response by Anonymous Coward · · Score: 0

      A summary of that wall-of-text "response" from the Docker "lead security engineer":

      "Bullshit, bullshit v1 bullshit. Bullshit discussions about bullshit CVE bullshit. (yes we know its broken) Bullshit v2 bullshit, next version bullshit Bullshit."

      If you can't dazzle them with your intelligence, baffle them with your bullshit.

      [Posting AC because mods....]

      Yep, that was one of the most substance-free, passive-aggressive, useless pieces of shit I've ever seen in what was supposed to be a technical discussion on a FOSS project. It was so bad —and I say this with all sincerity— I'm tempted to pack up and leave FOSS software development behind.

      Seriously: If insincere, intellectually lazy non-replies are the wave of the future, then I'm taking my board and going home.

      I have never seen such reprehensible complacence from a self-described 'security engineer' in my professional life.

    3. Re:Love that response by paulpach · · Score: 2

      If you can't dazzle them with your intelligence, baffle them with your bullshit.

      1) They have complete lack of image validation
      2) Docker prints "Image validated" as part of the pull.

      This is just flat out embarrassing, If I was the "lead security engineer" of that, I would actually worry about job stability.
      There is not a whole lot he could say that would save face.
      And yet, the response is truly terrible. He essentially says "yeap, we knew about it, shipped it anyway, and talked about fixing it"

      It does not take a PR genius to figure out a good response. For example: "Thank you for pointing it out, we will have a patch soon" or respond with an actual patch. This is not rocket science, validating a signature properly only takes a few lines of shell script or C code, and has been solved a long time ago by most other package managers.

    4. Re:Love that response by Anonymous Coward · · Score: 0

      More like, "There's no real security on this, we said there's no real security on this, we're already working on adding security to this, and in an admission of guilt... that message shouldn't use that language."

      But, you know, everyone has to get all righteously indignant.

  10. They must love xkcd by hendrik42 · · Score: 2, Funny
    1. Re: They must love xkcd by Anonymous Coward · · Score: 0

      Lol.

  11. Docker is dead by Drunkulus · · Score: 1

    Long live Rocket!

    1. Re:Docker is dead by Anonymous Coward · · Score: 0

      Rocket... from the guys that brought you systemd.

      No thanks.

    2. Re:Docker is dead by PCM2 · · Score: 2

      Err. Rocket is from the guys that brought you CoreOS. CoreOS uses systemd. Not the same thing.

      --
      Breakfast served all day!
    3. Re:Docker is dead by Anonymous Coward · · Score: 0

      It's a conspiracy! They're all in on it! Every last one of them is bringing you systemd!

    4. Re:Docker is dead by Anonymous Coward · · Score: 0

      > Long live Rocket!
      Docket? Rocker?

      I can't help it: it all looks the same to me. Lotsa marketing bullshit for sysadmins too lazy to do their jobs properly. Oh -- nearly forgot: those are called DevOps (CamelCaseIncluded(TM)) these days.

      I'm waiting to see all this mess spinning ever faster until it runs hot and suddenly seizes. That'll be fun.

  12. Re:I wont read TFS by Anonymous Coward · · Score: 0

    Is that after cutting off circulation to the hand so it feels like a stranger is stroking his cock?

  13. THESE AREN'T THE DROIDS YOU'RE LOOKING FOR by twitnutttt · · Score: 2

    I'm glad I'm not the only one who had that reaction. Unintelligible drivel!

    Here are my favorite excerpts:
    -- "There is nothing particularly new in Jonathan's post and I thank him for facilitating a conversation [about nothing particularly new apparently]."
    -- "Image security is of the upmost importance to us. For these reasons, we've [reached] many of the same conclusions [that there is no image security]."
    -- "v1 is not v2. v1 has a flawed design. we have a draft for v2. v2 will be better. v2 will be much more shiny. when you have v2, you will forget how insecure v1 is. until then, we recommend you use v2, because v1 is not secure. v2 is still in draft."
    -- "THESE AREN'T THE DROIDS YOU'RE LOOKING FOR."

  14. heh by Anonymous Coward · · Score: 0

    So did you email them in the first instance or did you go for the thunder and glory on slashdot first?

  15. Re:I wont read TFS by davester666 · · Score: 1

    That is a job for his hand.

    --
    Sleep your way to a whiter smile...date a dentist!
  16. Summary is scaremongering by Anonymous Coward · · Score: 2, Insightful

    Read the article, summary makes it sound as if Docker doesn't verify the checksums and it does. What his complaint is, that it verifies the checksum AFTER decompress, de-tar'ing from a HTTPS source, and only does a cursory check on the TAR file.

    He complains that the check on the TAR file is imperfect, which is true, and that the act of unpacking might reveal a vulnerability in the unpacker which could compromise the machine.

    So, to be clear, his proposed attack is "intercept the https source" (which is possible by the NSA/GCHQ due to the certificate authority weakness), they then send a TAR which triggers a bug in unpack and takes control of a machine (which surely needs this untar bug to be fixed rather than docker to slap a fix around it), and then creates a plausible clean version sent to Docker to be verified.

    It's then presented on slashdot, as "docker doesn't verify signatures" which is misleading to the point of deception.

    1. Re:Summary is scaremongering by Anonymous Coward · · Score: 0

      As I understand it the problem is thus:

      There are img files that contain an entire runnable docker app.
      Inside said file are file(s), one of which contains a hash of the other files. A manifest
      That manifest file is signed in some fashion.
      They never check that hash against the other files in the img.

      Which mean I can take Company X's img change everything except the manifest file and it will be all happy go luck and run/install what ever I put in the img file and Company X's has still signed the manifest.

    2. Re:Summary is scaremongering by TheSunborn · · Score: 1

      "Read the article, summary makes it sound as if Docker doesn't verify the checksums and it does. What his complaint is, that it verifies the checksum AFTER decompress, de-tar'ing from a HTTPS source, and only does a cursory check on the TAR file."

      Are you sure about that? From the description from the docker guy, it sounds like they don't verify it at all.

  17. Re:I wont read TFS by smittyoneeach · · Score: 1

    Especially when anti-Fascism Architecture ensures that the left hand knoweth not what the right hand doeth.

    --
    Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
  18. Docker is fucking awful by Anonymous Coward · · Score: 1

    It's one of those stupid buzzword masturbatory things that every quarter-assed IT person thinks is going to SAVE THE WORLD but isn't really any better than a bunch of scripts except that half of it (cf. this article) isn't really implemented properly yet and lack of understanding is excused with "well, it wasn't written in-house".

  19. What is Docker? by Anonymous Coward · · Score: 0

    As a member of the old internet, I scoff at your presumption that we even know what Docker is.
    So....It's a glorified app packager?
    How quaint.

  20. C is not a memory-safe language (rubbish) by Teunis · · Score: 1

    "C is not a memory-safe language" - for that comment alone, the entire review becomes untrusted.

    One fallacy means that the entire work might just be a continuous set of fallacies.
    (C is only memory-unsafe if not used safely - which, given that there's very few barriers to a programmer from shooting themselves in the foot - is always a risk)

  21. Fixing the message by billstewart · · Score: 1

    It would be ok if the message said "Manifest file contains correctly formatted checksum - still need to verify."

    That might also give you the hint that, if no message about "checksum verified correctly" appears later, probably no verification has been done.

    --

    Bill Stewart
    New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
  22. Docker images are bullshit by allo · · Score: 1

    Build a base tar.gz with debootstrap and use Dockerfiles. Downloading images is insecure by design. What makes you trust the docker verification? How are they supposed to spot a backdoor in a whole big image? Build it yourself, use Dockerfiles you read first.