Slashdot Mirror

← Back to Stories (view on slashdot.org)

How IKEA Patched Shellshock

Posted by samzenpus on from the protect-ya-neck dept.

jones_supa writes: Magnus Glantz, IT manager at IKEA, revealed that the Swedish furniture retailer has more than 3,500 Red Hat Enterprise Linux servers. With Shellshock, every single one of those servers needed to be patched to limit the risk of exploitation. So how did IKEA patch all those servers? Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming." On a more serious note, he said that it took approximately two and half hours to upgrade their infrastructure to defend against Shellshock. The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years.

81 of 154 comments (clear)

  1. What was the command? by Anonymous Coward · · Score: 1

    I imagine it was sudo rm -rf /, but I could be way off.

    1. Re:What was the command? by Joe_Dragon · · Score: 1

      yum update -y && reboot

    2. Re:What was the command? by hawguy · · Score: 4, Informative

      yum update -y && reboot

      You're going to type that on 3500 servers?

      I think you'll want to use your configuration management platform to kick off the update. That's how we did it -- applied the update to the dev servers, did some testing, then the same to qa, then preprod, then finally to the production servers. Took us more than 2.5 hours to test and validate everywhere, but actually pushing out the patch to 1200 servers was a single line command.

    3. Re:What was the command? by Trogre · · Score: 1

      Well I'd wrap it in a loop of some kind:

      for host in `cat /dev/storage/admin/servers.dat`; do ssh root@$host "yum update -y && reboot"; done

      --
      "Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
    4. Re:What was the command? by Acid-Duck · · Score: 2

      Why not do it the way our ancestors did it? :P

      for i in $(cat ips.txt); do
      XXXXXXXXX
      done;

    5. Re:What was the command? by pmgst17 · · Score: 4, Informative

      The article says they're using a Red Hat Satellite server and so if they wanted to run `yum update -y && init 6` on all of their systems, they could just push that out as a remote command to the systems / groups of systems. In Satellite, you can push out remote commands to groups of systems, so if they have their systems grouped, it would be an easy process to push that command to all of their systems.

    6. Re:What was the command? by hawguy · · Score: 2

      Well I'd wrap it in a loop of some kind:

      for host in `cat /dev/storage/admin/servers.dat`; do ssh root@$host "yum update -y && reboot"; done

      You're going to watch the output for 1000+ servers to see which ones failed?

    7. Re:What was the command? by TCM · · Score: 2

      You mean in an amateurish way that can overload shell buffers?

      Try

      while read i; do ...; done < ips.txt

      or

      xargs ... < ips.txt

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    8. Re:What was the command? by ArcherB · · Score: 1

      yum update -y && reboot

      Actually, it kicked off a bash script that consisted of 100,000 commands that took a team of programmers six months to write and debug. But to him, management, it was just a single command that he typed in and took all the credit.

      (it's a joke people)

      --
      There is no "I disagree" mod for a reason. Flamebait, Troll, and Overrated are not substitutes.
    9. Re:What was the command? by lucm · · Score: 1

      this is why God invented Ansible.

      --
      lucm, indeed.
    10. Re:What was the command? by rossz · · Score: 1

      We're currently evaluating Ansible. I expect us to make the switch permanently as part of our move to docker containers. Currently, our puppet manifests are unwieldy and a biatch to maintain.

      --
      -- Will program for bandwidth
    11. Re: What was the command? by MobileTatsu-NJG · · Score: 1

      Your joke mighta been funny if it had contained a humorous punchline.

      --

      "I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)

    12. Re:What was the command? by Bert64 · · Score: 1

      What about files which don't contain a . character?

      --
      http://spamdecoy.net - free throwaway anonymous email - avoid spam!
    13. Re:What was the command? by Anonymous Coward · · Score: 1

      We keep those.

    14. Re:What was the command? by Jupix · · Score: 2

      If you don't mind my asking, what's the difference between QA and preprod for you?

    15. Re:What was the command? by cinky · · Score: 1

      And how will you handle output from those servers? random errors? or will you just fire it up and hope for the best? I'd suggest using puppet or some similar configuration management tool...

    16. Re:What was the command? by sys64764 · · Score: 2

      duh yeah! Thats why we have intern's!

    17. Re:What was the command? by stderr_dk · · Score: 1

      mv *.* /dev/null

      With only one matching file, you'll get:

      mv: inter-device move failed: `foo.bar' to `/dev/null'; unable to remove target: Permission denied

      If you got more than one file matching that pattern, you'll get:

      mv: target `/dev/null' is not a directory

      But thanks for playing...

      --
      alias sudo="echo make it yourself #" ; # https://pipedot.org/~stderr & http://soylentnews.org/~stderr
    18. Re:What was the command? by pz · · Score: 2

      Indeed, you definitely do NOT want hundreds-to-thousands of servers doing an update all at the same time, or, worse, rebooting all at the same time. The first has the potential to saturate your network and bring the entire setup to its knees, and the second will blow your rack supplies. I speak from experience on the latter, having been the one who identified the issue with our weekly DB scrubbing procedure once the company I was working for grew to more than a half dozen servers.

      You want to stagger things by a few 10s of seconds per server on each rack to avoid power supply issues.

      --

      Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
    19. Re:What was the command? by paradxum · · Score: 1

      ok fine... try:

      for i in {1..3500}; do ssh server$i yum update -y; ssh server$i reboot; done

      better?

    20. Re:What was the command? by dreamchaser · · Score: 1

      In some enterprise shops it is just SOP to reboot, usually a policy written by some change management managerial type who doesn't know when a reboot is actually required.

    21. Re: What was the command? by jrumney · · Score: 1

      For the more security conscious, a safer option is sudo dd /dev/zero /dev/sda

    22. Re:What was the command? by fisted · · Score: 1

      Real sysadmins

      a) think before executing potentially disastrous commands, and therefore tend to not need the rm -i crutch
      b) automate the repetitive parts of their jobs, in which rm -i obviously does not make sense
      c) don't experiment around on production servers
      d) have arranged their systems so that accidentally removing stuff can be recovered from.

      Thanks for playing, though

      --
      CLI paste? paste.pr0.tips!
    23. Re:What was the command? by fisted · · Score: 1

      while read i; do ...; done < ips.txt

      How amateurish to spawn an unnecessary subshell.

      xargs ... < ips.txt

      Yes.

      --
      CLI paste? paste.pr0.tips!
    24. Re:What was the command? by TCM · · Score: 2

      If you alias rm to rm -i, what do you think rm -fr gets expanded to?

      Could it be rm -i -fr in which case the -f overrides the -i anyway? Oh great sysadmin, can you clarify?

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    25. Re:What was the command? by Anonymous Coward · · Score: 1

      Rebooting regularly is good practice to ensure your servers are capable of coming back up if something accidentally knocks them down unexpectedly. See also Netflix's Chaos Monkey for a different but similar concept.

    26. Re:What was the command? by neurovish · · Score: 1

      Here, scheduling the reboot of the 900 servers was the longest part of that patching effort.

      O'Reilly? You had to reboot? And you still get paid as a sysadmin?!!(sigh).

      Demonoid-Penguin - moderating (the non-stupid).

      If you're just running a generic "yum update", then you have pretty good chances a new kernel will be pulled in...so yeah a reboot was probably called for.

    27. Re:What was the command? by neurovish · · Score: 1

      Indeed, you definitely do NOT want hundreds-to-thousands of servers doing an update all at the same time, or, worse, rebooting all at the same time. The first has the potential to saturate your network and bring the entire setup to its knees, and the second will blow your rack supplies. I speak from experience on the latter, having been the one who identified the issue with our weekly DB scrubbing procedure once the company I was working for grew to more than a half dozen servers.

      You want to stagger things by a few 10s of seconds per server on each rack to avoid power supply issues.

      Man....I'd forgotten about the PDUs. Had that problem at one place where I brought down the DMZ because I rebooted a server. Fortunately that got a much needed datacenter review underway and people started distributing power correctly.
       

    28. Re:What was the command? by psyclone · · Score: 1

      pdsh FTW

    29. Re:What was the command? by Trogre · · Score: 1

      Well, no. You'd run that inside a screen session, and with an ampersand not a semicolon.

      --
      "Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
    30. Re:What was the command? by TCM · · Score: 1

      Not so fast, Sherlock.

      xargs doesn't handle shell functions, only external binaries.

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    31. Re:What was the command? by fisted · · Score: 1

      Well I don't know your preferred shell, but I suspect updating servers isn't implemented as shell built-ins, so we're good ;)

      --
      CLI paste? paste.pr0.tips!
    32. Re:What was the command? by TCM · · Score: 1

      Are you referring to the zsh option which also wouldn't protect you from rm -fr /, funny man?

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    33. Re:What was the command? by Acid-Duck · · Score: 1

      Cool story, bro.

  2. Someone post the one line command... by SeaFox · · Score: 1

    Let's save ourselves from unnecessary clickbait.

    1. Re: Someone post the one line command... by Anonymous Coward · · Score: 1

      The video is on the summit YouTube channel, but the command was ./patch

      I was there too, it was a really good presentation.

  3. They Were Only Able to Do It by Greyfox · · Score: 1

    They were only able to do it because they already had an affordable, high quality krampfor on hand. The whole thing would have fallen apart if not for that.

    --

    I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

  4. Re:that's it...thanks by Vip · · Score: 5, Interesting

    I was there. It was said in a very joking manner. From the moment he started he showed his sense of humour.

    In fact, his whole presentation was funny, amusing and had some good information.

    The idea that he showed a one line command to patch wasn't the biggest shock of the talk. (Sorry, I don't recall the command.) It was the fact that he patches the 3,500 servers ONCE A MONTH. Straight into production. This caused some questions and discussion.

    FTFA, "One of the potential challenges of constantly updating servers is the risk that applications break when new server operating system software is loaded. Glantz, however, isn't worried and noted that RHEL offers the promise of Application Binary Interface (ABI) compatibility across updates." The rest of his reasoning, and another amusing moment, is described at the end of the article.

    Vip

  5. Re: that's it...thanks by Anonymous Coward · · Score: 1

    ./patch

    but the interesting bit was the getting to that, yeah.

  6. stage management by PopeRatzo · · Score: 2

    The moment would have been perfect if he'd just dropped the mic.

    --
    You are welcome on my lawn.
  7. Re:that's it...thanks by rtb61 · · Score: 4, Insightful

    From the article the grandparent obviously did not read "Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming," as the audience erupted into boisterous applause.". So in fact top notch people skills.

    --
    Chaos - everything, everywhere, everywhen
  8. a solid business model helps. by nimbius · · Score: 4, Funny

    if its anything like my general Ikea experience, im sure the security ops team was handed a cardboard box labelled "Schelli schocc" with a 7 page manual full of bloated looking stick figures and a tiny hex wrench. they were then left to figure it out over a long night of busted knuckles and impromptu invented curse words. by dawn, either the prod environement passed a nessus scan or theyd built a bed...or both.

    --
    Good people go to bed earlier.
    1. Re:a solid business model helps. by JakartaDean · · Score: 1

      Only because I don't have mod points. I was laughing out loud 20 words in.

      --
      The subject who is truly loyal to the Chief Magistrate will neither advise nor submit to arbitrary measures (Junius)
    2. Re:a solid business model helps. by Shinobi · · Score: 4, Insightful

      If you have troubles putting together IKEA furniture, I imagine Duplo LEGO would be out of your league too...

    3. Re:a solid business model helps. by houghi · · Score: 1

      That is basically how Windows users describe Linux and that is what they use. So yeah, they were given the tools and made the thing themselves from components they bought (from RedHat)

      --
      Don't fight for your country, if your country does not fight for you.
    4. Re:a solid business model helps. by Bob+the+Super+Hamste · · Score: 2

      Well got the joy of putting together an IKEA loft bed without instructions. The model isn't sold anymore, I couldn't find the instructions online, and to add further insult to injury I didn't even have a picture of what is was suppose to look like. I did get it together correctly but it took longer than it should have, especially since I was initially told it was a bunk bed. The lesson I learned from that was don't let the wife buy stuff from her friends that I will have to deal with.

      --
      Time to offend someone
    5. Re:a solid business model helps. by LongearedBat · · Score: 1

      Sooo... you're a software developer, right?

    6. Re:a solid business model helps. by Bob+the+Super+Hamste · · Score: 1

      Well they had lots of the older instructions online but not for this loft bed. It just may have been too old for even that. I really haven't had a complaint about their stuff in general and even have some of their inexpensive pine shelves. By the way their pine furniture looks awesome if you sand it, stain it, and apply a couple of coats of poly, and while it is a bit more expensive than the particle board stuff it is a lot nicer especially with a good finish applied and will last longer.

      Yes I realize this is IKEA furniture, just because it is inexpensive and sold in flat pack doesn't mean it has to be complete crap. It just gets to live in the basement in rooms where hobbies are done where utility is more important that overall niceness.

      --
      Time to offend someone
  9. Configuration management by rminsk · · Score: 1

    So he is using some sort of configuration management. I modified and tested a puppet manifest and then deployed to to our production puppet server. Over the next 30 minutes I had updated over 1000 machines.

    1. Re:Configuration management by silas_moeckel · · Score: 1

      Shellshock took less than 4 hours to fix across 20k hardware boxes and many many vm's. Most of that was testing the puppet manifest.

      --
      No sir I dont like it.
  10. Re:Now we have ad-news? by amiga3D · · Score: 2

    I like Apple propaganda. It's much better than that awful Windoze propaganda.

  11. Just like their furniture by Tablizer · · Score: 3, Funny

    How IKEA Patched Shellshock

    By making the customers do most of it themselves.

    --
    Table-ized A.I.
    1. Re:Just like their furniture by Tablizer · · Score: 1

      Whaddya mean? They made me put in just about every screw and peg on a bookshelf I bought, not just the "last one". I wish it was only the last few.

      Note they couldn't pack it into a flat box if they did much of the construction themselves.

      --
      Table-ized A.I.
  12. In other news by belthize · · Score: 4, Insightful

    Man holding hammer demonstrates ease of driving a nail into wood. Thousands holding screwdrivers are amazed.

  13. Shellshock by Anonymous Coward · · Score: 1

    was is "chsh -s dash www_data"?

  14. Re:Now we have ad-news? by spongman · · Score: 2

    I like Apple propaganda. And hypnotoad.

  15. Re:that's it...thanks by Mats+Svensson · · Score: 1

    If tugboats were bigger, they could be the boats that tugboats tug.

  16. Re:wrong approach? by Anonymous Coward · · Score: 1, Insightful

    So, what you are saying is I haven't bothered to read anything, or look at anything, but here is my completely irrelevant opinion?

    Man, this place used to be something...

  17. Re:that's it...thanks by hcs_$reboot · · Score: 1

    It was in Perl:
    ./update-all-3500-servers-at-once.pl
    one line.

    --
    Slashdot, fix the reply notifications... You won't get away with it...
  18. Re: that's it...thanks by JohnVanVliet · · Score: 1

    the article did not say what it was , but anyone with redhat experience already KNOWS this
    as root do ...
    " yum update "

    two words , that is it

    --
    "I don't pitch OpenSUSE Linux to my friends, i let Microsoft do it for me
  19. Why a oneliner? by houghi · · Score: 1

    Why use a onelinerand what is in that oneliner?
    I would use a script or a program to run it. Thta can be run as a 'oneliner'.
    `sh /usr/local/bin/IKEA-Update` is also a onliner.

    It is also not importand what is in that oneliner. Is it the standard update, or does it contain their own command with 360 different programs in it, subroutines and numerous other points of failure.

    --
    Don't fight for your country, if your country does not fight for you.
  20. Re: Ikea running RH? by Anonymous Coward · · Score: 2, Insightful

    Professionals look and dress like professionals. If you insist on wearing grubby t-shirts and faded jeans at work don't be surprised if you're always kept out of the loop, never ever considered for promotion and ultimately the first to be let go when downsizing.

  21. News for nerds? by ruir · · Score: 1

    OMG, IKEA uses RH enterprise support for managing their servers... Slash *used* to be news for nerds. I have used scripts, after that RunDeck and now Ansible + Debian. And they do not need a subscription and better yet, are *distribution agnostic*.

    1. Re:News for nerds? by neurovish · · Score: 1

      OMG, IKEA uses RH enterprise support for managing their servers... Slash *used* to be news for nerds. I have used scripts, after that RunDeck and now Ansible + Debian. And they do not need a subscription and better yet, are *distribution agnostic*.

      Do you manage 3500 servers for a company with $32.65 billion in revenue?

    2. Re:News for nerds? by ruir · · Score: 1

      Have you ever seen a devop presentation from facebook or better yet twitter techs? This piece of infomercial is rubbish.

  22. Re: that's it...thanks by tomknight · · Score: 1

    Well, I sure as hell wouldn't run that on all my production systems without a wee bit of testing first...

    --
    Oh arse
  23. Ob by Hognoxious · · Score: 1

    # find /placewithtaxes -iregex ".*\(money\|geld\|argent\).*" -exec mv '{}' /offshore \;

    --
    Confucius say, "Find worm in apple - bad. Find half a worm - worse."
  24. Re: Ikea running RH? by goarilla · · Score: 1

    Sad but true. If you want to get taken seriously you need to put your custom on.

  25. Re:With an advertisement for RHEL... by ruir · · Score: 1

    Oh, but there is of course. We upgraded our 3k servers easily because we have a RH enterprise account. ;) The only interesting bit was we have all the procedure documented, but then they contradicted himselves and say the man goes full comando and updates everything live without testing. Apart from that, it is drivel.

  26. Re:With an advertisement for RHEL... by AK+Marc · · Score: 1

    the man goes full comando and updates everything live without testing.

    That's an assumption on your part. Sure, it may be implied, but isn't confirmed. I've seen places large enough that their OS provider would test on their behalf. So he can claim "no testing" and the answer is it was tested. Well tested. I've seen it done before.

    --
    Learn to love Alaska
  27. How quickly we forget Y2K by anorlunda · · Score: 1

    If the heyday of Y2K remediation, I helped set up a push of a SOE to 275,000 distributed PCs in a weekend. It went off without a hitch. Management was happy, but the cries of thousands of employees who lost all their personal files and documents were ignored.

    If you are willing to be heavy handed and brutal, you can accomplish miracles. Surely there is no news in that.

    1. Re:How quickly we forget Y2K by SuiteSisterMary · · Score: 1

      I think the idea is, if you have a SOE from the get-go, you don't need to be brutal.

      --
      Vintage computer games and RPG books available. Email me if you're interested.
  28. Re: that's it...thanks by cygnwolf · · Score: 1

    Any chance for a link to the video?

    --
    Free Pie! The Pie is Also Evil!
  29. Re:So what? by silas_moeckel · · Score: 1

    You have obviously never worked with your average big corp windows admin.

    --
    No sir I dont like it.
  30. Re: that's it...thanks by mrclevesque · · Score: 1

    https://www.youtube.com/watch?...

    -- Red Hat security in a post-Shellshock world - 2015 Red Hat Summit

  31. Re: that's it...thanks by jrumney · · Score: 1

    With 3500 servers, its probably worth setting up your own package archive. Then the command to patch all the servers would most likely be pushing your tested and approved package to your local archive to be pulled by all the production servers on their next poll for updates.

  32. Re: Ikea running RH? by cinky · · Score: 1

    yes, nothing like running thousands of machine without support from the OS devs. lot's of fun...

  33. This is what Ops is really about by plopez · · Score: 1

    "The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years."

    And why I regard DevOps as a disaster in the making. While "DevOps" isn't bad for small companies, like ones I've worked for, where you 'wear many hats' or a rapidly moving R and D environment it is very dangerous in a real production environment. Of course clueless management will use "DevOps" as a cost cutting measure and then after the disaster fire everyone and outsource everything, often with even worse results, for what is essentially bad management.

    But hey, they were Agile, Nimble, flexible, idiot sourced, and buzz word compliant.

    --
    putting the 'B' in LGBTQ+
  34. Re:wrong approach? by multi+io · · Score: 1

    I would've "bothered", but the talk isn't available online, apparently. So by your logic, nobody should comment anything here. Or /. shouldn't link to articles that are essentially just teasers.

  35. With satellite, it's easy by ebvwfbw · · Score: 1

    Go to satellite, click on errata, set it to update. If you have it set up for communications Ikea would probably have been done in a half hour at the most. Otherwise, when they check in. Up to 4 hours later.

    What's the big deal?

  36. The solution? by chris_clay · · Score: 1

    That article in the link is one of the worst I have ever read. No details are given about how they patched their systems. I'm assuming (like others) that they used "yum" to install the update. But no details are given about exactly what they did or how they handled it. Don't waste your time with the link.