Slashdot Mirror

← Back to Stories (view on slashdot.org)

How IKEA Patched Shellshock

Posted by samzenpus on from the protect-ya-neck dept.

jones_supa writes: Magnus Glantz, IT manager at IKEA, revealed that the Swedish furniture retailer has more than 3,500 Red Hat Enterprise Linux servers. With Shellshock, every single one of those servers needed to be patched to limit the risk of exploitation. So how did IKEA patch all those servers? Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming." On a more serious note, he said that it took approximately two and half hours to upgrade their infrastructure to defend against Shellshock. The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years.

154 comments

  1. What was the command? by Anonymous Coward · · Score: 1

    I imagine it was sudo rm -rf /, but I could be way off.

    1. Re:What was the command? by Joe_Dragon · · Score: 1

      yum update -y && reboot

    2. Re:What was the command? by hawguy · · Score: 4, Informative

      yum update -y && reboot

      You're going to type that on 3500 servers?

      I think you'll want to use your configuration management platform to kick off the update. That's how we did it -- applied the update to the dev servers, did some testing, then the same to qa, then preprod, then finally to the production servers. Took us more than 2.5 hours to test and validate everywhere, but actually pushing out the patch to 1200 servers was a single line command.

    3. Re: What was the command? by Anonymous Coward · · Score: 0

      # find allen_key

    4. Re:What was the command? by Anonymous Coward · · Score: 0

      Here, scheduling the reboot of the 900 servers was the longest part of that patching effort.

    5. Re:What was the command? by Anonymous Coward · · Score: 0

      it's sad that the article doesn't tell us the extremely funny command.

    6. Re:What was the command? by Trogre · · Score: 1

      Well I'd wrap it in a loop of some kind:

      for host in `cat /dev/storage/admin/servers.dat`; do ssh root@$host "yum update -y && reboot"; done

      --
      "Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
    7. Re:What was the command? by Acid-Duck · · Score: 2

      Why not do it the way our ancestors did it? :P

      for i in $(cat ips.txt); do
      XXXXXXXXX
      done;

    8. Re:What was the command? by Anonymous Coward · · Score: 0

      $ mco package bash update .... what more do you need?

      puppet + mcollective has rendered all recent security vulnerabilities a non-issue.

    9. Re:What was the command? by pmgst17 · · Score: 4, Informative

      The article says they're using a Red Hat Satellite server and so if they wanted to run `yum update -y && init 6` on all of their systems, they could just push that out as a remote command to the systems / groups of systems. In Satellite, you can push out remote commands to groups of systems, so if they have their systems grouped, it would be an easy process to push that command to all of their systems.

    10. Re:What was the command? by Anonymous Coward · · Score: 0

      No, the humor comes from the idea that instead of holding a long presentation he would just walk away with a single command. The command itself wasn't necessarily funny.

    11. Re:What was the command? by hawguy · · Score: 2

      Well I'd wrap it in a loop of some kind:

      for host in `cat /dev/storage/admin/servers.dat`; do ssh root@$host "yum update -y && reboot"; done

      You're going to watch the output for 1000+ servers to see which ones failed?

    12. Re:What was the command? by Anonymous Coward · · Score: 0

      All these single points of failure. This sort of one and done behavior with one auth to rule them all is what security nightmares are made of.

    13. Re:What was the command? by TCM · · Score: 2

      You mean in an amateurish way that can overload shell buffers?

      Try

      while read i; do ...; done < ips.txt

      or

      xargs ... < ips.txt

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    14. Re:What was the command? by ArcherB · · Score: 1

      yum update -y && reboot

      Actually, it kicked off a bash script that consisted of 100,000 commands that took a team of programmers six months to write and debug. But to him, management, it was just a single command that he typed in and took all the credit.

      (it's a joke people)

      --
      There is no "I disagree" mod for a reason. Flamebait, Troll, and Overrated are not substitutes.
    15. Re: What was the command? by Anonymous Coward · · Score: 0

      Yes, it's funny cuz it's true.

    16. Re:What was the command? by lucm · · Score: 1

      this is why God invented Ansible.

      --
      lucm, indeed.
    17. Re:What was the command? by Anonymous Coward · · Score: 0

      I think you missed your Docker containers. Thank you, come again.

    18. Re:What was the command? by Anonymous Coward · · Score: 0

      mv *.* /dev/null

    19. Re:What was the command? by rossz · · Score: 1

      We're currently evaluating Ansible. I expect us to make the switch permanently as part of our move to docker containers. Currently, our puppet manifests are unwieldy and a biatch to maintain.

      --
      -- Will program for bandwidth
    20. Re: What was the command? by MobileTatsu-NJG · · Score: 1

      Your joke mighta been funny if it had contained a humorous punchline.

      --

      "I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)

    21. Re:What was the command? by Bert64 · · Score: 1

      What about files which don't contain a . character?

      --
      http://spamdecoy.net - free throwaway anonymous email - avoid spam!
    22. Re:What was the command? by Anonymous Coward · · Score: 1

      We keep those.

    23. Re:What was the command? by davester666 · · Score: 0

      sudo rm -rf /

      should catch the stragglers

      --
      Sleep your way to a whiter smile...date a dentist!
    24. Re:What was the command? by Jupix · · Score: 2

      If you don't mind my asking, what's the difference between QA and preprod for you?

    25. Re:What was the command? by cinky · · Score: 1

      And how will you handle output from those servers? random errors? or will you just fire it up and hope for the best? I'd suggest using puppet or some similar configuration management tool...

    26. Re:What was the command? by sys64764 · · Score: 2

      duh yeah! Thats why we have intern's!

    27. Re:What was the command? by Anonymous Coward · · Score: 0

      sudo rm -rf /

      should catch the idiots

      There FTFY. I couldn't be bothered fixing the command you clearly don't understand (but you win the "most cli wrong" award) so that it'd actually work with idiots. (the only way to educate idiots like you is with continuous application of the club of knowledge).
      But if it had been written properly - it still will only work with idiots.

      grep rm ~/.bashrc
      alias rm='rm -i'

      Lesser idiots e.g. real sysdamins use that (and setopt rmstarwait for zsh). That's aside from chattr, SELinux/AppArmour/other security frameworks to prevent that sort of "saw it on the intertubes, disengage brain, do $stupid)". And... have you heard of backups?

      Demonoid-Penguin - moderating. I only moderate up - but I'm tempted to make an exception for "the davester"

    28. Re:What was the command? by Anonymous Coward · · Score: 0

      Here, scheduling the reboot of the 900 servers was the longest part of that patching effort.

      O'Reilly? You had to reboot? And you still get paid as a sysadmin?!!(sigh).

      Demonoid-Penguin - moderating (the non-stupid).

    29. Re:What was the command? by stderr_dk · · Score: 1

      mv *.* /dev/null

      With only one matching file, you'll get:

      mv: inter-device move failed: `foo.bar' to `/dev/null'; unable to remove target: Permission denied

      If you got more than one file matching that pattern, you'll get:

      mv: target `/dev/null' is not a directory

      But thanks for playing...

      --
      alias sudo="echo make it yourself #" ; # https://pipedot.org/~stderr & http://soylentnews.org/~stderr
    30. Re:What was the command? by pz · · Score: 2

      Indeed, you definitely do NOT want hundreds-to-thousands of servers doing an update all at the same time, or, worse, rebooting all at the same time. The first has the potential to saturate your network and bring the entire setup to its knees, and the second will blow your rack supplies. I speak from experience on the latter, having been the one who identified the issue with our weekly DB scrubbing procedure once the company I was working for grew to more than a half dozen servers.

      You want to stagger things by a few 10s of seconds per server on each rack to avoid power supply issues.

      --

      Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
    31. Re:What was the command? by paradxum · · Score: 1

      ok fine... try:

      for i in {1..3500}; do ssh server$i yum update -y; ssh server$i reboot; done

      better?

    32. Re:What was the command? by dreamchaser · · Score: 1

      In some enterprise shops it is just SOP to reboot, usually a policy written by some change management managerial type who doesn't know when a reboot is actually required.

    33. Re:What was the command? by Anonymous Coward · · Score: 0

      for server in `cat servers` ; do ; ssh root@$server yum update -y \&\& reboot ; done

    34. Re:What was the command? by Anonymous Coward · · Score: 0

      I'd guess QA is just QA and preprod might be user acceptance testing

    35. Re: What was the command? by jrumney · · Score: 1

      For the more security conscious, a safer option is sudo dd /dev/zero /dev/sda

    36. Re: What was the command? by Anonymous Coward · · Score: 0

      I would combine your version with that of previous AC.

      sudo dd /dev/zero /dev/null

      That should be pretty safe.

    37. Re:What was the command? by fisted · · Score: 1

      Real sysadmins

      a) think before executing potentially disastrous commands, and therefore tend to not need the rm -i crutch
      b) automate the repetitive parts of their jobs, in which rm -i obviously does not make sense
      c) don't experiment around on production servers
      d) have arranged their systems so that accidentally removing stuff can be recovered from.

      Thanks for playing, though

      --
      CLI paste? paste.pr0.tips!
    38. Re:What was the command? by fisted · · Score: 1

      while read i; do ...; done < ips.txt

      How amateurish to spawn an unnecessary subshell.

      xargs ... < ips.txt

      Yes.

      --
      CLI paste? paste.pr0.tips!
    39. Re:What was the command? by TCM · · Score: 2

      If you alias rm to rm -i, what do you think rm -fr gets expanded to?

      Could it be rm -i -fr in which case the -f overrides the -i anyway? Oh great sysadmin, can you clarify?

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    40. Re:What was the command? by Anonymous Coward · · Score: 1

      Rebooting regularly is good practice to ensure your servers are capable of coming back up if something accidentally knocks them down unexpectedly. See also Netflix's Chaos Monkey for a different but similar concept.

    41. Re:What was the command? by Anonymous Coward · · Score: 0

      The reboot is actually all most sysadmins can do. I worked in a company, where the outsourced IT guy tried to recover files lost by SVN corruption by rebooting the server multiple times. And when the miraculous file rebirth did not happen, the IT department raised their hands. Of course, they had been unable to do the backups too..

    42. Re:What was the command? by Anonymous Coward · · Score: 0

      yum update -y && reboot

      NO! NO! NO! WTF IKEA??? You just applied ALL available patches and upgrades to your systems.

      'yum update bash -y'

      That should do the trick. I apologize, I don't recall rebooting or not. I don't think you needed to reboot. Anyway, you're best off not typing in the command at all and using RHEL Satellite to push the update out. Then you can review the reports to make sure all systems received the update.

    43. Re:What was the command? by Anonymous Coward · · Score: 0

      Why would you use Satellite to initiate yum update -y && init 6 from the command line? You can simply select the bash patch and distribute to any or all of the subscribing systems. You can also issue a reboot from Satellite.

      Issuing a yum update command within Satellite us kind of like using your cell phone to call your cell phone to talk to yourself.

    44. Re:What was the command? by neurovish · · Score: 1

      Here, scheduling the reboot of the 900 servers was the longest part of that patching effort.

      O'Reilly? You had to reboot? And you still get paid as a sysadmin?!!(sigh).

      Demonoid-Penguin - moderating (the non-stupid).

      If you're just running a generic "yum update", then you have pretty good chances a new kernel will be pulled in...so yeah a reboot was probably called for.

    45. Re:What was the command? by neurovish · · Score: 1

      Indeed, you definitely do NOT want hundreds-to-thousands of servers doing an update all at the same time, or, worse, rebooting all at the same time. The first has the potential to saturate your network and bring the entire setup to its knees, and the second will blow your rack supplies. I speak from experience on the latter, having been the one who identified the issue with our weekly DB scrubbing procedure once the company I was working for grew to more than a half dozen servers.

      You want to stagger things by a few 10s of seconds per server on each rack to avoid power supply issues.

      Man....I'd forgotten about the PDUs. Had that problem at one place where I brought down the DMZ because I rebooted a server. Fortunately that got a much needed datacenter review underway and people started distributing power correctly.
       

    46. Re:What was the command? by psyclone · · Score: 1

      pdsh FTW

    47. Re: What was the command? by Anonymous Coward · · Score: 0

      I saw someone do that (well "sudo mv junkfile /dev/null").

      From his perspective, it worked. Junkfile was gone. But a few hours later, his filesystem was full. /dev/null was now a plain file, not a device. Everything that wrote to it in append mode just contributed to filing the disk.

      His next lesson was mknod.

    48. Re:What was the command? by Trogre · · Score: 1

      Well, no. You'd run that inside a screen session, and with an ampersand not a semicolon.

      --
      "Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
    49. Re: What was the command? by Anonymous Coward · · Score: 0

      Yum and apt do not have the ability to tell you that you need to reboot.

    50. Re:What was the command? by Anonymous Coward · · Score: 0

      If you alias rm to rm -i, what do you think rm -fr gets expanded to?

      Could it be rm -i -fr in which case the -f overrides the -i anyway? Oh great sysadmin, can you clarify?

      Like your reading skills - it translates to nothing. When your lips stop hurting try reading the rest of the post - oh great armchair sysadmin

    51. Re:What was the command? by Anonymous Coward · · Score: 0

      O'Reilly? You had to reboot? And you still get paid as a sysadmin?!!(sigh).

      Demonoid-Penguin - moderating (the non-stupid).

      If you're just running a generic "yum update", then you have pretty good chances a new kernel will be pulled in...so yeah a reboot was probably called for.

      If, yes. Which ignores the instance being discussed - where the company pays for Red Hat support with a premium SLA (and no, I have no opinion on their choice). That includes ksplice.

      You can be reasonably certain that later the same day they would, after some testing, begun reboots (when each server was not at peak demand). He was talking about immediate deployment of patches - not just fast deployment i.e. minimal disruption to normal services without putting off security updates.

      D-P

    52. Re: What was the command? by Anonymous Coward · · Score: 0

      Yum and apt do not have the ability to tell you that you need to reboot.

      Can't speak for yum as I don't have much recent experience at the operator level - though I find your claim highly unlikely. As for apt - you are full of shit, it definitely alerts you to a need for reboots.

      D-P

    53. Re:What was the command? by TCM · · Score: 1

      Not so fast, Sherlock.

      xargs doesn't handle shell functions, only external binaries.

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    54. Re:What was the command? by fisted · · Score: 1

      Well I don't know your preferred shell, but I suspect updating servers isn't implemented as shell built-ins, so we're good ;)

      --
      CLI paste? paste.pr0.tips!
    55. Re:What was the command? by Anonymous Coward · · Score: 0

      Did that same thing at my work.
      We were still on puppet 2.6 which didn't help things.

      Ansible is super awesome, but I also suspect that at least some of the massive improvements brought by the switch was from doing a clean re-write of how we automated things to make them more sane and maintainable. We also made sure to make documenting stuff a high priority.

      I love that Ansible can be used as both a big config management and deployment tool, and also as a framework for small one-off scripts that you need to run on a bunch of server.

    56. Re:What was the command? by TCM · · Score: 1

      Are you referring to the zsh option which also wouldn't protect you from rm -fr /, funny man?

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    57. Re:What was the command? by Acid-Duck · · Score: 1

      Cool story, bro.

  2. Someone post the one line command... by SeaFox · · Score: 1

    Let's save ourselves from unnecessary clickbait.

    1. Re:Someone post the one line command... by Anonymous Coward · · Score: 0

      Nothing's been posted from his presentation yet. So far, you'd have to have been at the Red Hat Summit to know the one-liner.

    2. Re: Someone post the one line command... by Anonymous Coward · · Score: 1

      The video is on the summit YouTube channel, but the command was ./patch

      I was there too, it was a really good presentation.

    3. Re:Someone post the one line command... by Anonymous Coward · · Score: 0

      yum update --cve CVE-2014-7169 ?

    4. Re:Someone post the one line command... by MobileTatsu-NJG · · Score: 0

      You're averse to clickbait so you fequent Slashdot?

      --

      "I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)

  3. They Were Only Able to Do It by Greyfox · · Score: 1

    They were only able to do it because they already had an affordable, high quality krampfor on hand. The whole thing would have fallen apart if not for that.

    --

    I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

  4. Re:Now we have ad-news? by Anonymous Coward · · Score: 0

    You'd rather read Microsoft or Apple propaganda all day?

  5. Re:that's it...thanks by Vip · · Score: 5, Interesting

    I was there. It was said in a very joking manner. From the moment he started he showed his sense of humour.

    In fact, his whole presentation was funny, amusing and had some good information.

    The idea that he showed a one line command to patch wasn't the biggest shock of the talk. (Sorry, I don't recall the command.) It was the fact that he patches the 3,500 servers ONCE A MONTH. Straight into production. This caused some questions and discussion.

    FTFA, "One of the potential challenges of constantly updating servers is the risk that applications break when new server operating system software is loaded. Glantz, however, isn't worried and noted that RHEL offers the promise of Application Binary Interface (ABI) compatibility across updates." The rest of his reasoning, and another amusing moment, is described at the end of the article.

    Vip

  6. Re: that's it...thanks by Anonymous Coward · · Score: 1

    ./patch

    but the interesting bit was the getting to that, yeah.

  7. Re:that's it...thanks by Anonymous Coward · · Score: 0

    for the IT-asshole quote that we all know. ZERO people skills.

    Actually he presented a great sense of humor when he said that "That's it, thanks for coming." Like the only needed fix was some kind of fix-all-servers command. That is people skills in my book.

  8. stage management by PopeRatzo · · Score: 2

    The moment would have been perfect if he'd just dropped the mic.

    --
    You are welcome on my lawn.
    1. Re:stage management by Anonymous Coward · · Score: 0

      KING BOB!

  9. Re:that's it...thanks by rtb61 · · Score: 4, Insightful

    From the article the grandparent obviously did not read "Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming," as the audience erupted into boisterous applause.". So in fact top notch people skills.

    --
    Chaos - everything, everywhere, everywhen
  10. Re:that's it...thanks by Anonymous Coward · · Score: 0

    You think you're more of a people person? Prove it bitch.

  11. Re:that's it...thanks by Anonymous Coward · · Score: 0

    Why do you sound like an IT asshole with zero people skills?

  12. a solid business model helps. by nimbius · · Score: 4, Funny

    if its anything like my general Ikea experience, im sure the security ops team was handed a cardboard box labelled "Schelli schocc" with a 7 page manual full of bloated looking stick figures and a tiny hex wrench. they were then left to figure it out over a long night of busted knuckles and impromptu invented curse words. by dawn, either the prod environement passed a nessus scan or theyd built a bed...or both.

    --
    Good people go to bed earlier.
    1. Re:a solid business model helps. by Anonymous Coward · · Score: 0

      Why is this not +5 funny?!!!!

    2. Re:a solid business model helps. by Anonymous Coward · · Score: 0

      Nobody has understood the stick-figures for doing that yet

    3. Re:a solid business model helps. by JakartaDean · · Score: 1

      Only because I don't have mod points. I was laughing out loud 20 words in.

      --
      The subject who is truly loyal to the Chief Magistrate will neither advise nor submit to arbitrary measures (Junius)
    4. Re:a solid business model helps. by Anonymous Coward · · Score: 0

      It isn't all that funny because "bloated looking stick figures" doesn't make any sense, and trips the reader up. Also "Impromptu invented curse words" is a fairly ungainly phrase. It could have been funny, but wasn't written by a very funny person, and so it wasn't.

      AC because I'm being horrible.

    5. Re:a solid business model helps. by Shinobi · · Score: 4, Insightful

      If you have troubles putting together IKEA furniture, I imagine Duplo LEGO would be out of your league too...

    6. Re:a solid business model helps. by houghi · · Score: 1

      That is basically how Windows users describe Linux and that is what they use. So yeah, they were given the tools and made the thing themselves from components they bought (from RedHat)

      --
      Don't fight for your country, if your country does not fight for you.
    7. Re:a solid business model helps. by Bob+the+Super+Hamste · · Score: 2

      Well got the joy of putting together an IKEA loft bed without instructions. The model isn't sold anymore, I couldn't find the instructions online, and to add further insult to injury I didn't even have a picture of what is was suppose to look like. I did get it together correctly but it took longer than it should have, especially since I was initially told it was a bunk bed. The lesson I learned from that was don't let the wife buy stuff from her friends that I will have to deal with.

      --
      Time to offend someone
    8. Re:a solid business model helps. by Anonymous Coward · · Score: 0

      plot twist: it was really a lamp

    9. Re:a solid business model helps. by Anonymous Coward · · Score: 0

      The model isn't sold anymore, I couldn't find the instructions online, and to add further insult to injury I didn't even have a picture of what is was suppose to look like.

      Should have called them, sometimes they are surprisingly helpful. While it depends highly on who you get to talk to there still are some people at IKEA that understands that it could be beneficial for them to help customers and non-customers alike.

      Now that you mentioned it it seems like an oversight on their side to not have the old assembly instructions online.

    10. Re:a solid business model helps. by LongearedBat · · Score: 1

      Sooo... you're a software developer, right?

    11. Re:a solid business model helps. by Bob+the+Super+Hamste · · Score: 1

      Well they had lots of the older instructions online but not for this loft bed. It just may have been too old for even that. I really haven't had a complaint about their stuff in general and even have some of their inexpensive pine shelves. By the way their pine furniture looks awesome if you sand it, stain it, and apply a couple of coats of poly, and while it is a bit more expensive than the particle board stuff it is a lot nicer especially with a good finish applied and will last longer.

      Yes I realize this is IKEA furniture, just because it is inexpensive and sold in flat pack doesn't mean it has to be complete crap. It just gets to live in the basement in rooms where hobbies are done where utility is more important that overall niceness.

      --
      Time to offend someone
  13. Configuration management by rminsk · · Score: 1

    So he is using some sort of configuration management. I modified and tested a puppet manifest and then deployed to to our production puppet server. Over the next 30 minutes I had updated over 1000 machines.

    1. Re:Configuration management by silas_moeckel · · Score: 1

      Shellshock took less than 4 hours to fix across 20k hardware boxes and many many vm's. Most of that was testing the puppet manifest.

      --
      No sir I dont like it.
  14. Re:Now we have ad-news? by amiga3D · · Score: 2

    I like Apple propaganda. It's much better than that awful Windoze propaganda.

  15. Cheats by Anonymous Coward · · Score: 0

    Sure ./updateIkeaServers is one line.

    But that's cheating if it's calling a 5 million line script.....

  16. Re: Ikea running RH? by Anonymous Coward · · Score: 0

    people with no enterprise experience stick out like a sore thumb in threads like these.

  17. Just like their furniture by Tablizer · · Score: 3, Funny

    How IKEA Patched Shellshock

    By making the customers do most of it themselves.

    --
    Table-ized A.I.
    1. Re:Just like their furniture by Anonymous Coward · · Score: 0

      The brilliance of IKEA is that they build the furniture, then ask the customer to put in the last screw.
      Customers that are easily manipulated will think they did most of the work and feel a sense of accomplishment.
      Also known as the IKEA effect

    2. Re:Just like their furniture by Tablizer · · Score: 1

      Whaddya mean? They made me put in just about every screw and peg on a bookshelf I bought, not just the "last one". I wish it was only the last few.

      Note they couldn't pack it into a flat box if they did much of the construction themselves.

      --
      Table-ized A.I.
  18. In other news by belthize · · Score: 4, Insightful

    Man holding hammer demonstrates ease of driving a nail into wood. Thousands holding screwdrivers are amazed.

  19. Re: Ikea running RH? by Anonymous Coward · · Score: 0

    Lol.

    You sound like a complete corporate shill.

    Let me guess... You have to wear a suit to work everyday, because a "manager" told you to?

    Gee, how did I know???

  20. What was the command? by Anonymous Coward · · Score: 0

    Sad article, they didn't even show us the command!

  21. Shellshock by Anonymous Coward · · Score: 1

    was is "chsh -s dash www_data"?

    1. Re:Shellshock by Anonymous Coward · · Score: 0

      NAME
                    chsh - change login shell

      SYNOPSIS
                    chsh [options] [LOGIN]

      DESCRIPTION
                    The chsh command changes the user login shell. This determines the name
                    of the user's initial login command. A normal user may only change the
                    login shell for her own account; the superuser may change the login
                    shell for any account.

      OPTIONS
                    -s, --shell SHELL
                            The name of the user's new login shell. Setting this field to blank
                            causes the system to select the default login shell.

  22. Re: that's it...thanks by Anonymous Coward · · Score: 0

    AC here, frequently a facetious troll, but seriously, is that ABI guarantee that reliable ?

  23. Re:Now we have ad-news? by Anonymous Coward · · Score: 0

    This so much. Why do we have to hear about the Linux garbage? It hasn't been a relevant OS for ages.

    BSD for the servers, Windows for the desktop. These are the professional choices.

  24. Re:Now we have ad-news? by Anonymous Coward · · Score: 0

    Dude, they patched 3500 linux servers in two hours from a command line because they have good automation tools.

    Perhaps you're just too fucking stupid to understand why this is interesting.

    If so, then kindly shut the fuck up and go watch cat videos and leave the rest of us to not have to put up with your bullshit.

  25. Re:Now we have ad-news? by Anonymous Coward · · Score: 0, Funny

    Apple is mainly a propaganda company, so that's no surprise. I always enjoy Linux propaganda for its amateur style and heavy use of hyperbole.

  26. wrong approach? by multi+io · · Score: 0, Troll

    I know nothing about IKEA's Linux setup and didn't see the talk, but "one-line Linux command" sounds like the wrong approach to something like this, at least if that command directly manipulates something on each server. Shell commands that an administrator issues interactively on a terminal can't be reproduced, tracked, or documented automatically. The right thing to do would probably be to change some "bash_version" parameter in the puppet hiera/chef/whatever configuration management system they use, from where the change will automatically be applied on all nodes, or use an internal rpm/yum server that all nodes install from automatically (governed, again, by the configuration management system) and upload the patched bash rpm to that.

    1. Re:wrong approach? by Anonymous Coward · · Score: 1, Insightful

      So, what you are saying is I haven't bothered to read anything, or look at anything, but here is my completely irrelevant opinion?

      Man, this place used to be something...

    2. Re:wrong approach? by Anonymous Coward · · Score: 0

      To be fair, the comments hold the same quality as the articles.

    3. Re:wrong approach? by Dog-Cow · · Score: 0

      You and the mod who chose Interesting are fucking idiots. Go kill yourselves to make the world a better place.

    4. Re:wrong approach? by multi+io · · Score: 1

      I would've "bothered", but the talk isn't available online, apparently. So by your logic, nobody should comment anything here. Or /. shouldn't link to articles that are essentially just teasers.

  27. Re:Now we have ad-news? by spongman · · Score: 2

    I like Apple propaganda. And hypnotoad.

  28. Re:that's it...thanks by Mats+Svensson · · Score: 1

    If tugboats were bigger, they could be the boats that tugboats tug.

  29. Re:that's it...thanks by hcs_$reboot · · Score: 1

    It was in Perl:
    ./update-all-3500-servers-at-once.pl
    one line.

    --
    Slashdot, fix the reply notifications... You won't get away with it...
  30. Re: that's it...thanks by JohnVanVliet · · Score: 1

    the article did not say what it was , but anyone with redhat experience already KNOWS this
    as root do ...
    " yum update "

    two words , that is it

    --
    "I don't pitch OpenSUSE Linux to my friends, i let Microsoft do it for me
  31. Why a oneliner? by houghi · · Score: 1

    Why use a onelinerand what is in that oneliner?
    I would use a script or a program to run it. Thta can be run as a 'oneliner'.
    `sh /usr/local/bin/IKEA-Update` is also a onliner.

    It is also not importand what is in that oneliner. Is it the standard update, or does it contain their own command with 360 different programs in it, subroutines and numerous other points of failure.

    --
    Don't fight for your country, if your country does not fight for you.
  32. With an advertisement for RHEL... by Anonymous Coward · · Score: 0

    Seriously, this is an embedded add for RHEL. There is no technical information beyond they use Linux Computers. The rest is all marketing for a Linux OS product. I hope eWeek, IKEA and /. at least got paid for this.

    1. Re:With an advertisement for RHEL... by ruir · · Score: 1

      Oh, but there is of course. We upgraded our 3k servers easily because we have a RH enterprise account. ;) The only interesting bit was we have all the procedure documented, but then they contradicted himselves and say the man goes full comando and updates everything live without testing. Apart from that, it is drivel.

    2. Re:With an advertisement for RHEL... by AK+Marc · · Score: 1

      the man goes full comando and updates everything live without testing.

      That's an assumption on your part. Sure, it may be implied, but isn't confirmed. I've seen places large enough that their OS provider would test on their behalf. So he can claim "no testing" and the answer is it was tested. Well tested. I've seen it done before.

      --
      Learn to love Alaska
  33. Re: Ikea running RH? by Anonymous Coward · · Score: 2, Insightful

    Professionals look and dress like professionals. If you insist on wearing grubby t-shirts and faded jeans at work don't be surprised if you're always kept out of the loop, never ever considered for promotion and ultimately the first to be let go when downsizing.

  34. News for nerds? by ruir · · Score: 1

    OMG, IKEA uses RH enterprise support for managing their servers... Slash *used* to be news for nerds. I have used scripts, after that RunDeck and now Ansible + Debian. And they do not need a subscription and better yet, are *distribution agnostic*.

    1. Re:News for nerds? by neurovish · · Score: 1

      OMG, IKEA uses RH enterprise support for managing their servers... Slash *used* to be news for nerds. I have used scripts, after that RunDeck and now Ansible + Debian. And they do not need a subscription and better yet, are *distribution agnostic*.

      Do you manage 3500 servers for a company with $32.65 billion in revenue?

    2. Re:News for nerds? by ruir · · Score: 1

      Have you ever seen a devop presentation from facebook or better yet twitter techs? This piece of infomercial is rubbish.

  35. Re: that's it...thanks by tomknight · · Score: 1

    Well, I sure as hell wouldn't run that on all my production systems without a wee bit of testing first...

    --
    Oh arse
  36. Ob by Hognoxious · · Score: 1

    # find /placewithtaxes -iregex ".*\(money\|geld\|argent\).*" -exec mv '{}' /offshore \;

    --
    Confucius say, "Find worm in apple - bad. Find half a worm - worse."
  37. Re: Ikea running RH? by goarilla · · Score: 1

    Sad but true. If you want to get taken seriously you need to put your custom on.

  38. The little country that could by Anonymous Coward · · Score: 0

    Now I wonder if the clowns over here in Circus America are too proud and stubborn to take some hints.

  39. So what? by Anonymous Coward · · Score: 0

    That's exactly how all sysadmins of the world did the upgrade. Or does anyone thinks sysadmins go system by system and apply the upgrade 3000 times?
    Most people even didn't have to issue any command.... Just approve a new package to be rolled out.

    Can't understand what's amusing of this article, appart of a obvious Red Hat ad placement.

    1. Re:So what? by silas_moeckel · · Score: 1

      You have obviously never worked with your average big corp windows admin.

      --
      No sir I dont like it.
  40. Re: that's it...thanks by Anonymous Coward · · Score: 0

    No, seriously, I was at that talk. The command was ./patch. But the talk was about explaining how they got to that point. The video is on the Red Hat Summit YouTube channel, too.

  41. How quickly we forget Y2K by anorlunda · · Score: 1

    If the heyday of Y2K remediation, I helped set up a push of a SOE to 275,000 distributed PCs in a weekend. It went off without a hitch. Management was happy, but the cries of thousands of employees who lost all their personal files and documents were ignored.

    If you are willing to be heavy handed and brutal, you can accomplish miracles. Surely there is no news in that.

    1. Re:How quickly we forget Y2K by SuiteSisterMary · · Score: 1

      I think the idea is, if you have a SOE from the get-go, you don't need to be brutal.

      --
      Vintage computer games and RPG books available. Email me if you're interested.
  42. Re: Ikea running RH? by Anonymous Coward · · Score: 0

    But but but...there's no way our culture would be so shallow!

    It's not that it's impossible to succeed while badly dressed, you're just throwing a roadblock in front of yourself. John Carmack (e.g.) gets to wear whatever he wants. J. Random Linuxuser should be wearing CK or better.

  43. Re: that's it...thanks by cygnwolf · · Score: 1

    Any chance for a link to the video?

    --
    Free Pie! The Pie is Also Evil!
  44. Re: that's it...thanks by mrclevesque · · Score: 1

    https://www.youtube.com/watch?...

    -- Red Hat security in a post-Shellshock world - 2015 Red Hat Summit

  45. Re: that's it...thanks by Anonymous Coward · · Score: 0

    My one line command was apt-get upgrade

    But YMMV if you use Red Hat based distros like IKEA.

  46. Re: that's it...thanks by jrumney · · Score: 1

    With 3500 servers, its probably worth setting up your own package archive. Then the command to patch all the servers would most likely be pushing your tested and approved package to your local archive to be pulled by all the production servers on their next poll for updates.

  47. Re: Ikea running RH? by cinky · · Score: 1

    yes, nothing like running thousands of machine without support from the OS devs. lot's of fun...

  48. Re: that's it...thanks by Anonymous Coward · · Score: 0

    I see Joshua Bressers talking not Magnus Glantz.

  49. Re: that's it...thanks by Anonymous Coward · · Score: 0

    Close - but it's not that video, that one is (Joshua Bresser's presentation ), the article refers to Magnus Glantz & Mattias Haern's presentation.

    Can't see their video in RedHat's list though.

  50. Re: that's it...thanks by Anonymous Coward · · Score: 0

    well... sort of.

    Except the article says that they're using Red Hat Satellite - so the updates were probably pushed from there.

  51. fixing these issues before they become issues by Anonymous Coward · · Score: 0

    Building systems that are secure is not hard. Implementing grsecurity.net patching on all servers, running nginx under chroot jail environments, using ssh whitelisting to prevent random IP's from entering your ssh server. These are pretty standard things. How does a user escalate privileges when they can't see any process outside their own, or can't compile and execute a piece of code outside the trusted paths of /usr/bin, /bin, etc? How do stack overflow exploits happen if the kernel prevents them (grsec). Things like "shellshock" and whatever the new thing coming out, these things don't matter when the user doing the attacking has no access to do anything except login and logout from the shell.

  52. Re: Ikea running RH? by Anonymous Coward · · Score: 0

    Very true. But that also means that you will stay at a company that thinks like that, and that promotion is more likely to be to a useless middle manager position with a raise that isn't proportional to the increased workload.
    Bad leaders doesn't recruit internally to management positions and they will focus more on appearance than performance.
    Then again, a job at a badly run company is better than no job at all.

    In my experience vendor that is all suited up and has a perfect smile does this to compensate for lack of competence. Usually the suit is just a salesperson and then you get to talk to someone who actually knows something.
    In the cases where companies have been all suits they have also been pretty sketchy. (Probably outright criminal but I didn't investigate further.)

  53. They Patched It The Dumb Way by Anonymous Coward · · Score: 0

    Step One
    Step Two

    That's patching it the smart way.

  54. for the people who really care by Anonymous Coward · · Score: 0

    https://www.youtube.com/watch?v=tke07oW5zN4

    your welcome

  55. Re:Now we have ad-news? by Anonymous Coward · · Score: 0

    except these good automation tools were available for the past 10 years. If they showed this to me in 2005 I would be very impressed. Now it's easy to do for almost anyone. cobbler for provisioning and puppet for config management work wonders, I didn't work with new config management frameworks (salt,chef etc) so I can't comment on them but I heard they are even better.

  56. Re: that's it...thanks by Anonymous Coward · · Score: 0

    It's not a Red Hat based distro, it's just a Red Hat distro.

  57. This is what Ops is really about by plopez · · Score: 1

    "The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years."

    And why I regard DevOps as a disaster in the making. While "DevOps" isn't bad for small companies, like ones I've worked for, where you 'wear many hats' or a rapidly moving R and D environment it is very dangerous in a real production environment. Of course clueless management will use "DevOps" as a cost cutting measure and then after the disaster fire everyone and outsource everything, often with even worse results, for what is essentially bad management.

    But hey, they were Agile, Nimble, flexible, idiot sourced, and buzz word compliant.

    --
    putting the 'B' in LGBTQ+
  58. Re:that's it...thanks by Anonymous Coward · · Score: 0

    Is a video of the presentation available online? So far, I've only found the entry of the presentation in the agenda for the redhat summit.

  59. Re: that's it...thanks by Anonymous Coward · · Score: 0

    That only does one machine. Neither apt-get, nor yum will login to the other 3499 machines.

    Perhaps you thought he was giving a presentation on "How to upgrade one machine".

  60. 42... no, seriously by Anonymous Coward · · Score: 0

    ikea has a, rhel server for every 42 employees (less than, actually, because they have 'more than' 3500 servers, not 'just' 3500).

    i know they have factory operations, online presence, corporate accounting/crm, multiple regions and locations across the globe, but still, that's just the rhel ones, not the servers they no doubt have that run windows or a flavor of *nix other than rhel... wtf?

  61. With satellite, it's easy by ebvwfbw · · Score: 1

    Go to satellite, click on errata, set it to update. If you have it set up for communications Ikea would probably have been done in a half hour at the most. Otherwise, when they check in. Up to 4 hours later.

    What's the big deal?

  62. The solution? by chris_clay · · Score: 1

    That article in the link is one of the worst I have ever read. No details are given about how they patched their systems. I'm assuming (like others) that they used "yum" to install the update. But no details are given about exactly what they did or how they handled it. Don't waste your time with the link.