Slashdot Mirror
How IKEA Patched Shellshock
jones_supa writes: Magnus Glantz, IT manager at IKEA, revealed that the Swedish furniture retailer has more than 3,500 Red Hat Enterprise Linux servers. With Shellshock, every single one of those servers needed to be patched to limit the risk of exploitation. So how did IKEA patch all those servers? Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming." On a more serious note, he said that it took approximately two and half hours to upgrade their infrastructure to defend against Shellshock. The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years.
yum update -y && reboot
You're going to type that on 3500 servers?
I think you'll want to use your configuration management platform to kick off the update. That's how we did it -- applied the update to the dev servers, did some testing, then the same to qa, then preprod, then finally to the production servers. Took us more than 2.5 hours to test and validate everywhere, but actually pushing out the patch to 1200 servers was a single line command.
I was there. It was said in a very joking manner. From the moment he started he showed his sense of humour.
In fact, his whole presentation was funny, amusing and had some good information.
The idea that he showed a one line command to patch wasn't the biggest shock of the talk. (Sorry, I don't recall the command.) It was the fact that he patches the 3,500 servers ONCE A MONTH. Straight into production. This caused some questions and discussion.
FTFA, "One of the potential challenges of constantly updating servers is the risk that applications break when new server operating system software is loaded. Glantz, however, isn't worried and noted that RHEL offers the promise of Application Binary Interface (ABI) compatibility across updates." The rest of his reasoning, and another amusing moment, is described at the end of the article.
Vip
The article says they're using a Red Hat Satellite server and so if they wanted to run `yum update -y && init 6` on all of their systems, they could just push that out as a remote command to the systems / groups of systems. In Satellite, you can push out remote commands to groups of systems, so if they have their systems grouped, it would be an easy process to push that command to all of their systems.
From the article the grandparent obviously did not read "Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming," as the audience erupted into boisterous applause.". So in fact top notch people skills.
Chaos - everything, everywhere, everywhen
if its anything like my general Ikea experience, im sure the security ops team was handed a cardboard box labelled "Schelli schocc" with a 7 page manual full of bloated looking stick figures and a tiny hex wrench. they were then left to figure it out over a long night of busted knuckles and impromptu invented curse words. by dawn, either the prod environement passed a nessus scan or theyd built a bed...or both.
Good people go to bed earlier.
By making the customers do most of it themselves.
Table-ized A.I.
Man holding hammer demonstrates ease of driving a nail into wood. Thousands holding screwdrivers are amazed.