Slashdot Mirror
How IKEA Patched Shellshock
jones_supa writes: Magnus Glantz, IT manager at IKEA, revealed that the Swedish furniture retailer has more than 3,500 Red Hat Enterprise Linux servers. With Shellshock, every single one of those servers needed to be patched to limit the risk of exploitation. So how did IKEA patch all those servers? Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming." On a more serious note, he said that it took approximately two and half hours to upgrade their infrastructure to defend against Shellshock. The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years.
yum update -y && reboot
You're going to type that on 3500 servers?
I think you'll want to use your configuration management platform to kick off the update. That's how we did it -- applied the update to the dev servers, did some testing, then the same to qa, then preprod, then finally to the production servers. Took us more than 2.5 hours to test and validate everywhere, but actually pushing out the patch to 1200 servers was a single line command.
Why not do it the way our ancestors did it? :P
for i in $(cat ips.txt); do
XXXXXXXXX
done;
I was there. It was said in a very joking manner. From the moment he started he showed his sense of humour.
In fact, his whole presentation was funny, amusing and had some good information.
The idea that he showed a one line command to patch wasn't the biggest shock of the talk. (Sorry, I don't recall the command.) It was the fact that he patches the 3,500 servers ONCE A MONTH. Straight into production. This caused some questions and discussion.
FTFA, "One of the potential challenges of constantly updating servers is the risk that applications break when new server operating system software is loaded. Glantz, however, isn't worried and noted that RHEL offers the promise of Application Binary Interface (ABI) compatibility across updates." The rest of his reasoning, and another amusing moment, is described at the end of the article.
Vip
The moment would have been perfect if he'd just dropped the mic.
You are welcome on my lawn.
The article says they're using a Red Hat Satellite server and so if they wanted to run `yum update -y && init 6` on all of their systems, they could just push that out as a remote command to the systems / groups of systems. In Satellite, you can push out remote commands to groups of systems, so if they have their systems grouped, it would be an easy process to push that command to all of their systems.
From the article the grandparent obviously did not read "Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming," as the audience erupted into boisterous applause.". So in fact top notch people skills.
Chaos - everything, everywhere, everywhen
if its anything like my general Ikea experience, im sure the security ops team was handed a cardboard box labelled "Schelli schocc" with a 7 page manual full of bloated looking stick figures and a tiny hex wrench. they were then left to figure it out over a long night of busted knuckles and impromptu invented curse words. by dawn, either the prod environement passed a nessus scan or theyd built a bed...or both.
Good people go to bed earlier.
I like Apple propaganda. It's much better than that awful Windoze propaganda.
Well I'd wrap it in a loop of some kind:
for host in `cat /dev/storage/admin/servers.dat`; do ssh root@$host "yum update -y && reboot"; done
You're going to watch the output for 1000+ servers to see which ones failed?
You mean in an amateurish way that can overload shell buffers?
Try
while read i; do ...; done < ips.txt
or
xargs ... < ips.txt
Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
By making the customers do most of it themselves.
Table-ized A.I.
Man holding hammer demonstrates ease of driving a nail into wood. Thousands holding screwdrivers are amazed.
I like Apple propaganda. And hypnotoad.
Professionals look and dress like professionals. If you insist on wearing grubby t-shirts and faded jeans at work don't be surprised if you're always kept out of the loop, never ever considered for promotion and ultimately the first to be let go when downsizing.
If you don't mind my asking, what's the difference between QA and preprod for you?
duh yeah! Thats why we have intern's!
Indeed, you definitely do NOT want hundreds-to-thousands of servers doing an update all at the same time, or, worse, rebooting all at the same time. The first has the potential to saturate your network and bring the entire setup to its knees, and the second will blow your rack supplies. I speak from experience on the latter, having been the one who identified the issue with our weekly DB scrubbing procedure once the company I was working for grew to more than a half dozen servers.
You want to stagger things by a few 10s of seconds per server on each rack to avoid power supply issues.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
If you alias rm to rm -i, what do you think rm -fr gets expanded to?
Could it be rm -i -fr in which case the -f overrides the -i anyway? Oh great sysadmin, can you clarify?
Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6