Slashdot Mirror
How IKEA Patched Shellshock
jones_supa writes: Magnus Glantz, IT manager at IKEA, revealed that the Swedish furniture retailer has more than 3,500 Red Hat Enterprise Linux servers. With Shellshock, every single one of those servers needed to be patched to limit the risk of exploitation. So how did IKEA patch all those servers? Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating "That's it, thanks for coming." On a more serious note, he said that it took approximately two and half hours to upgrade their infrastructure to defend against Shellshock. The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years.
for the IT-asshole quote that we all know. ZERO people skills.
I imagine it was sudo rm -rf /, but I could be way off.
Where can I unsubscribe from Red Hat propaganda?
Let's save ourselves from unnecessary clickbait.
They were only able to do it because they already had an affordable, high quality krampfor on hand. The whole thing would have fallen apart if not for that.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
The moment would have been perfect if he'd just dropped the mic.
You are welcome on my lawn.
They should use Debian, IMHO.
if its anything like my general Ikea experience, im sure the security ops team was handed a cardboard box labelled "Schelli schocc" with a 7 page manual full of bloated looking stick figures and a tiny hex wrench. they were then left to figure it out over a long night of busted knuckles and impromptu invented curse words. by dawn, either the prod environement passed a nessus scan or theyd built a bed...or both.
Good people go to bed earlier.
So he is using some sort of configuration management. I modified and tested a puppet manifest and then deployed to to our production puppet server. Over the next 30 minutes I had updated over 1000 machines.
Sure ./updateIkeaServers is one line.
But that's cheating if it's calling a 5 million line script.....
By making the customers do most of it themselves.
Table-ized A.I.
Man holding hammer demonstrates ease of driving a nail into wood. Thousands holding screwdrivers are amazed.
Sad article, they didn't even show us the command!
was is "chsh -s dash www_data"?
I know nothing about IKEA's Linux setup and didn't see the talk, but "one-line Linux command" sounds like the wrong approach to something like this, at least if that command directly manipulates something on each server. Shell commands that an administrator issues interactively on a terminal can't be reproduced, tracked, or documented automatically. The right thing to do would probably be to change some "bash_version" parameter in the puppet hiera/chef/whatever configuration management system they use, from where the change will automatically be applied on all nodes, or use an internal rpm/yum server that all nodes install from automatically (governed, again, by the configuration management system) and upload the patched bash rpm to that.
Why use a onelinerand what is in that oneliner? /usr/local/bin/IKEA-Update` is also a onliner.
I would use a script or a program to run it. Thta can be run as a 'oneliner'.
`sh
It is also not importand what is in that oneliner. Is it the standard update, or does it contain their own command with 360 different programs in it, subroutines and numerous other points of failure.
Don't fight for your country, if your country does not fight for you.
Seriously, this is an embedded add for RHEL. There is no technical information beyond they use Linux Computers. The rest is all marketing for a Linux OS product. I hope eWeek, IKEA and /. at least got paid for this.
OMG, IKEA uses RH enterprise support for managing their servers... Slash *used* to be news for nerds. I have used scripts, after that RunDeck and now Ansible + Debian. And they do not need a subscription and better yet, are *distribution agnostic*.
# find /placewithtaxes -iregex ".*\(money\|geld\|argent\).*" -exec mv '{}' /offshore \;
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Now I wonder if the clowns over here in Circus America are too proud and stubborn to take some hints.
Every vulnerability that's gone public can be fixed in an instant, and every one that's not can be used to infiltrate the whole infrastructure - and very efficiently too.
At least you've identified the problem, Mr. Ikea IT Manager.
That's exactly how all sysadmins of the world did the upgrade. Or does anyone thinks sysadmins go system by system and apply the upgrade 3000 times?
Most people even didn't have to issue any command.... Just approve a new package to be rolled out.
Can't understand what's amusing of this article, appart of a obvious Red Hat ad placement.
If the heyday of Y2K remediation, I helped set up a push of a SOE to 275,000 distributed PCs in a weekend. It went off without a hitch. Management was happy, but the cries of thousands of employees who lost all their personal files and documents were ignored.
If you are willing to be heavy handed and brutal, you can accomplish miracles. Surely there is no news in that.
He had to type a command.
In Microsoft land, we just right click the patch and click Deploy. Millions of servers; done!
But don't worry, you guys have somehow convinced Microsoft to reverse direction and go back to the command line. So, if you would prefer to type commands, you can type up a long script of insanely long PowerShell commands that will do the same thing in 1,000 times the time.
Building systems that are secure is not hard. Implementing grsecurity.net patching on all servers, running nginx under chroot jail environments, using ssh whitelisting to prevent random IP's from entering your ssh server. These are pretty standard things. How does a user escalate privileges when they can't see any process outside their own, or can't compile and execute a piece of code outside the trusted paths of /usr/bin, /bin, etc? How do stack overflow exploits happen if the kernel prevents them (grsec). Things like "shellshock" and whatever the new thing coming out, these things don't matter when the user doing the attacking has no access to do anything except login and logout from the shell.
Step One
Step Two
That's patching it the smart way.
https://www.youtube.com/watch?v=tke07oW5zN4
your welcome
"The key was having a consistent approach to system management, which begins with a well-defined Standard Operating Environment (SOE). Additionally, Glantz has defined a lifecycle management plan that describes the lifecycle of how Linux will be used at Ikea for the next seven years."
And why I regard DevOps as a disaster in the making. While "DevOps" isn't bad for small companies, like ones I've worked for, where you 'wear many hats' or a rapidly moving R and D environment it is very dangerous in a real production environment. Of course clueless management will use "DevOps" as a cost cutting measure and then after the disaster fire everyone and outsource everything, often with even worse results, for what is essentially bad management.
But hey, they were Agile, Nimble, flexible, idiot sourced, and buzz word compliant.
putting the 'B' in LGBTQ+
ikea has a, rhel server for every 42 employees (less than, actually, because they have 'more than' 3500 servers, not 'just' 3500).
i know they have factory operations, online presence, corporate accounting/crm, multiple regions and locations across the globe, but still, that's just the rhel ones, not the servers they no doubt have that run windows or a flavor of *nix other than rhel... wtf?
Go to satellite, click on errata, set it to update. If you have it set up for communications Ikea would probably have been done in a half hour at the most. Otherwise, when they check in. Up to 4 hours later.
What's the big deal?
That article in the link is one of the worst I have ever read. No details are given about how they patched their systems. I'm assuming (like others) that they used "yum" to install the update. But no details are given about exactly what they did or how they handled it. Don't waste your time with the link.