Slashdot Mirror
IPMI Protocol Vulnerabilities Have Long Shelf Life
msm1267 (2804139) writes "If enterprises are indeed moving services off premises and into the cloud, there are four letters those companies' IT organizations should be aware of: IPMI. Short for Intelligent Platform Management Interface, these tiny computers live as an embedded Linux system attached to the motherboards of big servers from vendors such as IBM, Dell and HP. IPMI is used by a Baseboard Management Controller (BMC) to manage Out-of-Band communication, essentially giving admins remote control over servers and devices, including memory, networking capabilities and storage. This is particularly useful for hosting providers and cloud services providers who must manage gear and data in varied locations.
Noted researchers Dan Farmer, creator of the SATAN vulnerability scanner, and HD Moore, creator of Metasploit, have been collaborating on research into the vulnerabilities present in IPMI and BMCs and the picture keeps getting uglier. Last July, Farmer and Moore published some research on the issue based upon work Farmer was doing under a DARPA Cyber Fast Track Grant that uncovered a host of vulnerabilities, and Internet-wide scans for the IPMI protocol conducted by Moore. Farmer released a paper called 'Sold Down the River,' in which he chastises big hardware vendors for ignoring security vulnerabilities and poor configurations that are trivial to find and exploit."
Noted researchers Dan Farmer, creator of the SATAN vulnerability scanner, and HD Moore, creator of Metasploit, have been collaborating on research into the vulnerabilities present in IPMI and BMCs and the picture keeps getting uglier. Last July, Farmer and Moore published some research on the issue based upon work Farmer was doing under a DARPA Cyber Fast Track Grant that uncovered a host of vulnerabilities, and Internet-wide scans for the IPMI protocol conducted by Moore. Farmer released a paper called 'Sold Down the River,' in which he chastises big hardware vendors for ignoring security vulnerabilities and poor configurations that are trivial to find and exploit."
Good thing IPMI gets some attention. IPMI doesn't seem to be very reliable at all. I was using IPMI fuzzer a while back from Codenomicon (same guys that found Heartbleed) in our data center and it was pretty sad how crappy the implementations were. Anyways - ended up disabling IPMI entirely in our environment.
Any function on the motherboard that is connected to the nic, IPMI/EFI/AMT/vPro/etc, are just back doors waiting to be kicked open (if not already opened by default).
If they go 100% Beta than this site will lose all of its users faster than water through a sieve.
Boycott Dice!
Boycott ThinkGeek!
Boycott Beta!
Y'know, TCP/IP is inherently insecure. In fact, there's no effective built-in security there. IPMI itself is not secure because the security should not be implemented there; any more than network security should not be implemented in TCP/IP. Since this is a server related issue, IPMI implementers and users are presumed to understand this. Workstation users need not concern themselves with this. What sane workstation user will pay the extra money to get hardware with RAC technology?
Since the IPMI runs Linux, and the researchers are uncovering tons of vulnerabilities on them, would it be correct to say that Linux shouldn't be used on those IPMI, and instead, Microsoft's Windows should be used, instead ?
It confounds me that still, in today's world, new technologies are designed as such:
Design core features.
Write and test/debug core features.
Then, if there is time, do a security audit and glue some security on top otherwise just release what you got.
Security must be built in from step one, step two, etc. and must be and integral part of the design.
Have we not learned our lesson yet?
Bad subject alert: the protocol itself is not vulnerable (any more than any other protocol), the problems are in the implementations (and lack of on-going support for most).
I always set up IPMI on a private VLAN, with only a couple of "trusted" hosts having access. Most things can be done with the "ipmitool" command-line program, or I can port-forward port 80 for the BMCs with a web interface. There are a few web-based BMCs with crappy Java applets for remote KVM (they mangled the VNC protocol just enough so regular VNC clients won't work); for those, I either set up a minimal X desktop VM or use a VPN to the trusted host.
Even firewalling it is not enough, unless you only let authenticated traffic from very few sources in. In any case, IPMI needs to have its own network segment, anybody putting it into the same segment as other traffic is grossly negligent and utterly incompetent.
And people putting IPMI where it is reachable from the Internet? I think that is grounds for immediate termination with a performance report that makes sure these people never do anything security-related ever again. That level of stupidity is hard to top.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
As an IT guy, I really like the concept of IPMI. I would love something like LogMeIn, but that allows you to take control of machines on a baseboard/lights-out level. The only problem is, there aren't any solutions that I'm aware of that offer that kind of easy, useful bulk management of lots of machines from a single pane. But more importantly, the concept of that kind of bulk management should trigger the thought, "Holy crap that opens a dangerous can of worms!" If lights-out management isn't secured properly, it gives an attacker a frightening level of access.
I don't know why they implemented these things without thinking it through. It's too hard to use legitimately, and too hard to manage security. I don't really even understand how I'm supposed to access and manage these systems in bulk, especially considered how often modern IT departments need to deal with remote machines that they never physically touch. If someone would develop a solution for IPMI that's not completely stupid-- think Meraki meets LogMeIn meets MDM-- an awful lot of IT departments would be falling all over themselves to buy hardware that supported it.
Mind you, doing all of this on a BMC might well crash or wedge it into a sullen silence, as they are very easy to DoS into submission even unwittingly (Iâ(TM)ve completely broken BMCs from my testing both Dell and HP servers.)
This sounds like it might be a usable workaround to plug the huge security hole, aim your own DDOS utility at it, definitely using a case of using sparkplug for a lugstud, but if it gets the job done...
am I smoking crack here or were the specification authors?
A common lament on my world.
You can have my SIG when you pry it from my cold, dead hands.
Given the numbers of attack vectors in data centers from social engineering to software to hardware faults, would you trust your company's IT data system ONLY to "cloud suppliers?"
Something like 2/3rds of small businesses that lose their digital data go out of business within 6 months, so it is a real risk issue to not have a totally local backup data system that can be brought up within a day.
Do you totally trust big cloud data centers?
Every enterprise I've worked for that uses IPMI (BMC, ALOM/ILOM, etc.) has put it on their intranet, not the internet - and as often as not, an especially inaccessible corner of the intranet.
So? That doesn't mean it shouldn't be updated and secured.
The initial point of entry for the attack against Target which caused tens of millions of CC numbers to be compromised was the network that the HVAC equipment sat on. Both Iran (Stuxnet) and the US DoD (agent.btz) had their air gaps jumped over. Networked printers have also been compromised and used to attack the rest of the network.
If an air gap can be attacked then so can your internal networks. Everything with an IP needs to be secured otherwise it can be used to attack other network elements. Just because it's "inside" means nothing.
I posted about the IPMI threat on Slashdot years ago, after reading the IPMI docs. Now, it's not only a real threat, it's one that's probably being widely exploited.
Even if IPMI packets aren't being accepted from the outside Internet, an IPMI vulnerability means that any break-in to any server allows an easy attack on all servers inside the firewall.
Anyway, for now, if you have a server, do
ipmitool -A NONE -H 1.2.3.4 bmc guid
(replacing 1.2.3.4 with the IP address) and see if it answers. If it responds to that from the outside world, you have a big problem.
Maybe they still have some functionality that allows IPMI over the regular NICs though.
Is it the kind of beast that is enabled while you never enabled it? Is there a way to test whether there is an IMPI service available from LAN? Would nmap -sU -p 623 $hostname be enough?
Windows is what got us here in the first place. Ever used old Sun gear with LOM? Nice and simple. Unfortunately Windows meant that server vendors needed to make ridiculously complex management cards that do clever things like remote mount optical drives and KVM console over IP, just because it's impossible to install or manage Windows via simple text console as it is with any real OS. This is what happens when an OS and hardware platform designed for desktops gets shoehorned into a 'server' - remote management becomes a complex hack and we end up with overly complicated and buggy management hardware.
One thing is that the materials do not distinguish 'service processors' from 'IPMI' the protocol.
The general facets on service processors broadly are no different than any 'appliance': vendors (particularly cheap ones) are lax about security and updates and there is not a lot you can do about it other than pick a vendor that seems to care or isolate the devices. This is nothing unique to the world of 'IPMI'.
In terms of IPMI, there are things in there that should be and in fact are effectively removed by some vendors today (cipher suite 0, auth none, null user). There are things that can be more complicated and probably should be limited (same username can mean different things on different ports or even the same port but different circumstances). Finally, there is the rather significant peculiarity of the 'password'. The 'password' is really a shared secret, meaning that the target must store it in the 'clear' ultimately. Additionally, the target issues a solved challenge first to prove itself to a client, meaning an unauthenticated entity can get a solved challenge and then offline crack the password if it is simple enough (roughly 1,000 times easier than cracking an entry in /etc/shadow).
So now what to do? Well for one, you should know whether your vendor will share a bmc on it's "normal" ethernet by default. You should have ipmi traffic unreachable by internet systems unless you really know what you are doing (it's not the best long haul protocol anyway). If you can stand it, use random passwords that are unique to each BMC (meaning that an offline attack is rendered futile and a janitor attack can only compromise the system that is already dissected). IPMI can be implemented and configured to be internet-facing secure, but there really isn't a lot of compelling reason to be internet-facing with it. Vendors like Dell, HP, and IBM are more likely to feel the pressure to provide safer defaults than bare board vendors and lower cost vendors.
XML is like violence. If it doesn't solve the problem, use more.