Slashdot Mirror
IPMI: Hack a Server That Is Turned Off
UnderAttack writes "A common joke in infosec is that you can't hack a server that is turned off. You better make sure that the power cord is unplugged, too. Otherwise, you may be exposed via IPMI, a component present on many servers for remote management that can be used to flash firmware, get a remote console and power cycle the server even after the normal power button has been pressed to turn the server off."
We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.
Also, this is not hacking, it's by design.
Saying "you can't hack ..." is just stupid because there's no bigger challenge. That's famous-last-words material right there.
IPMI is good when used correctly. A quality vendor (e.g. IBM) will select the appropriate security measures for default (*except* still allowing a default password, which really must be changed), but get a random white box, make sure that *this* won't work:
ipmitool -I lanplus -U correctusername -P someincorrectpassword -H youripmidevicehere power off
90% of the time it will. Cipher suite zero is the most idiotic thing in the IPMI spec, it by design allows you to access a system without knowing a password at all.
To Mitigate, ipmitool lan print 1. You'll see what cipher suites are supported and how they are restricted. Usually the supported ones will start out like '0,1,2,3'. In that case, you ipmitool lan set 1 cipher_privs Xaaa' Note the '1' may be different and the '0' in the protocol sequence isn't guaranteed, so adjust accordingly. Also use matching letters if the other cipher suites are not 'a'. xCAT on service processor setup disables cipher suite 0 automatically.
IPMI is actually capable of some very good behaviors if configured correctly.
-In the standard cipher suites, passwords are *never* sent over the wire (encrypted or in the clear) as of 2.0 (in 1.5 this was *usually* the case in sane implementations, but there was a mode for password on the wire). The password is actually a shared secret between client and service processor. In a good client, the service processor cannot be impersonated, as it *must* know the password to respond to the client correctly. ipmitool can be deceived but xCAT cannot.
-Additionally, payload can be AES encrypted as of 2.0 (and usually is, 'cipher suite 3' is the most common incarnation since it has been available since 2.0 of the spec).
-It is pretty much the *only* very specific spec for systems management. If the payload is the sequence of bytes '0,2,0' that means 'turn the server off' no matter who the vendor is. Other specifications tend to fall short of this, in the name of giving the vendors the 'freedom' to do as they wish.
That out of the way, article gets a few things wrong:
eliminate IPMI access over insecure protocols like HTTP. Use HTTPS with proper certificates, or SSH
IPMI cannot be done with HTTP(s) or SSH. IPMI *specifically* has its own UDP protocol in its remote incarnation. Most remote management solutions implementing *also* have a web and cli, but that's not covered in IPMI.
review the BIOS configuration option for IPMI. If you can't have a physical management network, at least try to use a VLAN if supported.
While this is possible in many current shared nic solutions, it generally suffers from two problems:
-In some shared NIC implementations, the OS driver for nic is more likely to mess up the remote access unless it behaves *just* right. This reduces the point of IPMI, to service a system when things have gone horribly wrong. Some older shared nic implementations made it risky even without the complication of tagging, but that's largely no longer an issue.
-It likely doesn't provide significant additional protection over an aliased private IP subnet. The theoretical additional benefit with VLAN being you are protecting service processor access if one server is compromised in-band. The problem is, you can generally coax the shared nic to let the OS onto any tagged vlan (just ask the local BMC what the compromised server's current vlan tag for IPMI traffic is, vconfig add eth0 , and usually you get right on that 'secure' vlan).
try to integrate IPMI authentication with existing authentication systems. Options typically include RADIUS and AD.
Actually, since the *standard* security mechanisms are all shared secret base, this isn't possible. There are some proprietary IPMI cipher suites that have the client transferring a password, but since normally that doesn't happen, you have no user password to check against the authentication server.
The video they showed was a web interface, *not* IPMI. I know companies like SuperMicro have really made this confusion, but I already mentioned that.
XML is like violence. If it doesn't solve the problem, use more.
That's all fine, as long as (a) the person setting up the machine actually knows that the motherboard includes IPMI and understands the significance, and (b) the password changing tools actually work. I was setting up a system with a Supermicro board recently, and the software was quite happy to let me set a reasonably secure password, but then apparently that password was too long for the log-in screen in another part of the system to accept! I was completely locked out. Yes, really. The solution required literally connecting up an entire PC's worth of peripherals and installing an entire OS just to get to the OS-based IPMI management tools -- which wouldn't even be an option for some cases where IPMI is supposed to help. Haven't these people ever heard of jumpers?
IPMI is one of those things that sounds great, and might be worth the hassle for admins who really do need to control a whole rack of machines or a whole set of blades in a rendering workstation or something similar. However, IME it's a half-baked technology (or, perhaps more fairly, the software tools rather than IPMI itself are usually of very low quality) and for most people it comes with a huge liability they might not even know they had.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
However, IME it's a half-baked technology
It's a very robust and reliable technology. Just because supermicro pushes half-baked tools doesn't mean the technology is flawed. People go with supermicro because they are cheaper than the IBM, Dell, and HP implementations, but you get what you pay for.
XML is like violence. If it doesn't solve the problem, use more.
I was working with a large financial institution that could not understand how a large number of servers were shut down when the servers were in a strictly restricted server room. The servers were running VMware's ESX. Looking thru the logs I was able to see evidence that the servers were shut down -- PowerButtonHandler log entry. The IT admin kept on insisting that this was in error as nobody had entered the server room thus nobody could shut down the hosts. It took quite a bit of explaining how IPMI works and that if one could access the network, then one could shutdown the hosts remotely. I demonstrated it to the admin on a surplus server who quickly turned very green. Seems they used a very simple password for their iLO that matched the log in name.
I also demonstrated how one could remotely reboot a windows server into something like a Knoppix ISO image and go surfing the hard drive thus demonstrating that physical security was not enough, only a start.
You must also disable cipher suite 0. Since it by design makes the password moot and all...
XML is like violence. If it doesn't solve the problem, use more.
I know nothing about IPMI so sorry if this is a dumb question:
If it doesn't require the CPU or RAM so be on then am I right in assuming that IPMI is some sort of separate system on a chip (or similar) on the same motherboard as the main computer? Or does it wake the main CPU up and use that?
If it is a SoC or a microcontroller with a few extra bits - has anyone hacked it to do more interesting stuff other than just update server parameters?
We use iLO. Nothing better than being able to observe what a server is doing during reboot. From outside of the LAN you must use VPN before getting access so it is fairly secure. OR You can reboot the machine and wonder what is happening. Sure the capability has some risks, but if you put some thought into your implementation it will be reasonably secure.
I just use ipmitool locally (on linux) to set up each new machine, and it always works fine (on Supermicro as well). Sometimes slightly different settings are required for different vendors, but the differences are minor and can be done in a script.
Once upon a time there was a giant button labelled "off" and "on". Generally one didn't have to think too hard about it's function. Unless you failed to realize that the original IBM PC contained an RC circuit to hinder clickers (if you were too abrupt with the one-finger reset, the switch was ON but the PC wasn't). Tellingly, when the power states were iconified there was no provision for "ON, but only if you waited long enough first". Universal language is not big on drawing distinctions. While the icons were a little more open to interpretation than most people supposed, you could usually find one prominent switch on any device with the semi-scrutable line art, and then toggle for satisfaction.
Personally, it's pretty clear to me that off means off. Off means not doing anything I don't know about, and preferably hardly anything I do know about. A battery-powered RTC falls into the category of things I know about. Beyond that, category membership is extremely limited. I'm already drawing the line if the RTC contains a wake-on-alarm feature which fails to activate an external strobe light when armed.
Perhaps we need a new ITC symbol meaning "not nearly so OFF as you might like to think". The self-evident circle could replaced by a baby Pacman with the missing wedge rotated around to signify a sleeping cap. Less off, but more cute. Baby Pacman dreams of electric sheep while his intestinal flora multiply promiscuously.
Man/woman, dead/alive, off/on. Eternal certitudes, RIP.
Disclaimer: I'm an IBM employee.
My personal take is that if you are going with supermicro or similar, you best understand how the standard(s) work and use vendor-agnostic tools and steer clear of the tools that SuperMicro and similar provide. For example, understand how IPMI security model works, disable cipher suite 0, refrain from using ipmi 1.5 straight authentication, set user table up so that the first user is effectively disabled, and all other ipmi account slots are all managed securely by your processes. Rather than use setup dialogs and utilities from supermicro, use tools like ipmish from freeipmi and/or ipmitool for one-off usage or to build your own infrastructure. If you want a bigger infrastructure that is a bit more designed from ground-up for mass server management, then something like xCAT can help, but it's not exactly easy to use, might be a bit more ambitious than what you want to do, and despite best efforts it will still be no substitute for understanding the security implications of the implementation provided by the vendor (e.g. right now it only messes with the first 4 user slots, assumes that if cipher suite 0 is there, it's the first one, and so on and so forth).
Now if this seems overly complicated, then buy from a vendor like IBM or HP (Dell I have heard varies more between the spectrum of direct Tyan/SuperMicro and IBM/HP standards, but perhaps closer to IBM and HP for the most part). The standards based tools still continue to work, but the factory default in modern systems is more carefully considered and what proprietary tooling provided is generally more robust than the white box vendors have time/resources to bother with. While these vendors have an increasingly difficult job to distinguish themselves from cheaper competitors from Taiwan, they still do spend some of the money entailed in the price premium on a more finished and confidence inspiring experience.
Keep in mind that as recently as 3 or so years ago, us 'tier one' vendors had pretty crappy defaults and inconsistent tooling as well. However, at least in IBM world, that was all very forcibly corrected for as of the 'M2' generation (the models that released with Nehalem processors).
XML is like violence. If it doesn't solve the problem, use more.
I read the stupid article and the entire discussion (didn't you read my previous comments?). IPMI is a great tool and there is no way I'm turning it off in a data center, instead of securing it. Not going to expose it to the internet, of course.
So far IPMI has managed to survive all the crap vendors try to add to it. Chalk one up for standards! It's gone through some growing pains (the original standard was badly broken and precursors tried to share the ethernet port on the mobo), but has settled down into a usable implementation in the last few years, as long as you only use open-source IPMI tools like 'ipmitool'. There are typically still hicups with the video sniffing implementations (particularly when the OS switches video modes), but serial port forwarding seems to work quite well.
In anycase, IPMI has been a godsend for those of us with smaller server installations. It removes the need for an addressable PDU and removes the need for a console server. It removes a lot of cables.
Once the BIOS is setup you don't need to connect a keyboard or monitor to the machine ever again, even when you are physically in the machine room. You just plug you laptop into the switch or Wifi and poof.
There are numerous ways to secure the IPMI net. If configured properly IPMI 2.0 has reasonable security but still can be DOS'd. The best thing about the standard is that it is trivial to insert machines, IP filters, and other things between the IPMI network and the rest of the world but the fact that it can connect directly to the internet also ensures that those extra gadgets are going to be fairly cheap in order to compete. Always only use the IPMI 2.0 UDP protocol... if the thing has a web server or ssh access (easy to test), turn those off straight away.
Mobos with IPMI tend to cost ~$20-$50 more and the IPMI module itself (when not built in) typically costs $30-$60. Considering what you get for it, it's damn cheap. IPMI has already saved me thousands of dollars worth of equipment, gas, and time.
-Matt