Slashdot Mirror
IPMI: Hack a Server That Is Turned Off
UnderAttack writes "A common joke in infosec is that you can't hack a server that is turned off. You better make sure that the power cord is unplugged, too. Otherwise, you may be exposed via IPMI, a component present on many servers for remote management that can be used to flash firmware, get a remote console and power cycle the server even after the normal power button has been pressed to turn the server off."
We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.
Also, this is not hacking, it's by design.
Saying "you can't hack ..." is just stupid because there's no bigger challenge. That's famous-last-words material right there.
IPMI is good when used correctly. A quality vendor (e.g. IBM) will select the appropriate security measures for default (*except* still allowing a default password, which really must be changed), but get a random white box, make sure that *this* won't work:
ipmitool -I lanplus -U correctusername -P someincorrectpassword -H youripmidevicehere power off
90% of the time it will. Cipher suite zero is the most idiotic thing in the IPMI spec, it by design allows you to access a system without knowing a password at all.
To Mitigate, ipmitool lan print 1. You'll see what cipher suites are supported and how they are restricted. Usually the supported ones will start out like '0,1,2,3'. In that case, you ipmitool lan set 1 cipher_privs Xaaa' Note the '1' may be different and the '0' in the protocol sequence isn't guaranteed, so adjust accordingly. Also use matching letters if the other cipher suites are not 'a'. xCAT on service processor setup disables cipher suite 0 automatically.
IPMI is actually capable of some very good behaviors if configured correctly.
-In the standard cipher suites, passwords are *never* sent over the wire (encrypted or in the clear) as of 2.0 (in 1.5 this was *usually* the case in sane implementations, but there was a mode for password on the wire). The password is actually a shared secret between client and service processor. In a good client, the service processor cannot be impersonated, as it *must* know the password to respond to the client correctly. ipmitool can be deceived but xCAT cannot.
-Additionally, payload can be AES encrypted as of 2.0 (and usually is, 'cipher suite 3' is the most common incarnation since it has been available since 2.0 of the spec).
-It is pretty much the *only* very specific spec for systems management. If the payload is the sequence of bytes '0,2,0' that means 'turn the server off' no matter who the vendor is. Other specifications tend to fall short of this, in the name of giving the vendors the 'freedom' to do as they wish.
That out of the way, article gets a few things wrong:
eliminate IPMI access over insecure protocols like HTTP. Use HTTPS with proper certificates, or SSH
IPMI cannot be done with HTTP(s) or SSH. IPMI *specifically* has its own UDP protocol in its remote incarnation. Most remote management solutions implementing *also* have a web and cli, but that's not covered in IPMI.
review the BIOS configuration option for IPMI. If you can't have a physical management network, at least try to use a VLAN if supported.
While this is possible in many current shared nic solutions, it generally suffers from two problems:
-In some shared NIC implementations, the OS driver for nic is more likely to mess up the remote access unless it behaves *just* right. This reduces the point of IPMI, to service a system when things have gone horribly wrong. Some older shared nic implementations made it risky even without the complication of tagging, but that's largely no longer an issue.
-It likely doesn't provide significant additional protection over an aliased private IP subnet. The theoretical additional benefit with VLAN being you are protecting service processor access if one server is compromised in-band. The problem is, you can generally coax the shared nic to let the OS onto any tagged vlan (just ask the local BMC what the compromised server's current vlan tag for IPMI traffic is, vconfig add eth0 , and usually you get right on that 'secure' vlan).
try to integrate IPMI authentication with existing authentication systems. Options typically include RADIUS and AD.
Actually, since the *standard* security mechanisms are all shared secret base, this isn't possible. There are some proprietary IPMI cipher suites that have the client transferring a password, but since normally that doesn't happen, you have no user password to check against the authentication server.
The video they showed was a web interface, *not* IPMI. I know companies like SuperMicro have really made this confusion, but I already mentioned that.
XML is like violence. If it doesn't solve the problem, use more.
However, IME it's a half-baked technology
It's a very robust and reliable technology. Just because supermicro pushes half-baked tools doesn't mean the technology is flawed. People go with supermicro because they are cheaper than the IBM, Dell, and HP implementations, but you get what you pay for.
XML is like violence. If it doesn't solve the problem, use more.
It's a SoC. Each vendor's solution exhibits different levels of restrictions and ambitions.
I recall years ago, one whitebox just gave you root ssh access to the linux system that was the BMC.
On the other hand, most of the time it's very tightly restricted to specifically do things like alerting, power on/off, remote video and serial, event logs, etc. Mostly because the whole point of the stack is to *always* work, and the more complicated you go, the higher the risk gets that you will fare no better than the OS instance on the server you are trying to manage.
IBM and the like will do things like system firmware management and changing things like hyperthreading via service processor, but still steers very clear of allowing open-ended usage of their service processor.
XML is like violence. If it doesn't solve the problem, use more.
Once upon a time there was a giant button labelled "off" and "on". Generally one didn't have to think too hard about it's function. Unless you failed to realize that the original IBM PC contained an RC circuit to hinder clickers (if you were too abrupt with the one-finger reset, the switch was ON but the PC wasn't). Tellingly, when the power states were iconified there was no provision for "ON, but only if you waited long enough first". Universal language is not big on drawing distinctions. While the icons were a little more open to interpretation than most people supposed, you could usually find one prominent switch on any device with the semi-scrutable line art, and then toggle for satisfaction.
Personally, it's pretty clear to me that off means off. Off means not doing anything I don't know about, and preferably hardly anything I do know about. A battery-powered RTC falls into the category of things I know about. Beyond that, category membership is extremely limited. I'm already drawing the line if the RTC contains a wake-on-alarm feature which fails to activate an external strobe light when armed.
Perhaps we need a new ITC symbol meaning "not nearly so OFF as you might like to think". The self-evident circle could replaced by a baby Pacman with the missing wedge rotated around to signify a sleeping cap. Less off, but more cute. Baby Pacman dreams of electric sheep while his intestinal flora multiply promiscuously.
Man/woman, dead/alive, off/on. Eternal certitudes, RIP.
Disclaimer: I'm an IBM employee.
My personal take is that if you are going with supermicro or similar, you best understand how the standard(s) work and use vendor-agnostic tools and steer clear of the tools that SuperMicro and similar provide. For example, understand how IPMI security model works, disable cipher suite 0, refrain from using ipmi 1.5 straight authentication, set user table up so that the first user is effectively disabled, and all other ipmi account slots are all managed securely by your processes. Rather than use setup dialogs and utilities from supermicro, use tools like ipmish from freeipmi and/or ipmitool for one-off usage or to build your own infrastructure. If you want a bigger infrastructure that is a bit more designed from ground-up for mass server management, then something like xCAT can help, but it's not exactly easy to use, might be a bit more ambitious than what you want to do, and despite best efforts it will still be no substitute for understanding the security implications of the implementation provided by the vendor (e.g. right now it only messes with the first 4 user slots, assumes that if cipher suite 0 is there, it's the first one, and so on and so forth).
Now if this seems overly complicated, then buy from a vendor like IBM or HP (Dell I have heard varies more between the spectrum of direct Tyan/SuperMicro and IBM/HP standards, but perhaps closer to IBM and HP for the most part). The standards based tools still continue to work, but the factory default in modern systems is more carefully considered and what proprietary tooling provided is generally more robust than the white box vendors have time/resources to bother with. While these vendors have an increasingly difficult job to distinguish themselves from cheaper competitors from Taiwan, they still do spend some of the money entailed in the price premium on a more finished and confidence inspiring experience.
Keep in mind that as recently as 3 or so years ago, us 'tier one' vendors had pretty crappy defaults and inconsistent tooling as well. However, at least in IBM world, that was all very forcibly corrected for as of the 'M2' generation (the models that released with Nehalem processors).
XML is like violence. If it doesn't solve the problem, use more.
So far IPMI has managed to survive all the crap vendors try to add to it. Chalk one up for standards! It's gone through some growing pains (the original standard was badly broken and precursors tried to share the ethernet port on the mobo), but has settled down into a usable implementation in the last few years, as long as you only use open-source IPMI tools like 'ipmitool'. There are typically still hicups with the video sniffing implementations (particularly when the OS switches video modes), but serial port forwarding seems to work quite well.
In anycase, IPMI has been a godsend for those of us with smaller server installations. It removes the need for an addressable PDU and removes the need for a console server. It removes a lot of cables.
Once the BIOS is setup you don't need to connect a keyboard or monitor to the machine ever again, even when you are physically in the machine room. You just plug you laptop into the switch or Wifi and poof.
There are numerous ways to secure the IPMI net. If configured properly IPMI 2.0 has reasonable security but still can be DOS'd. The best thing about the standard is that it is trivial to insert machines, IP filters, and other things between the IPMI network and the rest of the world but the fact that it can connect directly to the internet also ensures that those extra gadgets are going to be fairly cheap in order to compete. Always only use the IPMI 2.0 UDP protocol... if the thing has a web server or ssh access (easy to test), turn those off straight away.
Mobos with IPMI tend to cost ~$20-$50 more and the IPMI module itself (when not built in) typically costs $30-$60. Considering what you get for it, it's damn cheap. IPMI has already saved me thousands of dollars worth of equipment, gas, and time.
-Matt