Slashdot Mirror
IPMI: Hack a Server That Is Turned Off
UnderAttack writes "A common joke in infosec is that you can't hack a server that is turned off. You better make sure that the power cord is unplugged, too. Otherwise, you may be exposed via IPMI, a component present on many servers for remote management that can be used to flash firmware, get a remote console and power cycle the server even after the normal power button has been pressed to turn the server off."
We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.
Also, this is not hacking, it's by design.
Saying "you can't hack ..." is just stupid because there's no bigger challenge. That's famous-last-words material right there.
Any sane admin would change the management password when installing the server. And I don't remember any recent vulnerabilities related to ipmi password.
Seems to me this is more about specific vendor's BMC's used implement IPMI, and the lack of information they provide, and attention sysadmin's pay to securing them.
My solution was to never let a network cable anywhere near the dedicated iLO port. I am not a trusting soul.
Please do not read this sig. Thank you.
IPMI is good when used correctly. A quality vendor (e.g. IBM) will select the appropriate security measures for default (*except* still allowing a default password, which really must be changed), but get a random white box, make sure that *this* won't work:
ipmitool -I lanplus -U correctusername -P someincorrectpassword -H youripmidevicehere power off
90% of the time it will. Cipher suite zero is the most idiotic thing in the IPMI spec, it by design allows you to access a system without knowing a password at all.
To Mitigate, ipmitool lan print 1. You'll see what cipher suites are supported and how they are restricted. Usually the supported ones will start out like '0,1,2,3'. In that case, you ipmitool lan set 1 cipher_privs Xaaa' Note the '1' may be different and the '0' in the protocol sequence isn't guaranteed, so adjust accordingly. Also use matching letters if the other cipher suites are not 'a'. xCAT on service processor setup disables cipher suite 0 automatically.
IPMI is actually capable of some very good behaviors if configured correctly.
-In the standard cipher suites, passwords are *never* sent over the wire (encrypted or in the clear) as of 2.0 (in 1.5 this was *usually* the case in sane implementations, but there was a mode for password on the wire). The password is actually a shared secret between client and service processor. In a good client, the service processor cannot be impersonated, as it *must* know the password to respond to the client correctly. ipmitool can be deceived but xCAT cannot.
-Additionally, payload can be AES encrypted as of 2.0 (and usually is, 'cipher suite 3' is the most common incarnation since it has been available since 2.0 of the spec).
-It is pretty much the *only* very specific spec for systems management. If the payload is the sequence of bytes '0,2,0' that means 'turn the server off' no matter who the vendor is. Other specifications tend to fall short of this, in the name of giving the vendors the 'freedom' to do as they wish.
That out of the way, article gets a few things wrong:
eliminate IPMI access over insecure protocols like HTTP. Use HTTPS with proper certificates, or SSH
IPMI cannot be done with HTTP(s) or SSH. IPMI *specifically* has its own UDP protocol in its remote incarnation. Most remote management solutions implementing *also* have a web and cli, but that's not covered in IPMI.
review the BIOS configuration option for IPMI. If you can't have a physical management network, at least try to use a VLAN if supported.
While this is possible in many current shared nic solutions, it generally suffers from two problems:
-In some shared NIC implementations, the OS driver for nic is more likely to mess up the remote access unless it behaves *just* right. This reduces the point of IPMI, to service a system when things have gone horribly wrong. Some older shared nic implementations made it risky even without the complication of tagging, but that's largely no longer an issue.
-It likely doesn't provide significant additional protection over an aliased private IP subnet. The theoretical additional benefit with VLAN being you are protecting service processor access if one server is compromised in-band. The problem is, you can generally coax the shared nic to let the OS onto any tagged vlan (just ask the local BMC what the compromised server's current vlan tag for IPMI traffic is, vconfig add eth0 , and usually you get right on that 'secure' vlan).
try to integrate IPMI authentication with existing authentication systems. Options typically include RADIUS and AD.
Actually, since the *standard* security mechanisms are all shared secret base, this isn't possible. There are some proprietary IPMI cipher suites that have the client transferring a password, but since normally that doesn't happen, you have no user password to check against the authentication server.
The video they showed was a web interface, *not* IPMI. I know companies like SuperMicro have really made this confusion, but I already mentioned that.
XML is like violence. If it doesn't solve the problem, use more.
I was working with a large financial institution that could not understand how a large number of servers were shut down when the servers were in a strictly restricted server room. The servers were running VMware's ESX. Looking thru the logs I was able to see evidence that the servers were shut down -- PowerButtonHandler log entry. The IT admin kept on insisting that this was in error as nobody had entered the server room thus nobody could shut down the hosts. It took quite a bit of explaining how IPMI works and that if one could access the network, then one could shutdown the hosts remotely. I demonstrated it to the admin on a surplus server who quickly turned very green. Seems they used a very simple password for their iLO that matched the log in name.
I also demonstrated how one could remotely reboot a windows server into something like a Knoppix ISO image and go surfing the hard drive thus demonstrating that physical security was not enough, only a start.
I know nothing about IPMI so sorry if this is a dumb question:
If it doesn't require the CPU or RAM so be on then am I right in assuming that IPMI is some sort of separate system on a chip (or similar) on the same motherboard as the main computer? Or does it wake the main CPU up and use that?
If it is a SoC or a microcontroller with a few extra bits - has anyone hacked it to do more interesting stuff other than just update server parameters?
Once upon a time there was a giant button labelled "off" and "on". Generally one didn't have to think too hard about it's function. Unless you failed to realize that the original IBM PC contained an RC circuit to hinder clickers (if you were too abrupt with the one-finger reset, the switch was ON but the PC wasn't). Tellingly, when the power states were iconified there was no provision for "ON, but only if you waited long enough first". Universal language is not big on drawing distinctions. While the icons were a little more open to interpretation than most people supposed, you could usually find one prominent switch on any device with the semi-scrutable line art, and then toggle for satisfaction.
Personally, it's pretty clear to me that off means off. Off means not doing anything I don't know about, and preferably hardly anything I do know about. A battery-powered RTC falls into the category of things I know about. Beyond that, category membership is extremely limited. I'm already drawing the line if the RTC contains a wake-on-alarm feature which fails to activate an external strobe light when armed.
Perhaps we need a new ITC symbol meaning "not nearly so OFF as you might like to think". The self-evident circle could replaced by a baby Pacman with the missing wedge rotated around to signify a sleeping cap. Less off, but more cute. Baby Pacman dreams of electric sheep while his intestinal flora multiply promiscuously.
Man/woman, dead/alive, off/on. Eternal certitudes, RIP.
For the love of god don't watch the video at the bottom unless you want to try to punch the speaker through your screen.
Why is this worth a story? Are there really people that manage servers but do not even understand the basic functions of IPMI?
Well, I guess there are. But this story will not help them either.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Can IPMI simply be turned off?
Is there a setting in the BIOS to turn IPMI off?
Most important part of the sys-/security- admin role is to keep users from shooting themselves in the foot by denying them access to luxuries they don't really need, c.f. sudo - because root gives you everything so you don't NEED the root password. IPMI does not need to double as-/interact with- remote system management.
IPMI becomes most dangerous when SAs fail to apply those same rules and go for convenience features, like IPMI that doesn't have a dedicated port/wire, or IPMI that can talk to the bios.
So choose your IPMI sagely: Power status, power logs, sensor states via dedicated channels and NOT via bios backdoor, power on, power off, reset. You do NOT need backdoor BIOS access, that's what remote management consoles are for. You do NOT need boot order access. And you absolutely do NOT want the IPMI to expose itself to the CPU or main memory in any way that would allow the kernel or BIOS to talk with it any more than you want the root password in /etc/motd.
-- A change is as good as a reboot.
Seriously?
All the author does is give a less than cursory overview over IPMI and then handwaves about facts every serious admin knows about:
Don't expose IPMI, it's an attack vector, yadda yadda.
WTF?
We should NEVER give computers control over their own power switch. I watched the in horror when they switched PC's from a hard switch to a soft power switch that makes you hold it down for 4 sec's to power it off....giving them control over their own power button is just the start of them taking over...
AB HOC POSSUM VIDERE DOMUM TUUM
So far IPMI has managed to survive all the crap vendors try to add to it. Chalk one up for standards! It's gone through some growing pains (the original standard was badly broken and precursors tried to share the ethernet port on the mobo), but has settled down into a usable implementation in the last few years, as long as you only use open-source IPMI tools like 'ipmitool'. There are typically still hicups with the video sniffing implementations (particularly when the OS switches video modes), but serial port forwarding seems to work quite well.
In anycase, IPMI has been a godsend for those of us with smaller server installations. It removes the need for an addressable PDU and removes the need for a console server. It removes a lot of cables.
Once the BIOS is setup you don't need to connect a keyboard or monitor to the machine ever again, even when you are physically in the machine room. You just plug you laptop into the switch or Wifi and poof.
There are numerous ways to secure the IPMI net. If configured properly IPMI 2.0 has reasonable security but still can be DOS'd. The best thing about the standard is that it is trivial to insert machines, IP filters, and other things between the IPMI network and the rest of the world but the fact that it can connect directly to the internet also ensures that those extra gadgets are going to be fairly cheap in order to compete. Always only use the IPMI 2.0 UDP protocol... if the thing has a web server or ssh access (easy to test), turn those off straight away.
Mobos with IPMI tend to cost ~$20-$50 more and the IPMI module itself (when not built in) typically costs $30-$60. Considering what you get for it, it's damn cheap. IPMI has already saved me thousands of dollars worth of equipment, gas, and time.
-Matt
IPMI is useful but a huge security hole with some bad default settings. Some servers come with ADMIN/ADMIN as the factory default. Some Dell servers use "admin/calvin". The problem of securely getting a server from in the box to in service wasn't well worked out.
IPMI systems themselves can be vulnerable. Some are small ARM systems running Linux. Badly configured Linux. "Older versions of the X8SIL-F IPMI code accepted ssh connections no matter what password was given. The software would then check the password and reject or accept the connection, but there was a brief window to create ssh port forwards. People were getting spam/abuse complaints for their IPMI IPs because of this. " ... "A second problem is an anonymous user with a default password. The anonymous user seems to be fixed in firmware version 2.22."
There's also the question of whether there might be a built-in factory password/key that you can't detect. If someone wanted to build a backdoor into servers, the IPMI interface firmware would be a very good place to do it. Especially since the article linked above found one.
I've heard an adage similar to that joke: the least vulnerable computer is the one that's offline. It's not the question of whether it's powered on or not, it's really a question of whether or not it's physically connected to any network.
So, yeah, get to unplugging those thick blue cables in the back.
What servers do you use now without IPMI?
Small, low power ones. Supermicro makes an Atom mini-itx board but that is about it. There are many e350 boards which are cheaper and have some advantages over an Atom system but not one of them will operate headless until or unless the OS is running.
"The only secure computer is one that's unplugged, locked in a safe,
and buried 20 feet under the ground in a secret location... and I'm
not even too sure about that one" -- Dennis Huges
Now we just need to create concrete digging computer viruses and we are there.
I was not at all interested in the IPMI features (we don't use them) but we built a Supermicro based system because I couldn't find anything ready-made from Dell, HP, or IBM that offered dual Xeon, several PCIe slots for RAID cards, big RAM, and a case with 36 hot swap drive bays in a rackmount configuration (I don't think there was any in a "tower" config either though).
Did I miss something that is available for this? I'm asking because we're considering another such system using Sandy Bridge CPUs.
If you could hack the dusty server in my closet that lacks any sort of wireless interface and has absolutely no cables running to or from it currently, I'd give you $50.
No power source, no physical access, no hacking. Your statement is amusing, but not entirely accurate.
I've been half-joking for years about the fact that most computers (and printers, etc) are never truly "off" these days.
"The power button doesn't actually turn it off," I tell users, "It just tells the device to pretend it's off. Open it up and you'll see lights still on inside." Because you often can. "At least we can still pull the plug, so they can't take over the world just yet...." [nudge, wink, smile] Inside I am not smiling. Because I know that computers (and other electronic devices) increasingly have their own internal batteries....
http://alternatives.rzero.com/
If IBM (and do Dell and HP still belong on this list?) provides extra features I don't really need why should I buy their stuff for a lot more? Supermicro can sell me a 64 core system with 128GB of memory for $9k including tax, and if it gets the job done why not get two or three instead of paying more than that for a single IBM system with lower specs and a few extra features I don't need?
If you need the features you pay the money, but if you don't why "get what you pay for" when you don't need it?
Of course if one vendors implementation of a specific feature is fatally flawed then just be aware and don't use it, or if you need it buy something else. I don't really need this full remote access method so I don't use it.
"A common joke in infosec is that you can't hack a server that is turned off."
I've been working in infosec for 10 years and ever once have heard that joke. The two trueisms that I'm familiar with is (1) airgap - It's only secure if it isn't on the network. and (2) physical access = owned
Has somebody woken up from 10 or so years of sleep and found out that there is something like IPMI? Gosh!