Slashdot Mirror
Supermicro Fails At IPMI, Leaks Admin Passwords
drinkypoo writes: Zachary Wikholm of Security Incident Response Team (CARISIRT) has publicly announced a serious failure in IPMI BMC (management controller) security on at least 31,964 public-facing systems with motherboards made by SuperMicro: "Supermicro had created the password file PSBlock in plain text and left it open to the world on port 49152." These BMCs are running Linux 2.6.17 on a Nuvoton WPCM450 chip. An exploit will be rolled into metasploit shortly. There is already a patch available for the affected hardware.
Anyone who trusted SuperMicro for anything business critical gets what they deserved. I had the misfortune of working with their engineering department back in 2006/2007. They were absolutely clueless. Slapping random components together hoping to build good server motherboards, wondering why things would perform oddly or be unstable. They admittedly got it right more often than not, but thats not exactly what you want for servers. Stuff like this is proof they aren't serious business.
They forgot to pay their SCO licensing fee in order to legally use Lunix. Don't forget to pay your $699 licensing fee. Remember, the price goes up to $1399 at the end of July.
Some intrepid hacker should write a script to take control and apply the patch the vulnerable software.
It's not super-clear from the article what sort of systems there are, even with the Wikipedia link to IPMI. I mistakenly assumed that BMC was the configuration management company at first...
Without linking to XKCD, can anyone explain this to me like a child?
What use case? This sort of things should always be behind a firewall. Is it to hard to VPN in? Hell our supermicro IPMI's work rather well though a proxy on the firewall (dell and HP for that matter).
No sir I dont like it.
Everyone ASSumes it's secure by all those freaking eyes staring out into space, twidding thumbs, and whatnot. If you NEED to be safe and secure in your computing, go with Microsoft. Go with a proven WINner.
IPMI v2.0 has a design flaw that any anonymous remote attacker can request and get the salt and password hash for the admin user!
It is a design flaw that cannot be patched.
Better use all of the 20 character allowed maximum password length and rotate the password often!
Working on a product based around these now...
As far as I can tell, the Nuvoton WPCM450 is what contains the Matrox G200ew clone for graphics output. Thanks to XAA being discontinued in X.org, the MGA driver is practically unusable for X at this point(even with an ancient, 2d window manager).
Yet another reason to avoid this hardware.
http://www.masturbateforpeace.com/
By default, SuperMicro IPMI attaches to normal ethernet. So if you hook up a server to a public connection, you've exposed your IPMI. We caught this in a security audit, we added a dhcp honey pot to our static network to see if we could get any devices to announce themselves. We about shat our pants! There's probably a ton of people at risk not knowing this motherboard is insecure by default!
whoever manages the 31,964 public-facing systems and allows direct access to IPMI from the internet.
everyone knows IPMI is insecure so they have a shitty implementation of a piss poor protocol big fucking deal
In the article, SuperMicro has fix in update. However, the key takeaway is that thousands of people decided to not patch their SH1T. "This means at the point of this writing, there are 31,964 systems that have their passwords available on the open market. It gets a bit scarier when you review some of the password statistics. Out of those passwords, 3296 are the default combination." "Besides flashing, there is another (albeit unsupported) temporary fix. Most of the systems affected by this particular issue also have their “sh” shell accessible from the SMASH command line. If you login to the SMASH via ssh and run the command “shell sh”, you can drop into a functional SH shell. From there you can actually kill all “upnp” processes and their related children, which provides a functional fix. That is of course until the system is completely disconnected from power and reconnected, during which the IPMI module will reboot. This is what I have done for our own systems that were unable to be permanently fixed at this time. After continual monitoring, I am satisfied with the results and there has not been any noticeable impact on functionality."
The IPMI on my supermicro motherboard only works through one of network ports. In fact it has it's own dedicated port that is only for IPMII (the regular OS doesn't even see it). Though I have seen older motherboards that work like yours I think supermicro has moved in more recent products to dedicated IPMI ports, maybe because of this very reason. You should be configuring the IPMI even if you don't plan to use it, set it an IP and then blackhole that IP on your network. If you don't configure it you don't know what it's doing.
it's crazy to expose IPMI to the public net. yes, that might mean you need separate wiring for an internal subnet, and you might not be able to use all your ports for public access - just read the docs before you buy it.
By default, SuperMicro IPMI attaches to normal ethernet.
Yes, I saw a mention of that on G+ today, but I lost it. So I went to the source, I will save y'all the trouble of dicking with the PDF and jump straight to page 2-26 and excerpt the really interesting part:
YE GODS. At least it's in the manual, which no one reads. You can select a port once you've got the system up and running, and once you do that it will stick, but until then it operates unsafely, as above. And if by chance there's no link on the management port during boot, perhaps because the management switch is also being cycled, then IPMI will appear on another interface.
There's no excuse for not firewalling that off, but it's still also unacceptable behavior.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Our problem with the Supermicro IPMI units is that they eventually crash. Once down, we don't know any way to reboot them other than to power cycle the machine, which imposes downtime on the users. So we leave the IPMI down. This is Linux, perhaps it is different in some other OS.
This appears to be similar to a vulnerability in the Commodore 64 operating system that let malicious users reset the machine with a system command, also at 49152.
By default, SuperMicro IPMI attaches to normal ethernet. So if you hook up a server to a public connection, you've exposed your IPMI. We caught this in a security audit, we added a dhcp honey pot to our static network to see if we could get any devices to announce themselves. We about shat our pants! There's probably a ton of people at risk not knowing this motherboard is insecure by default!
Dedicated v. shared ethernet is going to vary per model and some models actually don't support shared ethernet.
Also even when using shared, that is only a danger when a DHCP server is present and the machines are being admined by a idiot who does not know what they are using.
Port 49152? Uh oh. A Commodore 64 could hack that!
> the IPMI interface is configured with a non-public IP address ... so while this is some sloppy-ass stuff on Supermicro's part, I am personally not that concerned.
In my case, those non-public IPs are part of a management network that is only accessible via a VPN. So we're safe UNLESS the VPN endpoint happens to have a flaw, or someone mistakenly plugs one of the management interfaces into the internet, not realizing that the "security" on the interface doesn't actually work.
Excellent information.
Ah, well. The only one of my SuperMicro boxes that had a public-facing IPMI address can't be reached; the IPMI software is borked and won't let me assign an IP address. It will take a 200km drive followed by a hard power cycle to get the IPMI up and running again.
Lack of attention, long term support and piss poor execution has completely overshadowed any conceivable benefit of management firmware across all systems vendors. They might as well have not even bothered in the first place.
I manage 10,000 of them. To date lower infant mortality and lower long term failures than I had seen previously with Dell and HP. They also ship a lot faster than Dell or HP. Anyone who exposes their IPMI interfaces to the public internet deserves the results.
Target did NOT expose their point of sale system to the Internet, but still ended up being owned with 170M credit card numbers (and other details) compromised. Just because your IPMI is inside (and even segmented off into a "secure" VLAN/subnet) means nothing. Hard on the shell, soft on the inside is the wrong kind of thinking.
Even air gap are of questionable use, as Iran (Stuxnet) and the US DoD (agent.btz) have learned:
https://www.schneier.com/blog/archives/2013/10/air_gaps.html
Seriously, "don't put it on the Internet" is a dumb piece of advice.
I found what appears to be a good permanent workaround from a Christian Hertel in the comment section of http://threatpost.com/plaintex...:
/nv/ipctrl/rultbl.sav
Another Hotfix in case there is no newer IPMI firmware release to upgrade to (so no way to fix the issue otherwise):
Login via SSH, then issue the following commands:
shell sh
iptables -I INPUT -p tcp --dport 49152 -j DROP
iptables-save >
I've tested it on my affected servers and have verified it works and survives a reboot of IPMI. However, I'm wondering if there's a reason I might regret blocking access to port 49152 for some reason.
Thanks for the workaround, Mr. Hertel!
Home FreeNAS box...
~ # telnet 192.168.7.7 49152 /PSBlock
Trying 192.168.7.7...
Connected to 192.168.7.7.
Escape character is '^]'.
GET
adminADMINmypasswordTTmyuseridmypassword4
Hey everybody.
First of all, wow. Never thought this would spread like this. SEND ME EMAILS PEOPLE. Don't be afraid to ask questions or challenge what I have said in the release. I'm open to suggestions. Got a good one? I'll seriously buy you a drink. sirt@cari.net, do it. Also I'm sorry if my grammar is atrocious. It's nearing midnight and it's been a long week. I don't enlgish too well after dinner and before morning tea.
Second, there are several reoccurring themes in every comment section of all of these articles. There is this extremely negative streak of people screaming "you must be a complete and utter moron for putting the OOB platforms on public interfaces". However, many of them forgot when they would have done the same thing. Now in light of my fellow researchers Dan Farmer and HD Moore's research on this matter, all service providers should have taken a second look at this, but alas not everybody knew about the vulnerabilities.
Let me refute a few points here:
1) They should be behind a firewall/why is that port not blocked/mine is behind a VPN
Okay folks. There are several issues here. First, don't put a bunch of vulnerable systems into a private network together. That will NOT end well for anybody involved. I feel sincere pity for anybody who falls prey to this notion. If you look at the ars technica article, HD helped stop this landslide of nonsense by posting basically that. Keep in mind that IPMItool can manipulate the IPMI interface. You want to talk about compromise? Put all your eggs in a single basket. There are ways of doing it, but they are difficult. I'm working on a system for my "home" net as we speak. Now let's talk VPN. This totally defeats the purpose of OOB AND a VPN simultaneously. Again, you can access the OOB network from the host, but this time it's in a VPN. Even if you use the dedicated port, you can do some pretty serious damage. And what if the VPN goes down? Then what?
Now I'm not defending the idea of public interfaces and BMCs. But seriously. People need to stop over-simplying this thing. Yeah on paper it looks stupid, but hindsight is 20/20. The real mistake made by a lot of people was trusting their vendors to do their due diligence, but that's another can of worms for another day of fishing.
2) The IPtables fix. /nv/ directory is loaded from scratch when the BMC chip is "reset". Also, the IPtables kernel module doesn't always load right. Keep in mind that this is an horribly bastardized version of the 2.6.17 kernel. There are things installed in that firmware that shouldn't be in ANY embedded platform if you ask me, but nobody did so I'll just shut up. I'll do more testing and let everybody know. This doesn't really help if the system doesn't have shell sh.
I've read about this IPtables fix that supposedly works across reboots. Reboots are not what causing the problem, it's the loss of power that does. Remember that OOB devices are always on, so long that they are connected to power. I remember attempting to use IPtables and I ran into a lot of problems. One being that I do recall that the
3) You can flash while the system is online.
HA! Good luck. I hope you like surprises and unexpected results. Don't even both attempting to go to 3.15 on the x9DRL. 7 of 10 x9drl boards break upon flashing. Not to mention you have to reboot for the thing to go into effect. On the newer bioses you must have the latest version of the mobo bios, prior to installing the security fix for the BMC. At any rate, the Supermicro documentation is usually a great tool however it can be wrong. If you encounter this please let me know and I can pass it directly to the Sr. Product Manager for review.
One last thing; Often times people forget that where there are customers, there are SLAs. If a client has chosen to run IPMI on a public interface (it
When consumer routers were shipped with hideously buggy firmware, enterprising developers replaced the badly constructed user space with a more sensible one in the form of OpenWRT etc. The result was a complete win.
These IPMI cards use a small handful of fairly standard SoCs across a wide variety of boards, and have (intentionally or not) root shell access, so the same should in principle be possible here. It would even be sufficient just to replace the user space and continue to use the shipped kernel. A dropbear for ssh access with scripts/binaries to power off, on, reset and connect to serial console would be both sufficient and a great improvement on what we have at the moment with the incompetent legacy IPMI protocol and equally poor firmware implementation.