Advantech Industrial Serial-To-Internet Gateways Left Wide Open (rapid7.com)

itwbennett writes: Researchers from Rapid7 have discovered a vulnerability in serial-to-IP gateway devices from Advantech that would allow the Internet-connected industrial devices to be accessible to anyone, with no password. In October, the Taiwanese firm patched the firmware in some of these devices to remove a hard-coded SSH (Secure Shell) key that would have allowed unauthorized access by remote attackers. But it overlooked an even bigger problem: Any password will unlock the gateways, which are used to connect legacy serial devices to TCP/IP and cellular networks in industrial environments around the world.

Comments

Why?

[U2xhc2hkb3QgU3Vja3M]

Why would industrial machines be connected to the Internet in the first place?

Re:Why?

[irrational_design]

I assume so they can be monitored from a remote location.

Re:Why?

[Anonymous Coward]

Serious answer from the real world: Because by CIO fiat we can't have PCs or any intelligent devices in remote locations anymore. Why? Because of The Cloud.

Re:Why?

[Anonymous Coward]

Why would industrial machines be connected to the Internet in the first place? So they can participate in stuxnet, an important industry networking protocol

Re:Why?

[bluefoxlucid]

Everyone wants fruity pebbles.

Re:Why?

[vux984]

That they are connected to the internet makes perfect sense for a lot of reasons.

That they are connected to the internet and reachable directly, and publicly on the other hand is total spectacular fail.

They should be behind firewalls, that only allow connections in from authorized remote monitoring ip blocks, over encrypted connections presenting the right certificates.

But the usual; is to just do the minimum possible so that its functional. Security simply isn't even a consideration that goes into these things.

Re:Why?

[OzPeter]

Why would industrial machines be connected to the Internet in the first place?

Just because it's a Serial to IP converter doesn't mean that it is connected to the Internet. These types of devices are used all the time to interface to legacy systems that can't talk IP to internal control systems.

And sure you could argue that in this day and age you need more than just boundary protection .. but of the attackers are already in your network I think that you are screwed regardless of what type of peripherals you have installed.

Re:Why?

[cfalcon]

I think it really depends on the application in question.

Pretend you had set up a firewall correctly a few years ago, but the firewall was set up before the port 32764 backdoor had been discovered. Uh oh, your properly configured firewall has a backdoor! There goes the IP block monitoring too (either the check is on the firewall, and the backdoor disables it, or its beyond the firewall and the firewall spoofs it, or both). Your certs are set up, but heartbleed exists (and you don't know about it, but your attackers do), and FREAK is live too (but hasn't been discovered yet) and the RNG that made them was compromised but no one knew that yet either.

So you did everything right, BUT your network can be totally taken apart remotely... because it's on the internet.

Maybe you need it to be on the net. If so, you'll do that. But maybe you could possibly, at higher cost, make do with it not on the net. Maybe that should be considered.

Re:Why?

[Anonymous Coward]

PCs left in labs or on factory floors get absolutely destroyed by mal/spyware acquired by workers browsing malicious sites and/or installing inappropriate things. It is sickening what they do to those machines. I've seen machines in a laboratory used exclusively by masters and PhD employees filled with porn, p2p file sharing stuff and multiple virus infections.

Smart managers close that window wherever they can.

Re:Why?

[Anonymous Coward]

And a device which runs Linux and an SSH server is not an intelligent device? WTF is that company's business if not security? I could cobble together something more secure from a $20 portable access point and OpenWRT between breakfast and lunch. Also, somebody should look into their license compliance. There's open source software in their firmwares.

Re:Why?

[aaarrrgggh]

To use a commodity modem rather than a proprietary one, generally. They (as a spass of device, not this particular manufacturer) were also a huge security improvement, as now it was possible to have things like radius-based authentication that could be centrally managed rather than no, or (shared) device level authentication.

Serial automation networks are a pain, and IP networks offered a number of huge benefits. Unfortunately there was about a 5-10 year period where there was no security, another 5-10 years of bad security, and another 5+ years left in dealing with poor security implementations. You can't air-gap everything out there, and leased lines don't scale for all systems.

Re:Why?

[teknosapien]

Having worked in an industry that monitored critical infrastructure equipment, we used to keep these behind firewalls in an RFC 1918 space.
The reason why they are connected is that these SNMP and Modbus devices are passively monitored for accuracy, trouble issues, power usage and capacity, But lately there has been an increase in active configuration through derived equations to shed loads , remotely start and stop generators, HVAC systems, etc...
Not having them password protected is an epic fail

Re:Why?

[ilsaloving]

Rightfully, they shouldn't be. A sensible configuration would be to have them connected to a TCP/IP network so that they can be plugged into a central monitoring system... and the network they are connected to would (ideally) be plugged into some sort of VPN appliance. You would then *only* be able to access the systems through that VPN connection.

However, if the network is somehow breached, then you have a problem. Or if the people you have setting things up are idiots and literally DO plug it directly to the internet, which is also distressingly likely.

Re:Why?

[vux984]

Every single one of those exploits is mitigated by whitelisting the incoming ip blocks authorized to connect.

The ip block restriction, means the port 32764 is only vulnerable from the whitelisted ip address. Heartbleed/FREAK/etc doesn't work if you can't connect.

You are right of course, that unknown flaws in the device and security software do present an attack surface. But a few layers of real security are a reasonable defense. No security can't be broken, but even a wooden door with a residential lock is a huge obstacle relative to a literal hole in the wall.

We're talking about these systems being mostly completely wide open, protected by nothing more than a default password in too many cases, if that.

If we were dealing with systems that were secured; whitelisted ip blocks, knock and callback connection protocols, proper certificate security, etc. Sure they could still be vulnerable to flaws known or as yet unknown, and that'd be an important conversation too... but it would be a very different conversation.

A stock linksys router with one port forwarding to a SCADA system seems to be the bar right now. We need to raise that bar quite bit before we need to start worrying about firewall firmware flaws :(

But maybe you could possibly, at higher cost, make do with it not on the net. Maybe that should be considered.

It definitely should be considered. But for a lot of stuff, I think a properly setup VPN tunnelled over the internet to a monitoring station is pretty reasonable. Not perfect, and not for everything, but "for a lot of stuff".

I'd like to see the telcos step up and offer disjoint networks as a mainstream service. Where they'll dedicate a DSL line or whatever to a block of space that's only routable to a few other subnets all allocated to your own sites, and then let businesses (utilities, governments, etc) subscribe. So that you can connect remote locations fairly easily, without them being on the capital-I Internet at all. We used to do this sort of thing with dialup before the internet. We need to bring the concept back.

Re:Why?

[aaarrrgggh]

Many of these devices on the market are actually security gateways and supposedly have integral protections. Unfortunately, many are half-assed implementations.

Re:Why?

[cfalcon]

> Every single one of those exploits is mitigated by whitelisting the incoming ip blocks authorized to connect.

In the affected 32674 routers, yes. But remember, if the router is compromised, the technical limitation of that particular exploit shouldn't be something you use to judge other exploits or risks.

Yes, the IP whitelist would have saved you there. But if the device you are trusting to enforce the whitelist is ITSELF compromised (as in this case!), why would you trust the backdoored device to defend you?

All the next vulnerability has to do is have a hook into the firewall rules to special case its "secret knock", and then no firewall rule can save you.

Re:Why?

[Anonymous Coward]

This clown was a tenured professor at Yale, and was brought to the attention of the police because he was downloading kiddie porn to his computer in his university lab.
http://www.nytimes.com/2002/02/16/nyregion/former-yale-professor-gets-20-years-for-molesting-boy-he-mentored.html

Re:Why?

[Anonymous Coward]

They shouldn't be for the most part (though I can imagine possible exceptions).

Serial to ethernet converters, though, are damn useful. I've used them for control/monitoring of remote (and not so remote) sensors many times. I mean, would you rather hook the (invariably serial interface with an antiquated protocol) sensor to a serial->ethernet, plug it into a nearby switch and start programming; or would you prefer to run out a few 100 meters of serial cable (potentially through an industrial plant) and try to find a spare serial port on a dedicated PC, interface that to the server and *then* start programming?

Re:Why?

[dbIII]

Yes, which is why for years people have been strongly advising against putting MS machines out naked on the net without the adult supervision of something else between them and the wild internet.
Of course those other things could be compromised as well but with the MS stuff there is a very long history of problems due to an allow by default mentality instead of blocking everything apart from the stuff you know that you want.

Re:Why?

[gweihir]

To make things cheaper. Historically, these were going over phone lines or dedicated wires. Of course, if people with zero understanding of what Internet security requires implement such solutions, it will typically cause an epic fail. And look, it does.

Re:Why?

[gstoddart]

Because everything connected to the internet these days.

Even if your "remote" access is across the building, it's the protocol which is used, because it's already implemented.

Advantech advertises such products as a simple way to bring remote management and data accessibility to thousands of industrial devices that cannot natively connect to TCP/IP networks.

The bigger question is why do keep accepting that apparently complete morons are in charge of building these devices?

Hard-coded SSH keys is pathetic. Allowing any password whatsoever to open the device?? That's some epic fail right there.

If you are writing a security system (and I'll use that term loosely), if you can't be arsed to test what happens if you put in a wrong password ... you have no damned business writing a security system.

First two tests: good password, bad password. If you didn't test bad password, you're fired.

This can really only come down to sheer incompetence, or outright fraud. Either you're too clueless to be in the game, or you know damned well you've done a piss-poor job but tried to hide it.

I keep saying, until corporations carry penalties and legal liability for being incompetent/lazy/indifferent about security, not a damned thing will change.

Anybody who bought one of these things needs to be demanding their damned money back.

Re:Why?

[Anonymous Coward]

Well, it's not necessarily industrial, and not necessarily the Internet. I've used similar stuff to rig remote control of an amateur telescope and camera for astrophotography. Main difference is the run from a backyard installation/post/shed-type observatory to your house might be longer than serial or USB cable can support, so you rig USB or Serial to Ethernet and run it to your PC in the house that way.

Re:Why?

[thegarbz]

Now in the actual real world we don't have CIOs in industrial settings. The answer is not because we don't put intelligent devices in remote locations, the answer is because we don't want to pay someone 3 hours of time to drive out to the remote location to take a reading.

This has zero to do with the cloud.

Re:Why?

[Anonymous Coward]

If your using nothing but packet filtering to segment this part of the network you are doing it wrong anyway.

Take a casino camera system for example. The pit boss can check any camera from his iPhone right? Sure but first he has to VPN to the LAN, from there he STILL doesn't have access to the camera network. He has access to a single multi homed RTSP proxy sitting between the normal security LAN and the device network.

So in order to hack into any of these cameras you have to first VPN into the LAN, exploit the RTSP proxy all the way to remote execute and THEN wiggle your way into the device network where you can log on to individual cameras with admin/admin.

"proper" design assumes these devices have zero functional security. So it doesn't matter than I STILL know a half a dozen manufacturers hard coded back doors, I'm not going to be able to use it.

Unconfirmed issue with dropbear implementation

[slincolne]

The linked article states that:

"Note that it is unconfirmed if this backdoor account is reachable on a production device by an otherwise unauthenticated attacker"

Has anyone seen independent evidence that you can SSH into one of these devices with the password "remote_debug_please" ?

Re:Unconfirmed issue with dropbear implementation

[Anonymous Coward]

Nice out of context quote. That was entirely unrelated to the primary vulnerability (where a known password wasn't even needed) and was simply an added "oh, btw, we saw something else interesting while looking at the code, and think this might be what it is."

Re:Unconfirmed issue with dropbear implementation

[Anonymous Coward]

It's a tangential point, but I didn't get the impression that's it's maliciously out of context.

I'd love to know if remote_debug_please works, and if so, at what privilege level.

Go go IoT!!

[ErichTheRed]

This is going to get very interesting as the IoT bubble continues inflating. I'm not in the industrial space, but I do work in an environment with lots of legacy serial devices. There is serious denial that these things still exist to a big extent -- most non-technical people assume everything is USB or has some other connectivity. PC manufacturers have gotten away from shipping PCs with serial ports, and often the solution touted is serial-to-Ethernet bridges like the ones in the article. This is especially true as the pressure to lighten up the edge devices increases (i.e. replace a PC with a tablet.)

The truth is that in any vertical market, very little is done to keep up with security. Look at the link - it took from November 11 to December 30 for the vendor to patch the firmware, and this was for a public, open-authentication level bug. If the IoT is going to catch on, stuff like this needs to be fixed. You can't just put a magic "put it on the Internet" box in front of a legacy device and assume the vendor is doing everything possible to find and fix flaws. This goes double for stuff like serial gateways that don't get much use outside of a few key sectors. (Hint: those key sectors tend to control a lot of very important infrastructure!!)

Re:Go go IoT!!

[Anonymous Coward]

I've used several serial to TCP/IP device. ALL on dedicated internal networks only to ready serial interfaced scales. Can write to them if you want to. Well, I guess you could issue a tare command (if you know the syntax). Also all production networks are firewalled to prevent any type of external access. So I'm going to say that the network devices are just as secure as the serial devices.

I think where the problems comes is where production environments do not properly isolate themselves from the internet.
Every plant I work at has separate office and production networks.

Re:Go go IoT!!

[castionsosa]

The problem is that the companies that drop the ball when it comes to devices and security have no incentive to change their ways. Even if their device pops up a terminal server prompt and allows any intruder full access, would there be consequences. Even if there were, the EULA effectively shields the company from harm, no matter how catastrophic the damage is.

It won't be the IoT vendors who will be troubling themselves about security. It either has to be their customers who vote with their wallets, or the government... The ideal would be an organization like UL, but instead of testing to see if a product is safe to plug in a wall socket, checks basic security attributes via white-box and black-box testing. For example, on an Internet facing pipe, there needs to be a very good design reason for just a password to allow authentication, as opposed to IP restrictions, OTP mechanisms, or other 2FA restrictions.

Maybe one of the rules should be least privilege. Why should a device have an always-on 3G connection, when instead, it could use a hardened hub and communicate with it via Bluetooth? Maybe IoT devices should be on their own subnet, with a hardened hub acting as a gateway/firewall, and no direct Internet connection possible.

Basic security 101, but seems forgotten. Until security (not security theater) is as part of the design as the form and advertising, I'm steering well clear. With some things, they are truly "done", and anything added is just stuff that is irrelevant or gimmicky.

Re:Go go IoT!!

[dj245]

This is going to get very interesting as the IoT bubble continues inflating. I'm not in the industrial space, but I do work in an environment with lots of legacy serial devices. There is serious denial that these things still exist to a big extent -- most non-technical people assume everything is USB or has some other connectivity. PC manufacturers have gotten away from shipping PCs with serial ports, and often the solution touted is serial-to-Ethernet bridges like the ones in the article. This is especially true as the pressure to lighten up the edge devices increases (i.e. replace a PC with a tablet.)

The truth is that in any vertical market, very little is done to keep up with security. Look at the link - it took from November 11 to December 30 for the vendor to patch the firmware, and this was for a public, open-authentication level bug. If the IoT is going to catch on, stuff like this needs to be fixed. You can't just put a magic "put it on the Internet" box in front of a legacy device and assume the vendor is doing everything possible to find and fix flaws. This goes double for stuff like serial gateways that don't get much use outside of a few key sectors. (Hint: those key sectors tend to control a lot of very important infrastructure!!)

The article summary mentions "internet connected industrial devices" but these are just serial to ethernet bridges/servers. Just looking at some of their products , it is clear to me that this type of equipment is intended for closed, air-gapped LAN networks. Anyone who puts these on an externally-facing IP address is just asking for trouble. That's not the vendor's fault, that's just a very bad implementation by the end-user or network designer.

Re:Go go IoT!!

[gweihir]

This is going to get very interesting as the IoT bubble continues inflating.

I disagree. It is just going to be all the same very old problems again, this time even more stupid because are all are known.

This is what happens when "cheaper than possible" personnel implements security-critical functionality. These people have no clue what they are doing.

Internet connected? Cool !

[Anonymous Coward]

list the IP address in this threat.

Seriously?

[Shirogitsune]

This...this seems like the prelude to Terminator's Judgement Day.

Do you want Judgement Day? Because that's how you get Judgement Day!

Unauthenticated Root Access on Telnet port

[HighOrbit]

There are also some IP network connected medical devices with virtually zero security. Check this out. This was definitely a WTF moment.
https://ics-cert.us-cert.gov/a...
https://web.nvd.nist.gov/view/...
and http://www.securityweek.com/se...