Researcher Claims Siemens Lied About Security Bugs

chicksdaddy writes "A month after an unknown gray hat hacker calling himself 'pr0f' used a three character password to hack his way onto Siemens software used to manage water treatment equipment in South Houston, Texas, a security researcher working for Google is accusing the company of trying to cover up the existence of other, more serious vulnerabilities in its products. Billy Rios has disclosed a range of vulnerabilities in Siemens SIMATIC software on his blog. The holes could allow a remote attacker to gain access to the Simatic user interface without a user name and password. Rios claims that he has disclosed the hole to Siemens and that the company has acknowledged the problem, only to deny its existence when a reporter asked for more information about the vulnerability."

Comments

Lose the remote...

[LostCluster]

The main problem these things have is that there's nothing more than password authentication protecting them from any random user getting in, and sometimes leak or get guessed.

For this kind of access there should be a technician dispatched to the site... no remote login should be allowed. Water control is a lot like Enron's electricity control in that a wipeout of any size can cause a complete mess of a local economy.

Re:Lose the remote...

[Endlisnis]

Welcome to the past, where everything is done manually! People are not interested in doing things in person, even important things. [Americans] want to bank online, have sex online, delouse their children online, purchase a home online, elect their officials online and everything else worth doing. Doing things is for suckers, but clicking a web-page and having something done FOR you is for winners.

Re:Lose the remote...

[plover]

I don't know about your community, but mine complains incessantly about taxes. If we had to have full-time SCADA engineers guaranteeing on site support 24x7, we'd have to pay more for water, sewer, gas, electricity, street lights, traffic control signals, and all those other industrial controllers that are hidden under little green boxes on the side of the roads.

And I live in a large, wealthy city that could afford such amenities. I'm picturing the poor bastards in Bumfuck, Idaho*, population 174, located three hours from the nearest grass-strip airport. Who exactly is going to monitor their town water pump and filtration plant? Are you and every other taxpayer going to agree to pony up an extra $500/year to have a SCADA engineer sitting in the town bar all day and night, just waiting for your pump to croak? Or are you going to contract with REMOTE-SCADA-R-US.com to remotely monitor and maintain your plant, and possibly fix issues in minutes instead of days?

I'm not saying that they should just plug it into the internet and walk off. But disconnected isn't even an option for a lot of installations.

*My apologies to any fatherless indigents living in or near Bumfuck, Idaho. I'm sure you're all very nice people.

Re:Lose the remote...

[inhuman_4]

The problem is not that they allow remote access. That is obviously useful. The problem is that they rig the thing up on the internet, sans firewall, where anyone can hammer away at it.

I don't see any reason why a company cannot run all of these systems through a a VPN where only the only people allowed on the VPN is the engineers who need to be. I suspect they they don't do this simply because they are lazy or incompetent.

Re:Lose the remote...

[gl4ss]

if you don't even need a password, there's no password authentication on the actions taken on the server even..

Re:Lose the remote...

[LostCluster]

A VPN is usually only protected by a password or maybe a stored secret which is just a more complex password. There needs to be better authentication such as calling headquarters, being recognized by voice, location, and password and then having them agree your move makes sense before it's put into play.

Huh.

[jd]

I seem to remember seeing SCADA vulnerabilities being added to vulnerability testing tools and IDS systems recently -- anyone know if this is related (ie: the tools now check for these non-existent flaws) or if the additions were to cover previously-reported bugs?

If the former, Siemens had best fix this damn fast. Infrastructure companies are in a corner - they don't have the cash for a major migration and alternative vendors are hardly thick on the ground. Some will be unable to afford decent security and others will be too politicized to secure their networks. Much of the infrastructure is too big and/or too expensive to duplicate, so the market is useless. The only place this can be fixed is at Siemens itself. The others that technically could won't and the rest can't.

The problem with the current paranoia over security is that you can't fix a fault you won't admit exists, companies won't deploy a fix if you tell them it's not needed, and so what you're ultimately left with is not security, merely obscurity.

Re:Huh.

[Anonymous Coward]

ISS started adding SCADA checks to it's Proventia line and it's scanner right around the Stuxnet publications. But, it's hard to test or verify the information. So, they basically have to fake it. Which sucks, because often how they claim something works is not how it really works. And, the only way to verify a problem with a related-check is when a customer files a report. Which again, they'll probably only bother in the event of a false-positive (they won't even look further if it's a false-negative). Basically, those checks are mostly worthless and I'm sure it's the same with their competitors.

Re:Huh.

[rahvin112]

The solution given the reality of Siemens is to simply disconnect these SCADA systems from the internet. Why anyone would hook up industrial controls to the wider network. And yes I mean every single workstation that has access to the SCADA machinery should be disconnected from the broader internet. If that means people need two computers on their desk that's the solution. If that means you have to dispatch someone to the office to fix things, that's better than some hacker causing a massive failure by mis-configuring valves intentionally or through ignorance.

I'll tell you something, these local municipalities can afford a little overtime and computers far more than destroyed infrastructure.

Re:Huh.

[Anonymous Coward]

That works - right up until somebody hooks up a cross-over cable between two networks, or management says "It takes too long that way, just get it working, we'll fix it up when we get the budget."

You're absolutely correct that these systems shouldn't be on the Internet - shouldn't be reachable from the 'net, shouldn't even let workers plug in USB keys (or hard drives, or CDs, or ...). But unless and until there is criminal liability put in place for this sort of thing, it's just going to keep happening. Basic human nature. Security gets in the way, so security goes out the window, until something goes wrong ... and then security comes back in for six months (or the auditors go away), at which time it goes out the window again. (Or the auditors don't give a damn, they just want to check the box on their report. "Are backups being done?" "Yes, they're being sent to /dev/null." "Great! *tick*.")

The short of it is: every proposal to fix this must take human nature into account. Until that's done, we're pissing into the wind.

Re:Huh.

[jd]

Precisely, and trying to shape human nature through regulation is where you run into the politicized nature of these environments. Yes, these systems should never be connected to the Internet (even by indirect means such as sneaker net) and yes fixing the problem must take human nature into account, but it must do so cleverly because applying the fix also involves human nature. That's a tough one.

Re:Huh.

[Kyusaku Natsume]

The easier fix would be to provide a nice machine with almost fully open access to the web and/or internet, and a good enough machine to keep control of the SCADA system, that way you have your system operators happy, alert all the time and without any need to break security procedures.

pr0f exploit a hoax?

[Anonymous Coward]

I may misremember things, but wasn't the whole water-treatment plant 'hack' a legitimate worker logging in from his holiday in Russia, and some wannabe hacker claimed responsibility with fake screenshots?

Re:pr0f exploit a hoax?

[Em Adespoton]

That was a different water-treatment event; in fact, it's the one that prompted pr0f to pull his attack, because nobody was taking the security holes seriously: http://nakedsecurity.sophos.com/2011/11/22/interview-with-scada-hacker-pr0f-about-the-state-of-infrastructure-security/

Verticle markets

[U8MyData]

I has been my expereince that many verticle market applications are poorly written with security being the least of their concerns. Now that *everything* is seemingly connected, extended and exposed to the internet it is only compounding the problem. Security needs to be built in to the core development tools tightly in order to stop these kind of things from happening. The last thing an application developer wants to worry about is security getting in the way of his/her code development. If I see another VB6 app out there I am going to, well, it's not pretty.

Re:Verticle markets

[Anonymous Coward]

If I see another VB6 app out there I am going to, well, it's not pretty.

You is go horizontle?

Um, no, that's a BAD idea

[gerf]

The problem is not necessarily with Siemens. Industrial controls in general are not inherently meant to be accessible over large networks. They're designed to run reliably as they are, not with patches and updates. This applies to anything from Siemens/Fanux/Rexroth/Allen-Bradley/Mitsubishi to Cognex cameras to ABB/Fanuc/Kuka robots, or any little bastardized system in between.

Why not? Well, there is a ton of weird, unique software that runs on industrial controllers. They run some really embedded HMI (Human Machine Interface) software on top of, say, XP Embedded, or even NT4 or Win2k or some Linux flavor, or WinCE. If you start throwing out patches to those systems, there is a very very good probability that at some point, the system that you are updating will fail due to the update. Heck, Siemens updates regularly break its own software, much less Windows patches. If you try, and screw things up, you're forced to revert to some old dated backup or Ghost image stored in a filing cabinet on a CD-R or server if you're lucky. If you're not lucky, you call the vendor in to fix your broken system. Hopefully they are competent enough to have a backup from their last visit 6 years ago, and work from there, losing all your work in the meantime. So, you have machine downtime of hours, days, or even weeks if you're not lucky. How much does downtime cost? It depends on how many systems you took down, and the product. Conservatively, anywhere from $5,000 to $1,000,000 per hour.

What to do? You obviously can't push out patches. But, there is a lot of good that comes from monitoring machines, their productivity, uptime, faults, etc, remotely. By taking these systems off of an internal network, you also lose productivity in efficiency losses. So, you're forced to be the High Priest of IT and lock down a network like no other. No outside USB sticks, manufacturing firewalled off from the rest of the plant, and all kinds of restrictions that make users angry. It sucks, but it's possible. Unfortunately, small time manufacturers with their one part time learn-on-the-fly IT guy probably won't do it right. Perhaps this is where the DHS can come in to help, in the name of national security?

Re:Um, no, that's a BAD idea

[plover]

The problem is not necessarily with Siemens.

The problem lies squarely with Siemens. They made the choice to build systems with expected design lifetimes of more than 20 years on top of platforms that have a known support lifetime of less than that.

Their needs for OS services are not great. They could have developed their own operating system. They could have bought a small embedded operating system that they would then support in-house. They could have licensed an open source OS such as FreeBSD. They don't need a GUI. They don't need audio support. They don't need to support eSATA drives or Bluetooth or mice or USB or nVidia or any of the thousands of drivers that Windows ships with by default.

Everything about these systems needing to be super-long-term stable has been known in this industry since the early 1980s. Their engineers have understood these principles for as long as SCADA has existed. They knew better than to pick a flavor-of-the-month consumer OS. And they still did it. I don't care if it was a manager who thought "Microsoft Windows programmers are a dime a dozen so we can do this on the cheap," or if the CEO was photographed in bed with a Microsoft swallow, they were responsible for the choice.

Siemens knowingly chose to build their entire SCADA empire on top of an insecure platform. And that is entirely their fault.

Re:Um, no, that's a BAD idea

[thsths]

> The problem lies squarely with Siemens.

> They don't need a GUI.

Actually both is wrong. They need a GUI, and the problem is with the customers. The customers chose Siemens over some other SCADA solution exactly because it runs on Windows. Siemens made this decision very early on (I think in the NT4.0 days), and from a business perspective it is more successful then they could have imagined.

Technically it was wrong, I agree there, but would you rather go bust with the right technology or prosper with the wrong one? Because there were other big players in the market, using HP, Solaris etc. And they are all in trouble now both with the technology and with the business.

And to be honest, if you have an incompetent IT department, do you think they can secure an old version of HP/UX?

Re:Um, no, that's a BAD idea

[Twylite]

You are ignoring the essential role of HMI in SCADA systems. A SCADA can acquire data and coordinate components without a UI, but operators cannot monitor a plant or take corrective action without an HMI.

The HMI is graphical and allows the operator to override normal operation in order to respond to abnormal situations. It needs all the input and output devices a normal workstation requires.

You are also ignoring the issue of data storage by SCADA systems, and the generation of reports on that data which are used by various business departments in real-time. A Manufacturing Execution System may provide real-time reports for sales staff so that they can give customers accurate estimates of completion/delivery dates. Orders are added to a queue and will be automatically executed by the SCADA. Stores will receive low-stock notices for just-in-time ordering. Line stops exceeding 2 hours will result in automatic escalation to the COO.

This level of automation brings huge business benefits. The business is more responsive to customer needs, and there are fewer manual steps involved in completing an order (leading to fewer mistakes, less waste, fewer unsatisfied customers). The downside is that the business network is directly connected to the MES and the SCADA in a manner that allows at least some commands to be issued (as opposed to having read-only access to a database). An air-wall is not possible.

So now you have PCs on the business network able to interact with a MES which is necessarily able to access the network with the SCADA and HMI. And there's a 100% chance that the business PCs have e-mail access, which means that somewhere there is a physical cable to the outside world.

They could have developed their own operating system

Yeah, because they have extensive expertise in OS development and oodles of cash to throw at the problem, and as we well know the available commercial and free embedded OSes never have bugs.

The problem is that the environment is not conducive to upgrades/patches and is hard to isolate logically. The economic reality is that for any given SCADA environment the risk* inherent in regular upgrades is larger than the risk of a malicious attack (for now).

* = (likelihood of event) x (cost of event), where cost includes recovery plus the direct and opportunity costs of downtime.

Re:Um, no, that's a BAD idea

[catmistake]

The problem is not necessarily with Siemens

Hackers certainly aren't to blame. Generally ranting here, but what I've seen make headlines is the 'hacker' (definintion in contention) equivalent of waking up one day and finding water all over your bathroom and realizing someone's been in there splashing water everywhere. Besides the overreaction to all it is, which is just annoying and expensive (time) is the overengineering to prevent it from happining again. Serious people seriously doing serious work seriously... we are the easiest marks. These kinds of 'hacks' and interested perpetrators are just as susceptable as the bathroom will always be to breaches using the same bag of tricks against the 'hacker.'

This is why marketing should merge with IT. Using well established PR techniques, and without having to waste time ever expensively fixing code that kind of already works; all that is necessary is to campaign against the personality, individual, group of individuals, activity or even thought of activity that actually utilizing that or any security holes is whatever we say it is. Besides distraction or diversion, another possible stacked method is to draw attention to a scapegoat, and the security researcher (or loosely put, 'hacker') could be used, as could any thing/one. To illustrate an example, if the view of doing anything unorthodox could be coaxed into sharing the reactions to other or all unorthodox activity, then anyone that finds a bug or any activity possible beyond that could share the same public condemnation of the pedophile, date-rapist or whathaveyou. Point being, from a business perspective, manipulating people is far less expensive than manipulating code. IT is all about work smarter not harder. If breaching code can cast a dubious new quality of 'insecure' on a system, then its valid to say that if breaching code were socially uncool enough, individuals wouldn't engage in it anymore. To prevent governments from doing it, the method we know that works involves leaking footage of them relentlessly spying on us, which explains all the water everywhere.

Re:Um, no, that's a BAD idea

[plover]

I saw HMIs back in the early 1980s that were built on dumb terminals, with colored line drawings on a 25x80 screen, and the menu represented by a series of function keys listed across the bottom of the screen. There's even an ancient serial terminal in the North Star Building's elevator lobby in Minneapolis that still displays a curses-drawn picture of the elevator system operating. These screens are certainly adequate to graphically represent the systems they are controlling in a very understandable manner, and if you follow the links in TFA and take a look at those screenshot pictures the guy grabbed, those drawings that South Houston uses were not vastly improved by the addition of clip art pumps. Windows(TM) was not responsible for carrying lots of meaning to the operators.

I completely understand the desire for systems that are easy to access, that integrate better with the business, and the desire to maximize profits by selling cheap, off the shelf solutions. But even back when they were switching to Windows NT in the mid '90s, security was a prominent issue. Viruses were widespread. Engineers were appalled that people were going to use Windows in various real-world system controllers. And Microsoft had long established the practice of periodic upgrade paths, requiring newer and bigger hardware to keep up. Every one of these problems was well known and understood to present a real risk, yet they did it anyway.

Even picking a different commercial graphical OS would have helped somewhat. Heterogeneous systems have an inherent resistance to accidental security problems, and provide a good buffer against stupid malware problems like Windows viruses. Of course, we know heterogeneity did not save Iran from Stuxnet. Stuxnet's Windows payload knew how to cross the boundary and attack their SCADA network, too. But the non-physical attack surfaces of the SCADA controller could have been realistically reduced to almost zero on a proprietary OS. The Windows XP controller they ran was Swiss cheese, and was penetrated by 4 different zero day exploits, including both network and USB vectors. Had that system been running QNX, for example, and loading new configurations via ancient bubble memory cartridges or paper tape (or even a simple serial cable), Iran would probably still have their enrichment plant up and running at full capacity.

Siemens could probably even keep their current sketchy Windows systems as the controlling systems, if they were to be just a tiny bit smarter and place a firmware-based firewall in front of any system updates. A tiny microprocessor program running on an embedded platform could operate the USB port they upload new programs into, then transmit those programs onto the controller via serial cable. One very simple, fully verifiable service could listen on that serial port, and fully validate every byte of input before allowing any of it to be stored. While this wouldn't stop malicious machine code being delivered, it would prevent the real time hijacking of the controller itself.

Re:Um, no, that's a BAD idea

[gl4ss]

embedded operating system wouldn't have helped.. when network was never meant to be attached to public internet in the first place.

Re:Um, no, that's a BAD idea

[plover]

embedded operating system wouldn't have helped.. when network was never meant to be attached to public internet in the first place.

A custom embedded OS, one they could maintain themselves, would have helped tremendously in preserving the longevity of the investment. If the controllers back then had 8MB of RAM, they simply would not have added crap to their OS patches making them require machines larger than 8MB. They had full control of the hardware (they were building it, after all.) An embedded OS could have been maintained for many decades while remaining within the 8MB constraint of their oldest systems. Windows, on the other hand, set the standard for needing ever increasing amounts of RAM on a very short cycle, which no hardware available in 1995 could currently contain.

Building a factory is a giant company-risking investment. It's an all-in bet with the expectation of many years of future sales paying off the mortgage. As such, many of the systems they're built with don't change over a long period of time. I know of a machine shop that is still running equipment originally built to make parts for the second world war. Over the last 15 years the owner has slowly been converting the cam driven equipment to CNC control, but that's a very expensive investment for a small business. When you're creating a system that you know could last 60 years, making a choice of OS that has had no example of support lasting more than a decade is irresponsible. Companies like Siemens are why Microsoft has offered extended XP support all the way until 2014, but that's little consolation to anyone who expects their Siemens equipment to last until 2050.

Re:SCADA is internal stuff

[KXeron]

The huge problem is that the CEOs and other management of industries do not understand, nor do they want to understand security. They want the SCADA systems to be online so HQ can check up on them without sending anyone out and without needing to contact any of the operators. They want to be able to audit certain parameters and such however don't want the added expense of running a physically airgapped network, they see the Internet as "convenient" and "cheap", the only things that matter to them.

This problem will not go away until management of these companies are personally held legally (I'd argue even criminally) responsible for malfunctions and intrusions as a result of an internet connected system.

Is it still lying if you don't know you're lying?

[arglebargle_xiv]

The OP claimed that Siemens lied about the security of their SIMATIC controllers, but don't you have to know you're lying in order to lie? Having dealt with Siemens over these things in the past (at one point we debated flying someone to Munich to club them repeatedly over the head until they realised there was a serious, showstopper flaw in their control system), it's quite probable that they genuinely believe that they're secure. We ended up using Allen-Bradley gear in the end, which also sucked, but not as much as the Siemens stuff.

Re:Is it still lying if you don't know you're lyin

[dropadrop]

The OP claimed that Siemens lied about the security of their SIMATIC controllers, but don't you have to know you're lying in order to lie? Having dealt with Siemens over these things in the past (at one point we debated flying someone to Munich to club them repeatedly over the head until they realised there was a serious, showstopper flaw in their control system), it's quite probable that they genuinely believe that they're secure. We ended up using Allen-Bradley gear in the end, which also sucked, but not as much as the Siemens stuff.

That could be used as an explanation to escape just about any lie.. :)

I guess the point is, that if a security researcher sends you detailed information on vulnerabilities in your system, then either don't answer, or give a decent reply. If after 6 months the Siemens guy was not lying it means they are not very competent. It's not like this was a complicated issue...

Re:Siemens sucks

[thsths]

You misunderstand Siemens. It used to be a great electrical company, but nowadays it is an investment bank. What sets it apart from other investments banks is that they deal in technology (and foreign exchange), and they have a very good understanding of technology. The production/engineering/service segments are really just used for better investment leverage. That does not necessarily mean that the engineering sucks, actually some of it is brilliant. But either way it is not as relevant as it may seem.

Re:SCADA is internal stuff

[rioki]

Yes, they want that. But there are technologies to enable this over the network... Just use a VPN and be done with it. The requirement is legin, the execution was lousy.

Re:Siemens sucks

[Almost-Retired]

No, I don't think they "have a very good understanding of technology"

Many moons ago, when I setup my first NATed local network here, I bought a Siemans router. I set it up with a 12 character PW for admin purposes, the maximum it would allow. It was rooted and bricked 3 days later. If it was that easily attacked, I sure as hell didn't want it and took it back to Circuit City. They agreed, and weren't surprised that it came back.

So I next brought home a Linksys BESFR41, which in a pinch I can still use. But it was eventually replaced with dd-wrt running on an old x86 box, whose radio never worked despite registering it, so now I have a netgear something or other whose radio used wpa2 with about a 120 char passphrase, and Just Works(TM).

Maybe things have changed in the last decade, but I personally don't use the word Siemans and technology in the same sentence.

Now, for the person who used Bumfuck, Utah as an example, what makes you think they would have anything more sophisticated than a pressure switch, adjusted for the height of the water storage tank, to control their water pumps?

Sheesh, that ain't high tech, needing a computerized system to run it. The town clerk probably goes around reading the meters & sending out the bills on an old pentium powered box running winderz 3.1 from floppies, likely without any connection except the printers parport cable.

So Bumfuck, Utah's water supply is not subject to a terrorists attack via this here intertubes, and far safer than any bigger towns that is all "modern & computerized".

Cheers & a merry Christmas to all, Gene