Slashdot Mirror
Power Grid Insecurities Examined
Joe Barr writes "Chris Gulker has taken a long and careful look at the infrastructure of our power grids and has come to some rather unsettling conclusions." A good read that outlines where the current power grid is at, and suggests some paths for the future that may help avoid future blackouts.
In most states, if you generate your own power (ie solar), you can feed it back to the grid, and the electric companies are required to credit you! Any excess power you have can make you money. Sure, it's an investment up front to move to solar, but it is doable, and some states even offer tax credits.
A fundemental weakness of the grid is its over-centralisation. Another argument for environmentally friendly local power generation schemes. Cover your house with "solar" roof tiles that generate power that is fed back to the local grid, etc..
"You lied to me! There is a Swansea!"
However, reading the text, the problem seemed more that the plant operators had indiscriminately attached critical systems to the Internet without proper firewall security in place, which seems to me to be a human, not a computer or OS, flaw.
... for Verano.
And if you connect ANY critical operating system to the Internet, frankly, you're insane. There's no sensible reason to do so. Monitoring your systems is fine, that's what a management network is for... but the actual core of the critical system should be as close to that powered-down concrete encased computer as possible.
Subscribe for free to my show!
Legacy systems will provide more resistance to viruses than any MS based system mainly due to the lack of coders with the knowhow to write viruses for such systems. Though when paried next to and on networks containing Microsoft based systems a MSVirus could cause havoc just by crippling the network that those systems rely on.
In any case a system using NFS/NIS would be especially vulnerable to traffic floods by MSVirii due to the lockups that can happen when high traffic causes such file/security systems to fail.
I've seen flapping interfaces on certain cisco equipment that have made messes of NFS and NIS based systems requireing a total reboot of the entire network from the top down. And the flapping can be caused by recent MSBlaster virii that has recently seen action.
As a safety precaution the legacy networks should be extremely firewalled, and not allowed to work on any shared media that also caters to any Microsoft systems. Such seperation of the network would prevent either from spamming the other to death. Also in many critical areas private networks with private loops vs being carried over the internet should be considered with backups such a MicroWave or Sattelite communications to critical centers in case of any large infrastructure outages in your carriers network.
Best part of the article, and hilarious:
While legacy control systems are often UNIX-based ("Control-Alt-Delete scares power plant operators," Ahern said) and thus immune to MS worms and virii, their 10-megabit networking technologies can easily be overwhelmed. "Even the load from leading intrusion detection and monitoring systems can create a denial of service and shut these plants down," Ahern said.
everything in moderation
I work for a utility in protection and process engineering and we do not have any remote ability to change settings. As stated in the comment section of the article control and protection systems do not normally have any remote access even to on-site network operators. This philosophy protects everyone from the utility (employees/technicians) to the customer.
One key issue that seems to be on everyone's mind is the latest MS Blaster virus, could it have caused the outage? Not likely. As stated above our protection and control systems send data via leased phone lines and/or private fiber and do not have any connection to the Internet. Thus no possible way of receiving a virus.
Finally, to all of you who are dying and just can't understand why the investigation is taking such a long time...hang on! Part of my job is to study disturbances on the grid (ie why did the lights go out?). The studies take anywhere from a day to months to explain what happened. And remember the 1965 blackout study took over a year to finish.
...and many of you are liable to freeze (or in southern parts bake) in the dark. If it weren't for BC Hydro selling power to California's PG&E over the common power grid on the west coast it would have been a certainty. Moreover, PG&E DEFAULTED on MILLIONS of dollars owed for said power to BC Hydro--so perhaps the proper term would be BC GAVE California power. Sooo...who uses who's power grid?
Also, before you start singing a round of "Blame Canada" it has been determined to a high degree of certainty by industry experts that the most recent power outage originated in the US (notwithstanding out boneheaded prime minister's impulsive comments on the matter before anything was determined). One thing is for certain--it was the Homer Simpsons on BOTH sides of the border that allowed the outage to propigate to the extent it did (operator error, scheduled outages that left the whole system running at capacity, etc...).
Deregulation has been bungled in its implementation all over the continent, but moreso in the US and particularly in California (well...EVERYTHING involving goverenment in California is royally fscked and has been for the better part of the last decade). The process was always politicised and the fledgling market manipulated by the established players and governments no matter where deregulation happened.
The concept is sound however...creaky old mandated monopolies should be broken up and the system made as open as technically possible to as many potential generation sources as possible. Decades of monopoly (in generation particularly) set us all up for the situation we are in now.
As a result, we presently have a handful of creaky, large utilities running creaky, large power plants with obsolete technology--and newer technology tacked on with duct tape and baling twine with little attention to stability and security. This has nothing to do with what country you are in--it is the situation continent-wide.
I've worked in the industry and have seen it first hand--and this was BEFORE the industry was deregulated (they still had several 1988-era 386s and a 286 in use--in 1996!). The argument then was that competition would compel established players to innovate and become more efficient. NOTHING has changed in these plants since deregulation--they are moving no slower OR faster in bringing new capacity to the grid. Only now demand has reached critical levels as predicted by some years ago. Only the argument has changed. Now instead of being the solution, deregulation is cited as the reason for problems (careless cost cutting rather than being sheltered from competition).
I'm astonished (but not entirely surprised) that since I was last in a power plant that there has been enough integration of critical systems into the general network that blaster-like infections could disrupt operations. Back in the mid 90's where I was, there were two distinct networks with NO connection at all (be it physical or not). If course, the 'net wasn't what it is now either and dozens of on-site employees had to rely on a 56k leased line for outside access.
Hopefully the blackout made everyone feel vulnerable enough to wake up and put at least as much or more into security and stability as they did into y2k compliance...
The software and management side don't tell the whole story. Combine that with the power grid physical security and infrastructure issues and then you have a glimmer of how thin the electric thread we depend on really is. That's not being paranoid, that's being practical. It's a challenge from a cost position to be completely grid independent, no matter where you live. But it is feasible, at least technically, to be less grid dependent. The best cost/benefit balance I've found is to have enough wattage to run the refrigerator, water pump, computer (of course), furnace fan and some lights. Doesn't leave enough juice to run a central A/C, clothes drier, or the other big draws. You really learn just how much electricity we use when you design an alternative power system. And it costs a lot of money.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
Economics come in to play here a bit as well.
The market for buying and selling excess power is VERY active and exists primarily on the internet. Multi-million dollar deals are made quickly, and while they can be made in advance, they may also be made at the whim of mother nature (excessive heat causing a company to purchase power, or a drop in temp making excess power available).
Implementing the deal means interacting with control systems. I will admit to ignorance of how this happens exactly; but I suspect that the traders aren't driving to the power plant or transmission control centers and doing it themselves.
For a company that has efficient generation, they can make a great deal of money selling excess power. This means their customers don't have to pay quite as much.
Here is the real issue: Everybody wants better security; but just tell anyone that you're going to have to up their rates to provide it and see what the reaction is.
Hot Damn! It's the Soggy Bottom Boys!
Hackers controlling the power grid? Utter and total bull.
I work in IT for a major power company. Our control systems have never been hooked to our own network, let alone the Internet, and never will be. How stupid does this guy think we are?
We've been running computerized control systems in nuclear and other types of generation plants for years. We've had computers in substations and control stations monitoring, controlling and reporting status before most industries even knew what to do with them. I saw my first Z-80 processor in a SCADA system shortly after the Z-80 came out. It could talk any of 5 different control protocols and replaced 2 seven-foot racks of hot, high-current RTL and DTL control logic. It was a thing of beauty.
We're not newbs at this. And no way do any of our control systems run Windows. Get real.
Why would we even want to hook up a generating plant or substation to a network just so it can be controlled from anywhere in the world, BY ANYBODY? No way. No how. Nuh-uh. Ain't gonna happen.
We can't even monitor what's happening on the system from the company's own computer network. It's all totally seperate. And for good reason. Who wants a disgruntled employee or just some joker who's bored messing with the system? The only people who can make operational changes to the system are the people actually present at the secured control center or at the generation plants.
We run quarterly modem audits, company-wide, looking for unauthorized lines with modem. We even restrict who gets an analog phone line and whether they can receive calls on that line. Computers attached to the control systems get NO modems. Never ever.
Even our remote monitoring terminals at regional work centers require dedicated connections to the control center and are receive only. The control computers think the remote monitors are printers and only send data, not receive so they can't be hacked from there either.
It's impossible to get to our control system through the Internet. It could probably be done to some degree (perhaps sending a 'breaker open' command to a key substation, if you know which one), but only by hijacking an existing dedicated connection undetected, which is getting harder as we connect stations via fiber optic.
(Often we connect stations by installing the fiber near the high voltage lines on our towers, a security measure in and of itself. Imagine splicing a broken fiber hanging off a helicopter platform while the line 12 feet below you is energized to 350 thousand volts. No, I haven't done it, but I watched it being done and the crew earned every penny.)
If any utility out there has their control systems connected to computers that can be reached via the Internet (or modem for that matter), the persons responsible should be taken out and shot. Then taken to a doctor, stitched back up and shot again. Same for their bosses all the way up to the CEO.
Sorry if I seen a bit testy on this subject, the subject of keeping the control system secure has been drilled into me for more years than I care to remember. Now it's just automatic.
However, on the subject of aging infrastructure, I totally agree. I blame deregulation. Every utility is now trying to cut each other's throat trying to grab customers away from each other. To cut costs (and thus lower their prices to better compete), most if not all utilities have cut their expenses by eliminting maintenance, lengthening replacement schedules and cutting staff, specifically skilled line workers). It's a race to the bottom to see who can provide the cheapest service. And it will probably go on until the whole thing blows up on them. And unfortunately, us as well.
Beta sux! Join the Slashcott! http://hardware.slashdot.org/comments.pl?sid=4760465&cid=46173047
For purposes of this discussion, in the industry there are two things: generators and high voltage lines.
Now, once upon a time in the good old US of A, an official of a steel plant woe'd the outrageous slings suffered at being forced to buy energy from a utility due to that fact that his plant was located in said utility's fiefdom. In the industry, this is urban-lore explanation of how deregulation started.
Guv'ment steps in. There'll be no Ma-Bell style bust-ups; rather, generators will be managed seperately from HV lines. Energy from generators could be sold and purchased by company employees. The high voltage lines, however, were supposed to be managed by a company that also managed serveral other neighboring utilities lines, wherein a reliability advantage would be gained (by the super-regional managing entity) from seeing confidential, real-time, system information from several utilities.
How does the guv'ment force this? It can't; want's to, but can't.
How can the guv'ment encourage this? Money. Promiss to deregulate (remove price caps on) the renting of hv lines: be in a regional transmission organization (RTO) for two years or so, and regulate prices yo dam sef after that. As a taste, generation side price caps were removed right away.
The other selling point was a feel-good tactic. The islands-of-monopolies system hasn't led to inftrastructure upgrades that match demand, as each company *optimizes* like crazy to compete with the neighbors. Structure a business environment wherein an entity can develope that is soley about transmission, and things will take care of themselves.
With deregulation, and the price-wars over energy that immediately followed (i.e.: the greed that lured marketers - who control the generators - to prevent key generators from running, in kalifornia, just because they could make the 10000% mark-up (no joke) they felt they deserved and thereby causing a cooperation-dependent system to crash under the strain of all the bickering.) spurred a tremendous about of generation to be built, in both the form of large coal-fire plants as well as strategically placed gas-turbin "peekers".
Should FERC's simple deregulation goal for transmission ever be realized, it stands that the transmission infrastructure will see the same boon. In fact, the only RTO to date, MISO, has already laid out plans for new lines, with strong numbers indicating improved reliability and improved energy market.
On Enron . . . hehehe . . . deregulation did not take it down. Enron took Enron down. Bonuses were paid in advaced for deals made. Very DUMB in the high-activity, deregulated market! Many deals ran for years totaly tens of millions. Bonuses should have been paid out on a cash flow basis, i.e.: pay the bonus monies out as the energy (in the contract) is actually used and paid for). Secondly, too many *managers* were able to arbitrarily up the value on a previous contract. Why would they do this? It increased their group's bonus.
The genius of deregulation lies not in the ethics or ethos of capatilism, nor in that it lies in direct opposition to monoplistic tendencies. Rather, like the Linux world, where a vast number of minds focussed on an issue and produced a superior product, deregulation will increase the number of greedy bastards trying to meddle with the infrastructure such that it will accomodate their business deals. It's the number of minds brought to the table, despite their market economy drive, that makes deregulation a positive thing.
Oh, something worth noting: utilities are, for the most part, fighting deregulation. Compliances are half-heated at best and down-right subversive at the norm. "Believe everything you hear; nothing can be too impossibly bad." -Oscar Wilde
I work at a company where we sell grid control sofware (SCADA software for in-market lingo)
We had a product which used a particular UNIX, not a BSD or Linux, but the real high dollar, blessed by AT&T stuff. It hardly mattered because so many of our customers are not computer people, they are power engineers. They're not interested in event the user/group/everyone security model, they are interested in which breakers to open or close in the event a thunderstorm takes out this power line.
As a result, many of the UNIX systems were set up for conveinence, not security, and anything that reduced conveinence created cries of frustration from our customers (and developers). Eventually we succumbed to pressure from our customer base, and now large portions of our system have been replaced with MS Windows systems. The customers (our power companies) love it.
You can't sell security to those who don't want to buy it, but you can always complain when it's not there.
http://www.gepower.com/corporate/en_us/aboutgeps/
I think we should do what makes the most sense. For instance, if we're burning fuel to make heat and we need electricity too, we should look at heat engines to convert a little heat to power along the way. It probably makes more sense to create storable fuels via chemical or biological processes (like crop wastes or the hydrogen from algae trick) instead of converting solar or nuclear electricity into hydrogen. Then there are the no-brainers, like compact fluorescent bulbs, hybrid vehicles, insulation and daylighting. None of this is rocket science, it's just attention to detail.
Time is Nature's way of keeping everything from happening at once... the bitch.
Wish I had some mod points to add an insightful your way, because you're right.
I've seen some of these "isolated" power-grid lans compromised because it was "critical" that the data be fed into the marketing department or server appliations which determined optimal generation schedules based on the ability to sell "excess" power when it's most profitable.
The days of assuming you can secure via isolation are gone in the power market, but the debugging and testing cycles are so complete that it takes at least a year to implement a new anything. So despite CNN making this the "story of the year", a solution won't be available until well after the media decides that a particularly brutal murder is much much more newsworthy (or something to that efect).
Meanwhile thousands of developers that have always assumed their code was safe from attack because of physical (ie isolation) security are now scratching their heads on how to refactor these systems while trying not to be sidetracked by the security rabble-rousers who are asking if the system will withstand the latest exotic attack X (which requires someone to duplicate almost valid messages via a morris code trainer attached to an ethernet cable).
Unfortunately the most dangerous of these rabble-rousers come in two forms, lobbists and consultants. Although they complain the loudest about the problem, secretly they are in favor of keeping the problem around as long as possible because they only make money while it is still a problem. These people are rarely die-hard techs, but they know how to play the media like a violin.
If there's anything that 9/11 taught me (and should have taught the rest of us), it's that sometimes, the "best" attack is a low-tech one...
We can have high-tech biochemical sniffers looking for anthrax and C4, etc., but who really would have thought of stealing a plane or two and flying it into a building? Really - think about it. It's pretty low tech, but extremely effective...
Same thing with the power infrastructure - why worry about hacking in? Figuring out passwords and all that nonsense when the FUCKING INFRASTRUCTURE IS OUT IN THE OPEN!?!?!
Drive down any road - and you're likely to see a power line, a transformer, etc... I'm sure we ALL know where at least one substation or transmission line is located. AND they're out in the open...
Have the brains engaged yet? Think about it folks - dig out the old graph theory notes from your data structures classes and then plot out the national power grid -- just the big ole transmission lines...
What happens if you make some cuts in that graph? Wanna bet that about 7 pieces of wire would do it?
You don't even need explosives... some wire, maybe a bicycle chain or two and a modified potato launcher would do the trick... and blamo - lots of chaos and commotion... (and yes, I DO know someone who was a complete moron when he was 14 yrs old and tossed a bicycle chain into a transformer at a local substation.... but I digress).
How are you planning to protect the entire infrastructure against attack? Even if it's redundant, and resiliant - a bit of thought and you're right back where you started....
I don't have solution to this intractable problem - Do You?
Actually, they were wonderfully designed.
Read the research documentation that came out in the 80's, the pinnacle of SCADA system research.
Oh, and then that pesky TCP/IP became available, so people moved from tons of serial cables to cheaper CAT3/5. If you didn't migrate your system, you went out of business. Problem is, who could afford to re-design their software from the ground up to use a non-realtime network in a manner resembling realtime?
So SCADA has long moved from "real-time" to "really fast". Or they isolate the real-time requirements to parts of the system where it can still be achived.
Maybe at your company, Dilbert.
Ever look at the point where your company swaps information with other utilities? I don't mean credit histories. After all, nobody would be foolish enough to bridge the control networks with the outside world... and you don't have any TCP/IP running around between the pretty displays and those SCADA systems, right? And it's all the most modern internal IE based web stuff with active-X and java and a little
I mean, no PHB or mid-level manager in a deregulated utility with dreams of being the next guy to win the Montana Power and Light Institutional Ethics award would ever confuse all that fiberoptic cable running control information with, say, providing a little internet backhaul, right? And your exchange administrator, he's always on top of it, right?
Timmy... Timmmmmmmy... Wake up, Timmy - it's time for school! Hurry up, sleepyhead. Were you dreaming?
All I can say is: Imagine a world where an MCSE hefting consultant with little more than some Netbui LAN experience and a puffy resume managed to persuade the folks (who were a tad out of touch with this new fangled internet thing) that he had the answers about how to update the utilities and bring their business into the 21st century. Next, imagine how a fixation with buzzwords and a poor grasp of how the protocols actually work might lead you to build an ugly communications system that was so obviously crocked up from using access and visual basic that it made your head spin. Oh yeah, guess what else? with active server pages and cold fusion, it can be web based!
Now imagine connecting hundreds of power companies together to transfer electricity according to a message passing system (well, a handful of variants built by private companies to the half-baked specs mentioned above...) and then deciding that it was cheaper and better to send those messages over the internet. Unencrypted.
I can't tell you the number of times I've seen supposedly protected private internal "important business" networks bridged to the internet to overcome the limitations of legacy (read vax) hardware and an endless stream of rotating programmers with no ability to understand cause and effect. Or flat files. Or how to debug serial interfaces. The fact that these "private" networks actually pass all the SCADA information and decentralized control information back and forth mixed right in with MSN, AOL & yahoo messenger traffic might make one wonder, but heck, such important traffic will have decent firewall and IDS systems.
Yep. After making all the mistakes enough times to learn from them, the system will one day be pretty good. They're mucking about with XML now, and succeeding at turning really simple processes into expensive, unreliable software... which controls our grid. It's better now than it was. On the plus side, I get to go home early when we can't run the network because it's all dark. Sometimes I think having the occasional blackout keeps us from being too pompous and pretending that we actually understand and control everything. Thank
maybe not?
there was a story after the blackouts that back in Febuary 2003 a nuke power plant in Ohio somewhere lost it's safety systems for over 5 hours because of a worm/virus that took down the M$ system they were running. The story was on the news the same day they were reporting the Blaster worm messing up the switches in a Baltimore train yard. yikes!
I am sick of control, this might not be the right place to talk out about canada's problems in general but lets say the US already has control of our power, as proven with the california state vs bc hydro, they also control our lumber industry (softwood trade agreement), our wheat industry, our cattle industry (thanks to mad cow), we might aswell give it up or get invaded at this point. No one cares about us and we are so small that we get bullied into everything anyways. I say divert all the rivers leading into the states into the lower half of Alberta and Saskachewan (to those not familiar with canada its the 2nd and 3rd most western provinces) cut the power lines (thus fixing the grid problem), stop all exports and imports to the states, and give them the middle finger.
Or insecurities in computers.
Recent grid failure in the U.S. and Ontario may (likely?) be related to computer problems.
You are being MICROattacked, from various angles, in a SOFT manner.
1) Because a computer power supply says it's 400W, that does not mean that it's consuming 400W when you turn your computer on.
2) Solar cells manufactured 30 years ago still output 80% capacity or better. Solar cells manufactured today will last 40-50 years easily.
3) 7 kWH/day is more than enough for the average energy-conserving household. If you are an energy abuser, solar is not for you.
4) Oil companies own most if not all of the photovoltaic cell foundries. bpsolar.com is but one example.
5) You'd be a moron to use car batteries
6) No way in hell you're getting 20% efficiency out of commercially available solar panels. More likely 13-15%.
7) No way you're getting 8 hours/day useable, either
8) You are seriously misinformed about the mean solar flux density, which is generally accepted to be approximately 1362 watts per meter squared, slightly less at sea level, and not the 750 w/m^2 that you purport.
9) You do not document your (outrageously false) claim that solar cells do not return as much energy as is expended to create them. This is a common fallacy from the "conservation of energy" crowd. The solar cells do not themselves provide energy (you are not storing energy in the solar cell by creating it), so there is no link between the energy expended creating them and the energy they convert from incident raidation. The only conservation of energy argument that you can make about solar cells is that they output less than is incident. Anything else is completely specious.
10) Cost is not so much an issue in some places. http://www.dsireusa.org/index.cfm
In general, your rant is simply a torrent of ignorant, misinformed gibberish that is completely unmeritorious and not worthy of any consideration whatsoever. Your analytical methods are sophopmoric at best. Do your homework next time.
While the article was right when it comes to internal networks to the control stations (such as ISOs) the extent of insecurity in the energy bussiness is far greater that most people can think of. The fact of the mater is the reason most of the grid is immune to hacker attacks these days are the devices that control power transmission at the lowest level (relays, they control the circuit breakers) are all vt100/rs-232 terminal devices hooked up to aging modems 19.2 is the fastest I know of. Theses relays form the base level of what the power industry calls SCADA (system control and data acquisition). Unfortunately, the vast majority of relays still use the default password, and of course even if it is changed the password is probably going to be the same across all of a companies relays (I haven't seen a relay that has a password attempt lockout either). Of course nobody war-dials anymore so these devices go untouched. Security through antiquity.
I'm assuming whenthey say 10 megabit they mean 10 megabit ethernet.
Repeat after me: "Ethernet is not an appropriate networking technology for industrial control systems!"
This is exactly the type of environment that tokenbus (IEEE 802.4) was designed to handle. Tokenbus can guarantee QoS and does not require a "master" node, so it is immune to that kind of single point of failure. Tokenbus was designed with factory automation in mind - IIRC the major auto manufacturers in the US were big players in the committee - so it is optimized for the industrial environment.
FYI, tokenring is similar, but not identical. Tokenring is a simpler standard that requires a master node. A ring can be locked up if the master node goes into a strange state. Rings are fit for applications where a network failure would be inconvenient, not tragic.