Slashdot Mirror
Power Grid Insecurities Examined
Joe Barr writes "Chris Gulker has taken a long and careful look at the infrastructure of our power grids and has come to some rather unsettling conclusions." A good read that outlines where the current power grid is at, and suggests some paths for the future that may help avoid future blackouts.
Wonderful -- as I read the article, plastered in the center of the page is the ad:
... it frankly scares the hell out of me.
:)
"Microsoft - Big business ambition. Small business resources. Get your FREE 6-month trial now. Windows Small Business Server 2003".
The very fact that the power grid, atm's, so on and so forth -- hell, I worked on the power supply to a embedded PC today for a newspaper printing press that had NT on it
There I'll be sitting there in front of my OS X or Linux box. Can't be too smug I suppose with no power. No telephone. No gas. No cash to buy bread. Hell, the auto-checkout lanes (which I refuse to use on principle) at Jewel are Mickey-MouseSoft based. Certainly no Internet.
For my business' I absolutely refused to allow a Windows server of any type in the datacenter. I still say, "are you nuts?". Yet people still did it. Once again, Bill Gates will get a chance to screw us I guess.
So, when is the next worm due to hit? At least my TiVo will still work...
In most states, if you generate your own power (ie solar), you can feed it back to the grid, and the electric companies are required to credit you! Any excess power you have can make you money. Sure, it's an investment up front to move to solar, but it is doable, and some states even offer tax credits.
Says Skroch: "If you have too much security [i.e., no network connections], then the power plant probably won't work."
power plants worked long before the internet was created. no important computer controlling very important things should ever be put on the internet.
The article does bring up a valid point. Many times, when large systems attempt are forced into security by fear, they overdo it, and the system becomes nearly unusable to the users, who have to run around in circles with security measures.
The lesson? Security is nice, but lets not go biometrics and 30 different passwords just to check the email.
You will be baked, and there will be cake.
"The situation is so bad, experts say, that bored script kiddies could soon be knocking out power stations as easily as they concoct viruses from toolkits available on the Web."
:)
Is it any easier now then it has ever been? It always seemed pretty simple to me. Go down to your local, unmanned, power station and blow it up. Get your buddies and some trucks and knock down some high tension wires. wheeeeee.
Why do people get excited by this? It might be my misanthropic nihilism talking, but shit happens. Every day. Deal with it.
You might lose power, you might lose running water, you might get hit by a bus.
Even if you hole up in a shack to protect yourself from the script kiddies, psychopaths, terrorists and/or government... you're still gonna die!
Have fun!
--- Do you believe in the day?
The power industry needs to be reinvesting profits in infrastructure (powerlines), not stock dividends. The same companies should have been upgrading their command and control systems to prevent chain reaction blackouts. Am I expected to believe the computer systems that manage the cooling rods in the nearest nuke plant are secure?
Seriously consider the economic impact of the grid failure compared to the recent worm problems. Then think about a nasty combination of the two.
Karma: Censored (mostly affected by decency laws)
However, reading the text, the problem seemed more that the plant operators had indiscriminately attached critical systems to the Internet without proper firewall security in place, which seems to me to be a human, not a computer or OS, flaw.
Well of course Power Grid is feeling particularly insecure right now. I mean it's old and weak and obsolete and just got caught with it's pants down a few weeks ago. That kind of spectacular failure is bound to make anything or anyone feel pretty insecure. I doubt the last thing Power Grid wants is to have its insecurities examined publically! C'mon, people, let's not kick it while it's down!
watch this
... for Verano.
And if you connect ANY critical operating system to the Internet, frankly, you're insane. There's no sensible reason to do so. Monitoring your systems is fine, that's what a management network is for... but the actual core of the critical system should be as close to that powered-down concrete encased computer as possible.
Subscribe for free to my show!
Did anyone actually read this garbage before they posted it. This is absolute nonsense. The blackout had _nothing_ to do with computers, much less internet security. The blackout happened because a half-rate utility (First Energy) tried to squeak through an emergency without buying expensive power or shedding load. Period. They operated lines until the sagged into brush. Some small subtransmission and distribution lines had twice rated load. Do the math. That's four times the temperature or over 400C. That had zippo to do with M$ or any bleepin' computer.
It used to be that the utilities were highly regulated entities that had their profit margins basically regulated by the states they were in. They had to provide a given amount of reliability, and rate increases (and occasionally refunds!) were carefully scrutinized as to where the money went. You couldn't raise rates without showing some meaningful improvement that resulted from it.
Then along came degregulation, where the power seller and the power generator became two different things (which makes even less sense than the deregulated-but-shared local phone loop). Utility companies wanted out of the power generation arena -- too expensive, too many regulations, it was better to be in the new "commodity" end of the business, arbitraging power. So they split themselves into trading companies and generation companies, taking all the cash into the trading companies, who were deregulated and could spend it freely.
And then 10 years later, Enron and the whole deregulated power "market" has collapsed, and we wonder why we're 15-20 years behind the curve on power grid and other key infrastructure elements. All the money got spent on speculating in the newly deregulated power markets, and its all gone.
Nobody really pays any less for electricity, I don't have a bunch of people knocking on my door offering me their window electricity or biodiesel electricity or their pig shit methane electricity for that matter.
I only have the sheepish looking local utility trying to explain to me how they're trying to fix the power infrastructure built in the 1970s with the cash made in the 1980s which was spent in the 1990s on the promise of getting rich in the new millenium. When in fact, they actually need me to pay the prices of the next millenium for the service delivered in the 1990s, and, oh, would I please only use as much power as I did in the 1970s?
The valve at a dam probably doesn't need to be turned very often, so it's economically tempting to save the cost of 24/7 onsite coverage and have one central operations center.
Remote monitoring is all but imperative. The plants are already in a cooperative network sharing their power. Everyone on the grid needs at least basic information about what's going on.
None of which is ANY excuse for a direct or indirect connection to the public Internet. This is a job for a private network, and I don't mean a VPN that can be DOS'ed when a worm spreads through the public network.
Legacy systems will provide more resistance to viruses than any MS based system mainly due to the lack of coders with the knowhow to write viruses for such systems. Though when paried next to and on networks containing Microsoft based systems a MSVirus could cause havoc just by crippling the network that those systems rely on.
In any case a system using NFS/NIS would be especially vulnerable to traffic floods by MSVirii due to the lockups that can happen when high traffic causes such file/security systems to fail.
I've seen flapping interfaces on certain cisco equipment that have made messes of NFS and NIS based systems requireing a total reboot of the entire network from the top down. And the flapping can be caused by recent MSBlaster virii that has recently seen action.
As a safety precaution the legacy networks should be extremely firewalled, and not allowed to work on any shared media that also caters to any Microsoft systems. Such seperation of the network would prevent either from spamming the other to death. Also in many critical areas private networks with private loops vs being carried over the internet should be considered with backups such a MicroWave or Sattelite communications to critical centers in case of any large infrastructure outages in your carriers network.
I have taken myself off the grid years ago,using Solar,Wind,Hydro power(tapped into the abandoned Hardburly Deep mine and using the water to generate power) and have a 20kw diesel generator for backup.
Most of the power grid problem stems from the fact that very little maintainence is being done.The Power lines out here have been here since the late 1950s or early 1960. Every time it rains,you can watch an electricial light show less than 50 ft from my home.(Phone calls to the power co.does no good,so I informed the Public Service Comission about it,sending a video tape of the light show.AEP now has 10 days to change the lines out or get fined to the tune of $50k/day!)
Greedy utilities have brought this on themselves.Cutting jobs for the maintainence personell,doing nothing about aging lines, and then asking "WHY is this happening?
"We call ourselves Homo Sapiens Spaiens.Our true name should be Home Stupidus"
Geek Hillbilly
Haha, what grand scheme of things?
Humanity isn't trying to reach for the pinnacle of its capabilities, it's trying to find more comfortable ways to live and fuck.
People want more power so they can do more cool shit, and do it cheaper. That's it.
Leading in all forms of waste and corruption. Nice example for the future. Here's a primer on human nature -- more of anything doesn't make people use it smarter, it makes them squander it faster. Western society is terrible for this.
Your post is an attempt to be modded insightful by using big words to sound profound. Nothing you've said makes any sense.
occultae nullus est respectus musicae - originally a Greek proverb
On the verge of existence? That must have been Schroedingers' Bird - the last of which may or may not be going to have been eaten by a cat.
Recycle PCs and build a wireless community network www.hillsborough.org.nz
No, it's a Canadian unit - 1 Canadian meter is only worth .8 standard meters. ;-)
I work for a utility in protection and process engineering and we do not have any remote ability to change settings. As stated in the comment section of the article control and protection systems do not normally have any remote access even to on-site network operators. This philosophy protects everyone from the utility (employees/technicians) to the customer.
One key issue that seems to be on everyone's mind is the latest MS Blaster virus, could it have caused the outage? Not likely. As stated above our protection and control systems send data via leased phone lines and/or private fiber and do not have any connection to the Internet. Thus no possible way of receiving a virus.
Finally, to all of you who are dying and just can't understand why the investigation is taking such a long time...hang on! Part of my job is to study disturbances on the grid (ie why did the lights go out?). The studies take anywhere from a day to months to explain what happened. And remember the 1965 blackout study took over a year to finish.
...and many of you are liable to freeze (or in southern parts bake) in the dark. If it weren't for BC Hydro selling power to California's PG&E over the common power grid on the west coast it would have been a certainty. Moreover, PG&E DEFAULTED on MILLIONS of dollars owed for said power to BC Hydro--so perhaps the proper term would be BC GAVE California power. Sooo...who uses who's power grid?
Also, before you start singing a round of "Blame Canada" it has been determined to a high degree of certainty by industry experts that the most recent power outage originated in the US (notwithstanding out boneheaded prime minister's impulsive comments on the matter before anything was determined). One thing is for certain--it was the Homer Simpsons on BOTH sides of the border that allowed the outage to propigate to the extent it did (operator error, scheduled outages that left the whole system running at capacity, etc...).
Deregulation has been bungled in its implementation all over the continent, but moreso in the US and particularly in California (well...EVERYTHING involving goverenment in California is royally fscked and has been for the better part of the last decade). The process was always politicised and the fledgling market manipulated by the established players and governments no matter where deregulation happened.
The concept is sound however...creaky old mandated monopolies should be broken up and the system made as open as technically possible to as many potential generation sources as possible. Decades of monopoly (in generation particularly) set us all up for the situation we are in now.
As a result, we presently have a handful of creaky, large utilities running creaky, large power plants with obsolete technology--and newer technology tacked on with duct tape and baling twine with little attention to stability and security. This has nothing to do with what country you are in--it is the situation continent-wide.
I've worked in the industry and have seen it first hand--and this was BEFORE the industry was deregulated (they still had several 1988-era 386s and a 286 in use--in 1996!). The argument then was that competition would compel established players to innovate and become more efficient. NOTHING has changed in these plants since deregulation--they are moving no slower OR faster in bringing new capacity to the grid. Only now demand has reached critical levels as predicted by some years ago. Only the argument has changed. Now instead of being the solution, deregulation is cited as the reason for problems (careless cost cutting rather than being sheltered from competition).
I'm astonished (but not entirely surprised) that since I was last in a power plant that there has been enough integration of critical systems into the general network that blaster-like infections could disrupt operations. Back in the mid 90's where I was, there were two distinct networks with NO connection at all (be it physical or not). If course, the 'net wasn't what it is now either and dozens of on-site employees had to rely on a 56k leased line for outside access.
Hopefully the blackout made everyone feel vulnerable enough to wake up and put at least as much or more into security and stability as they did into y2k compliance...
The current problems with the grid are due to an un-ethical power struggle between the US and Canada. We need to phase into a system where a neutral party oversees the whole grid.
Ohmygod, this thread is sick!
Desi Noise, Live!
Hackers controlling the power grid? Utter and total bull.
I work in IT for a major power company. Our control systems have never been hooked to our own network, let alone the Internet, and never will be. How stupid does this guy think we are?
We've been running computerized control systems in nuclear and other types of generation plants for years. We've had computers in substations and control stations monitoring, controlling and reporting status before most industries even knew what to do with them. I saw my first Z-80 processor in a SCADA system shortly after the Z-80 came out. It could talk any of 5 different control protocols and replaced 2 seven-foot racks of hot, high-current RTL and DTL control logic. It was a thing of beauty.
We're not newbs at this. And no way do any of our control systems run Windows. Get real.
Why would we even want to hook up a generating plant or substation to a network just so it can be controlled from anywhere in the world, BY ANYBODY? No way. No how. Nuh-uh. Ain't gonna happen.
We can't even monitor what's happening on the system from the company's own computer network. It's all totally seperate. And for good reason. Who wants a disgruntled employee or just some joker who's bored messing with the system? The only people who can make operational changes to the system are the people actually present at the secured control center or at the generation plants.
We run quarterly modem audits, company-wide, looking for unauthorized lines with modem. We even restrict who gets an analog phone line and whether they can receive calls on that line. Computers attached to the control systems get NO modems. Never ever.
Even our remote monitoring terminals at regional work centers require dedicated connections to the control center and are receive only. The control computers think the remote monitors are printers and only send data, not receive so they can't be hacked from there either.
It's impossible to get to our control system through the Internet. It could probably be done to some degree (perhaps sending a 'breaker open' command to a key substation, if you know which one), but only by hijacking an existing dedicated connection undetected, which is getting harder as we connect stations via fiber optic.
(Often we connect stations by installing the fiber near the high voltage lines on our towers, a security measure in and of itself. Imagine splicing a broken fiber hanging off a helicopter platform while the line 12 feet below you is energized to 350 thousand volts. No, I haven't done it, but I watched it being done and the crew earned every penny.)
If any utility out there has their control systems connected to computers that can be reached via the Internet (or modem for that matter), the persons responsible should be taken out and shot. Then taken to a doctor, stitched back up and shot again. Same for their bosses all the way up to the CEO.
Sorry if I seen a bit testy on this subject, the subject of keeping the control system secure has been drilled into me for more years than I care to remember. Now it's just automatic.
However, on the subject of aging infrastructure, I totally agree. I blame deregulation. Every utility is now trying to cut each other's throat trying to grab customers away from each other. To cut costs (and thus lower their prices to better compete), most if not all utilities have cut their expenses by eliminting maintenance, lengthening replacement schedules and cutting staff, specifically skilled line workers). It's a race to the bottom to see who can provide the cheapest service. And it will probably go on until the whole thing blows up on them. And unfortunately, us as well.
Beta sux! Join the Slashcott! http://hardware.slashdot.org/comments.pl?sid=4760465&cid=46173047
A script kiddy would never bring down the power grid...If they did, they'd be bored out of their Internet-dependent minds. Can you imagine these types of kids playing scrabble or cards?!? Or worse yet, being forced to take the opportunity of a black-out to spend quality time with their families. The Horror!
--
Luck is just skill you didn't know you had.
If there's anything that 9/11 taught me (and should have taught the rest of us), it's that sometimes, the "best" attack is a low-tech one...
We can have high-tech biochemical sniffers looking for anthrax and C4, etc., but who really would have thought of stealing a plane or two and flying it into a building? Really - think about it. It's pretty low tech, but extremely effective...
Same thing with the power infrastructure - why worry about hacking in? Figuring out passwords and all that nonsense when the FUCKING INFRASTRUCTURE IS OUT IN THE OPEN!?!?!
Drive down any road - and you're likely to see a power line, a transformer, etc... I'm sure we ALL know where at least one substation or transmission line is located. AND they're out in the open...
Have the brains engaged yet? Think about it folks - dig out the old graph theory notes from your data structures classes and then plot out the national power grid -- just the big ole transmission lines...
What happens if you make some cuts in that graph? Wanna bet that about 7 pieces of wire would do it?
You don't even need explosives... some wire, maybe a bicycle chain or two and a modified potato launcher would do the trick... and blamo - lots of chaos and commotion... (and yes, I DO know someone who was a complete moron when he was 14 yrs old and tossed a bicycle chain into a transformer at a local substation.... but I digress).
How are you planning to protect the entire infrastructure against attack? Even if it's redundant, and resiliant - a bit of thought and you're right back where you started....
I don't have solution to this intractable problem - Do You?
Actually, they were wonderfully designed.
Read the research documentation that came out in the 80's, the pinnacle of SCADA system research.
Oh, and then that pesky TCP/IP became available, so people moved from tons of serial cables to cheaper CAT3/5. If you didn't migrate your system, you went out of business. Problem is, who could afford to re-design their software from the ground up to use a non-realtime network in a manner resembling realtime?
So SCADA has long moved from "real-time" to "really fast". Or they isolate the real-time requirements to parts of the system where it can still be achived.
I am sick of control, this might not be the right place to talk out about canada's problems in general but lets say the US already has control of our power, as proven with the california state vs bc hydro, they also control our lumber industry (softwood trade agreement), our wheat industry, our cattle industry (thanks to mad cow), we might aswell give it up or get invaded at this point. No one cares about us and we are so small that we get bullied into everything anyways. I say divert all the rivers leading into the states into the lower half of Alberta and Saskachewan (to those not familiar with canada its the 2nd and 3rd most western provinces) cut the power lines (thus fixing the grid problem), stop all exports and imports to the states, and give them the middle finger.
We're starting to see a few problems appear more than once, though.
-
Telecom vulnerability to power failure.
-
"Non-critical" systems that aren't.
-
Cross-connection between business systems and control systems
That's a more realistic picture of what's going on.AT&T was determinedly independent of the power grid in the days of Ma Bell. Every central office ran on 48VDC storage batteries, with backup generators. The backup generators were started once a week, and run for several hours once a month. Once a year, each central office ran for 24 hours cut off from external power.
That was a long time ago, back when AT&T was a regulated monopoly common carrier. In the new, competitive era, that depth of backup can no longer be assumed. Carriers in trouble (WorldCom, Adelphia) tend to cut things like that.
The details aren't in yet, but it's beginning to look as if, during the recent big blackout, some comm links went down very early, so that the fault information that's supposed to divide the grid cleanly into islands didn't get through. Once all the logs have been correlated, it will be clear what happened.
A few weeks ago, CSX, the railroad, had a shutdown due to a virus. Railroad signalling has used "code lines" for decades, for remote control of switches and signals. These are basically serial links over which commands and responses are sent. The safety logic is local, but if you lose a code line, the dispatcher can't throw switches and route trains.
The tendency to centralize train control has resulted in a need to transmit code line signals hundreds or thousands of miles. So they tend to be multiplexed over telecom-like facilities. CSX apparently routed theirs over their in-house general purpose network. The routers in that network were managed by a network management system that ran on Windows. When the Windows machines went down, system management of the routers stopped, and, after a while, this apparently took some key routers down. So a "non-critical" system actually stopped train movements.
It's really convenient to be able to see what the plant is doing from your desktop. Order processing is more efficient if the sales network connects to the factory network. Energy traders need to be able to see what the power plants are doing, and give directions to power dispatchers. These things all create vulnerable paths.
I'm assuming whenthey say 10 megabit they mean 10 megabit ethernet.
Repeat after me: "Ethernet is not an appropriate networking technology for industrial control systems!"
This is exactly the type of environment that tokenbus (IEEE 802.4) was designed to handle. Tokenbus can guarantee QoS and does not require a "master" node, so it is immune to that kind of single point of failure. Tokenbus was designed with factory automation in mind - IIRC the major auto manufacturers in the US were big players in the committee - so it is optimized for the industrial environment.
FYI, tokenring is similar, but not identical. Tokenring is a simpler standard that requires a master node. A ring can be locked up if the master node goes into a strange state. Rings are fit for applications where a network failure would be inconvenient, not tragic.