Software bug was just one part of bigger problem
on
Tracking the Blackout Bug
·
· Score: 5, Informative
The software bug was just one piece of a much bigger problem; I wouldn't want to overstate its' role. There were many other factors; here are just a few:
Poor vegetation management probably played an even bigger role as overloaded power lines warmed up, expanded and sagged into trees and bushes that were supposed to have been cut back.
Poor communications between utilities played a major role.
This whole section of the transmission system was known to be unstable.
An inadequate regulatory structure lacked teeth to deal with known problems.
Lack of adequate transmission line capacity
If all these other problems hadn't been in place, the software bug might never have surfaced. And certainly, the rpoblems would have been contained within a much smaller area -- maybe just First Energy's service area.
An article featured on Slashdot last year lays out the underlying complexity of the power grid very well: "The World's Largest Machine"
I think that the trial lawyers will take care of this issue for us. People like to fuss about trial lawyers and there certainly are abuses, but in this case, I expect that First Energy will get burned pretty bad in court.
There's a school of thought that says that whether it's by intention or default (probably default), in the U.S. we have trial lawyers and the court system filling a role played by government regulators in Europe. Even with the abuses of the legal system, some folks believe that we're better off with more lawsuits and fewer regulations. The reason given is that regulations don't always have flexibility to adapt to new situations quickly where as tort law + punitive damages does.
I'm really not a fan of trial lawyers, but I see how they play a useful role.
Slashdot had a great story on the blackout last year:
Guinnessy writes "The latest issues of the Industrial Physicist suggests that 'the vast system of electricity generation, transmission, and distribution that covers the United States and Canada is essentially a single machine -- by many measures, the world's biggest machine.' The article says that because deregulation ignored the physics of the machine, we have blackouts, a fact the industry warned regulators about in 1998. It has some nice hard science data for those interested in why we're going to get some more blackouts in the future unless Congress gets its act together."
I work with power utilities -- this is the best single explanation I've seen of the underlying problems of transmission management and regulation in the U.S.
In WW II, several U.S submarines are believed to have torpedoed themselves. One known to have had this happen was the Tang (there were some survivors).
The torepedoes were defective.
Modern torpedoes have interlocks to prevent circular runs; if the torpedo turns through too large an angle (say 300 degrees), the torpedo either shuts down or at least dis-arms itself.
The "killer app" for many FTTH projects is -- get this -- responsive, locally-based, reliable service.
U.S. municipal power utilities are currently building FTTH networks to serve 100,000s of customers.
Most of these are built in small towns that have endured wretched service from their incumbent telephone and cable TV incumbents. Local residents want an alternative and turn to local government.
For a decade, small towns have successfully built and operated cable TV systems using HFC (hybrid fiber coax) technology.
By about a year ago, FTTH costs had dropped low enough to make it actually cheaper for a power utility to run ADSS fiber cable than coax. So these FTTH projects are just an extension of a trend that's been going on for years.
"The current technologies are still pretty much limited at 40Gb/s for one single fiber."
That's true, so then you deploy DWDM (dense wavelength division multiplexing) to multiplex 50 or 100 (or more) wavelengths of light, each carrying 10 or 40 Gb/s in traffic.
Add to that all the dark (unused) fiber deployed in long haul terrestrial networks in the U.S. and we have a lot of backbone fiber capacity. Typical fiber counts on the long-haul cables deployed in the late 1990s were 144 to 288 fibers or more.
There are many more North American cities with fiber systems we didn't design. The weblog Community Broadband Networks has links to a number of them.
The weblog also has a summary page with about 1800 article links you can skim. About half cover municipal broadband projects of some sort.
In Utah, incumbent monopoly telco Qwest's modest investment of $50,000 in campaign contributions and its' powerful lobbyists (one is the son-in-law of the State Senate President) may be enough to kill the UTOPIA 18-city initiative to build a publicly-owned FTTH (fiber to the home) system. A bill (openly crafted by Qwest) that would effectively outlaw city's financing the project sailed out of the Senate and threatens to become law. This action comes after 18 city councils have voted to join UTOPIA and 6 have already made financial guarantees . The UTOPIA system is based on an open-access model allowing multiple competing providers to offer voice, data and video services to subscribers.
This comes as the Salt Lake City Tribune, a strong foe of the UTOPIA initiative, ran an article wondering why Utah is losing its' position as a major technology center.
There are many confused and mis-informed comments on this subject here. If someone's really interested, they can take about 15 minutes to read the actual text of the FCC's notice of proposed rule making (NPRM).
I've got more background on my blog, which cover BPL, FTTH and wireless broadband news. (You can also search the archives using the built-in search function).
Finally, the Virginia Journal of Law and Technology had a draft article on the technology and legal issues that was posted on the FCC's web site a month or two ago.
Reports of BPL's demise in Europe are premature.
The European Union is spending many millions of dollars on developing and promoting a continent-wide BPL standard:
The FCC's notice specifically notes that the Japanese government in January opened up to BPL trials:
http://www.soumu.go.jp/s-news/2004/040121_1.html
That document is in Japanese; you can get an approximate Babelfish translation at:
http://babelfish.altavista.com/babelfish/tr
BPL (broadband over power lines) is not economical in truly rural areas where the power utility's "line density" is typically 10 customers per mile or less. The system's attenuation requires to many repeaters for the number of customers.
Unlike Canada, where many small towns are getting DSL, in the U.S., many small rural towns have been written off by the big Bells and there is no broadband available. Even in a really small town, line densities are typically 50 homes per mile, which is ideal, economically for BPL.
The majority of folks in the U.S. that live in what are considered rural areas actually do live in a small community, not on a farm.
Even in big towns, one Amperion wireless unit does not serve that many homes.
"They are not transmitting shit over powerlines. They are transmitting over a fiber optic network which they installed with the power lines, then using 802.11b for the "last mile" to the consumer homes.This is not the same thing as the internet over power line debate, with all of the shielding and signal issues."
Wrong!!
The Amperion system uses RF signals injected on the power conductors for the 'last mile' to the Amperion unit on the conductor that then transmits it through the air as Wi-Fi the last 100 feet or so to the subscriber.
Some broadband over power line (BPL) systems use fiber for 'backhaul' from the injection point (often at a substation) to the utilities routers. For instance, City of Manassas Utilities is doing something like that using AFL equipment. The last mile on that system is BPL, however.
("Last mile" refers to the run from the narest aggregation point to the subscriber -- it can actually be more or less than a mile)
There have actually been a number of these fiber in sewer system deployments, usually in urban areas where digging up streets is very expensive (>$500,000 in Manhattan). Scottish Water has a project underway now.
New Orleans is considering a variation on the idea using an innovative "burst pipe" system.
1. You need the power company's permission to attach your injectors and repeaters to their medium voltage (typically between 7,000 and 35,000 volt) distribution lines.
2. You need special skills, people and equipment to install this gear without getting electrocuted. (The power companies already have the skills, gear and people -- installing this stuff is easy.)
3. in the U.S. here are a number of OSHA regulations to comply with to work anywhere near a power conductor.
There are cases where non-power utilities are offering BPL (broadband over power line) service, but it's only with the cooperation of the local utility:
Broadband over power line (BPL) systems are economical for rural towns where there may be 40 or 50 homes per mile of power line.
Once you really get into the cuntryside, where line densities are typically 10 homes per mile (or less), these systems are not very economical because of all the repeaters needed. BPL signals attenuate pretty rapidly as you increase distances.
The Amperion unit on the pole uses Wi-Fi through the air for the connection from the pole to the house. All the subscriber needs is a Wi-Fi unit of his own.
There are other power line broadband systems from other vendors that use a special proprietary modem that plugs into the 120v outlet in the customer home and has an Ethernet output.
"When they install power lines the sometimes include an optical fibre cable inside the cores sheathing, so your have say 3 huge copper conducting cores and a skinny little optical cable as well, all wrapped up by a protective PVC sheath etc.. It doesn't cost a lot extra as its installed and manufactured at the same time as the power cable. Its this otical ring theyre tapping into with their wireless network."
The above is incorrect.
We specialize in fiber cable systems for power utilities. (See the Fiber Planners web site for more info on what we do)
Power utilities build fiber into their conductors in 3 situations:
1. They use optical groundwire (OPGW) on high voltage transmission lines between cities. This is an aluminum conductor with fibers in it that is placed above the power conductors and used as combination lightning guard and communications cable. This is widely deployed.
2. On the latest high voltage underground cables, they may use one fiber as a temperature sensor. These cables are not widely deployed. There are real issues associated with adding anymore fibers to that kind of cable for communications -- it's cheaper to just bury a separate fiber-only cable nearby, unless you're deploying an undesea cable, which leads to #3.
3. A few undersea power cables (such as might feed an offshore island) may include fibers for communications.
Most fiber cable deployed by power utilities is all-dielectric (contains nothing conductive) and hung or buried near the conductors on medium voltage power distribution systems.
The Amperion system in Sault Ste. Marie uses HF radio signals propagated down PUC's standard metallic power conductors to Wi-Fi units outside subscribers' homes. The Wi-Fi unit then takes that HF signal and retransmits a Wi-Fi signal through the air the last 100 feet or so to the subscriber.
The frequencies on used all of the commercially deployed powerline broadband systems are in the 2 to 80 MHz range. The Amperion system uses these lower frequencies (often confusingly referred to as HF or high frequency) to distribute the signals across the last mile --except for the last 100 feet or so. It then uses Wi-Fi to make the last connection. The Amperion unit hangs on the medium voltage power line, getting its' power from the same line. This unit converts the HF, last mile power line signal to a Wi Fi signal for transmission to the house nearby.
The only signal deliberately injected on to the power line is the HF signal which is much, much lower than a 2.4 GHz. Wi-Fi signal.
Even using so-called HF (high frequency) signals (3 to 30 MHz.) on power lines is tricky -- that's one reason we didn't see this technology 10 or 20 years ago. The signals attenuate rapidly and need regeneration every several hundred or thousand meters. To the extent that the power line picks up some of the Wi-Fi signal, attenuation is much higher at 2.4 GHz.
Most radio frequency concerns associated with these systems focus on possible interference to military and amateur radio operations in the HF range, not other 2.4 GHz. devices.
Con Ed's fiber network is huge. (Many of the pictures of high voltage fiber systems on my employer's web site are of Con Ed lines in Westchester County). Con Ed serves 1000s of businesses in the NYC area with high speed fiber services.
Con Ed Communications' "What's Happening" web page gives a sense of the scope of their business
Con Ed also has a small BPL (broadband over power line) trial in northern Westchester County.
There are three powerline communications applications in use:
1. Broadband over power line systems described here are all last-mile access systems for use on medium voltage (approx 1 kV to 35 kV) and low voltage (under 1 kV). These are for linking internet users to an ISP (either the power utility or someone partnering with the utility). These are broadband speed systems.
2. In home power line broadband for linking computers and other devices within the home over short distance. These are all low voltage, broadband speed systems. The HomePlug specification was developed by manufacturers in cooperation with the ARRL and other HF spectrum users.
3. Traditional narrowband power line communications systems used for power systems for several decades for remote meter reading and relaying. These operate on many different voltages but at low speeds. (You can read 1000s of meters per hour using 4800 baud speeds.) These systems are not at issue.
Adelaide, Australia is already using its' street light infrastructure to support a municipal wireless network ("citilan") in the central business district:
An added benefit is a virtually guaranteed job after college and very useful experience.
I went to Officer Candidate School (OCS) after college so I didn't have the benefit of the military paying my way.
After the Navy, I went back to grad school. After grad school, I found corporate recruiters really liked ex-military officers with technical backgrounds. Corporate recruiters were more interested in my ex-military classmates and I than the rest of the class.
Given the esteem in which our military is held by the public these days, I would expect this to be even more true.
It also helps to serve in an elite unit (submarines, aircraft, commandos).
Design Your Own Home(R) 3D WalkAround(TM)
http://www.abracadata.com/dyoh3dwalkaround-macinto sh.html
It's by the same people, Abracadata, that did the older program you mentioned.
We used it a year or two ago running system 9.x, so I would expect it would work just fine with OS X in Classic.
I think it's just what you're looking for.
Slashdot Mirror
The software bug was just one piece of a much bigger problem; I wouldn't want to overstate its' role. There were many other factors; here are just a few:
Poor vegetation management probably played an even bigger role as overloaded power lines warmed up, expanded and sagged into trees and bushes that were supposed to have been cut back.
Poor communications between utilities played a major role.
This whole section of the transmission system was known to be unstable.
An inadequate regulatory structure lacked teeth to deal with known problems.
Lack of adequate transmission line capacity
If all these other problems hadn't been in place, the software bug might never have surfaced. And certainly, the rpoblems would have been contained within a much smaller area -- maybe just First Energy's service area.
An article featured on Slashdot last year lays out the underlying complexity of the power grid very well: "The World's Largest Machine"
I think that the trial lawyers will take care of this issue for us. People like to fuss about trial lawyers and there certainly are abuses, but in this case, I expect that First Energy will get burned pretty bad in court.
There's a school of thought that says that whether it's by intention or default (probably default), in the U.S. we have trial lawyers and the court system filling a role played by government regulators in Europe. Even with the abuses of the legal system, some folks believe that we're better off with more lawsuits and fewer regulations. The reason given is that regulations don't always have flexibility to adapt to new situations quickly where as tort law + punitive damages does.
I'm really not a fan of trial lawyers, but I see how they play a useful role.
Slashdot had a great story on the blackout last year:
Guinnessy writes "The latest issues of the Industrial Physicist suggests that 'the vast system of electricity generation, transmission, and distribution that covers the United States and Canada is essentially a single machine -- by many measures, the world's biggest machine.' The article says that because deregulation ignored the physics of the machine, we have blackouts, a fact the industry warned regulators about in 1998. It has some nice hard science data for those interested in why we're going to get some more blackouts in the future unless Congress gets its act together." I work with power utilities -- this is the best single explanation I've seen of the underlying problems of transmission management and regulation in the U.S.
In WW II, several U.S submarines are believed to have torpedoed themselves. One known to have had this happen was the Tang (there were some survivors).
The torepedoes were defective.
Modern torpedoes have interlocks to prevent circular runs; if the torpedo turns through too large an angle (say 300 degrees), the torpedo either shuts down or at least dis-arms itself.
The "killer app" for many FTTH projects is -- get this -- responsive, locally-based, reliable service.
U.S. municipal power utilities are currently building FTTH networks to serve 100,000s of customers.
Most of these are built in small towns that have endured wretched service from their incumbent telephone and cable TV incumbents. Local residents want an alternative and turn to local government.
For a decade, small towns have successfully built and operated cable TV systems using HFC (hybrid fiber coax) technology.
By about a year ago, FTTH costs had dropped low enough to make it actually cheaper for a power utility to run ADSS fiber cable than coax. So these FTTH projects are just an extension of a trend that's been going on for years.
"The current technologies are still pretty much limited at 40Gb/s for one single fiber."
That's true, so then you deploy DWDM (dense wavelength division multiplexing) to multiplex 50 or 100 (or more) wavelengths of light, each carrying 10 or 40 Gb/s in traffic.
Add to that all the dark (unused) fiber deployed in long haul terrestrial networks in the U.S. and we have a lot of backbone fiber capacity. Typical fiber counts on the long-haul cables deployed in the late 1990s were 144 to 288 fibers or more.
Our company has designed fiber cable systems for about 60 power utilities, most of them municipally owned.
Here's a page with links to most of them.
There are many more North American cities with fiber systems we didn't design. The weblog Community Broadband Networks has links to a number of them. The weblog also has a summary page with about 1800 article links you can skim. About half cover municipal broadband projects of some sort.
In Utah, incumbent monopoly telco Qwest's modest investment of $50,000 in campaign contributions and its' powerful lobbyists (one is the son-in-law of the State Senate President) may be enough to kill the UTOPIA 18-city initiative to build a publicly-owned FTTH (fiber to the home) system. A bill (openly crafted by Qwest) that would effectively outlaw city's financing the project sailed out of the Senate and threatens to become law. This action comes after 18 city councils have voted to join UTOPIA and 6 have already made financial guarantees . The UTOPIA system is based on an open-access model allowing multiple competing providers to offer voice, data and video services to subscribers.
m unityfiber_archive.html#107630357108945975
This comes as the Salt Lake City Tribune, a strong foe of the UTOPIA initiative, ran an article wondering why Utah is losing its' position as a major technology center.
There are more UTOPIA links at http://communityfiber.blogspot.com/2004_02_08_com
There are many confused and mis-informed comments on this subject here. If someone's really interested, they can take about 15 minutes to read the actual text of the FCC's notice of proposed rule making (NPRM).
I've got more background on my blog, which cover BPL, FTTH and wireless broadband news. (You can also search the archives using the built-in search function).
Finally, the Virginia Journal of Law and Technology had a draft article on the technology and legal issues that was posted on the FCC's web site a month or two ago.
Reports of BPL's demise in Europe are premature. The European Union is spending many millions of dollars on developing and promoting a continent-wide BPL standard:
"European Union sponsors 20 million euro BPL (broadband over power line) project"
The FCC's notice specifically notes that the Japanese government in January opened up to BPL trials: http://www.soumu.go.jp/s-news/2004/040121_1.html That document is in Japanese; you can get an approximate Babelfish translation at: http://babelfish.altavista.com/babelfish/tr
BPL (broadband over power lines) is not economical in truly rural areas where the power utility's "line density" is typically 10 customers per mile or less. The system's attenuation requires to many repeaters for the number of customers.
Unlike Canada, where many small towns are getting DSL, in the U.S., many small rural towns have been written off by the big Bells and there is no broadband available. Even in a really small town, line densities are typically 50 homes per mile, which is ideal, economically for BPL.
The majority of folks in the U.S. that live in what are considered rural areas actually do live in a small community, not on a farm.
Even in big towns, one Amperion wireless unit does not serve that many homes.
"They are not transmitting shit over powerlines. They are transmitting over a fiber optic network which they installed with the power lines, then using 802.11b for the "last mile" to the consumer homes.This is not the same thing as the internet over power line debate, with all of the shielding and signal issues."
Wrong!!
The Amperion system uses RF signals injected on the power conductors for the 'last mile' to the Amperion unit on the conductor that then transmits it through the air as Wi-Fi the last 100 feet or so to the subscriber.
Some broadband over power line (BPL) systems use fiber for 'backhaul' from the injection point (often at a substation) to the utilities routers. For instance, City of Manassas Utilities is doing something like that using AFL equipment. The last mile on that system is BPL, however.
("Last mile" refers to the run from the narest aggregation point to the subscriber -- it can actually be more or less than a mile)
There have actually been a number of these fiber in sewer system deployments, usually in urban areas where digging up streets is very expensive (>$500,000 in Manhattan). Scottish Water has a project underway now.
New Orleans is considering a variation on the idea using an innovative "burst pipe" system.
1. You need the power company's permission to attach your injectors and repeaters to their medium voltage (typically between 7,000 and 35,000 volt) distribution lines.
2. You need special skills, people and equipment to install this gear without getting electrocuted. (The power companies already have the skills, gear and people -- installing this stuff is easy.)
3. in the U.S. here are a number of OSHA regulations to comply with to work anywhere near a power conductor.
There are cases where non-power utilities are offering BPL (broadband over power line) service, but it's only with the cooperation of the local utility:
Penn Yan Municipal Utilities Board/DVI
City of Manassas Utilities/Prospect Street Broadband
Broadband over power line (BPL) systems are economical for rural towns where there may be 40 or 50 homes per mile of power line.
Once you really get into the cuntryside, where line densities are typically 10 homes per mile (or less), these systems are not very economical because of all the repeaters needed. BPL signals attenuate pretty rapidly as you increase distances.
The Amperion unit on the pole uses Wi-Fi through the air for the connection from the pole to the house. All the subscriber needs is a Wi-Fi unit of his own.
There are other power line broadband systems from other vendors that use a special proprietary modem that plugs into the 120v outlet in the customer home and has an Ethernet output.
"When they install power lines the sometimes include an optical fibre cable inside the cores sheathing, so your have say 3 huge copper conducting cores and a skinny little optical cable as well, all wrapped up by a protective PVC sheath etc.. It doesn't cost a lot extra as its installed and manufactured at the same time as the power cable. Its this otical ring theyre tapping into with their wireless network."
The above is incorrect. We specialize in fiber cable systems for power utilities. (See the Fiber Planners web site for more info on what we do)
Power utilities build fiber into their conductors in 3 situations:
1. They use optical groundwire (OPGW) on high voltage transmission lines between cities. This is an aluminum conductor with fibers in it that is placed above the power conductors and used as combination lightning guard and communications cable. This is widely deployed.
2. On the latest high voltage underground cables, they may use one fiber as a temperature sensor. These cables are not widely deployed. There are real issues associated with adding anymore fibers to that kind of cable for communications -- it's cheaper to just bury a separate fiber-only cable nearby, unless you're deploying an undesea cable, which leads to #3.
3. A few undersea power cables (such as might feed an offshore island) may include fibers for communications.
Most fiber cable deployed by power utilities is all-dielectric (contains nothing conductive) and hung or buried near the conductors on medium voltage power distribution systems.
The Amperion system in Sault Ste. Marie uses HF radio signals propagated down PUC's standard metallic power conductors to Wi-Fi units outside subscribers' homes. The Wi-Fi unit then takes that HF signal and retransmits a Wi-Fi signal through the air the last 100 feet or so to the subscriber.
The frequencies on used all of the commercially deployed powerline broadband systems are in the 2 to 80 MHz range. The Amperion system uses these lower frequencies (often confusingly referred to as HF or high frequency) to distribute the signals across the last mile --except for the last 100 feet or so. It then uses Wi-Fi to make the last connection. The Amperion unit hangs on the medium voltage power line, getting its' power from the same line. This unit converts the HF, last mile power line signal to a Wi Fi signal for transmission to the house nearby.
The only signal deliberately injected on to the power line is the HF signal which is much, much lower than a 2.4 GHz. Wi-Fi signal.
Even using so-called HF (high frequency) signals (3 to 30 MHz.) on power lines is tricky -- that's one reason we didn't see this technology 10 or 20 years ago. The signals attenuate rapidly and need regeneration every several hundred or thousand meters. To the extent that the power line picks up some of the Wi-Fi signal, attenuation is much higher at 2.4 GHz.
Most radio frequency concerns associated with these systems focus on possible interference to military and amateur radio operations in the HF range, not other 2.4 GHz. devices.
Con Ed's fiber network is huge. (Many of the pictures of high voltage fiber systems on my employer's web site are of Con Ed lines in Westchester County). Con Ed serves 1000s of businesses in the NYC area with high speed fiber services.
Con Ed Communications' "What's Happening" web page gives a sense of the scope of their business
Con Ed also has a small BPL (broadband over power line) trial in northern Westchester County.
A.B.
There are three powerline communications applications in use:
1. Broadband over power line systems described here are all last-mile access systems for use on medium voltage (approx 1 kV to 35 kV) and low voltage (under 1 kV). These are for linking internet users to an ISP (either the power utility or someone partnering with the utility). These are broadband speed systems.
2. In home power line broadband for linking computers and other devices within the home over short distance. These are all low voltage, broadband speed systems. The HomePlug specification was developed by manufacturers in cooperation with the ARRL and other HF spectrum users.
3. Traditional narrowband power line communications systems used for power systems for several decades for remote meter reading and relaying. These operate on many different voltages but at low speeds. (You can read 1000s of meters per hour using 4800 baud speeds.) These systems are not at issue.
Community Broadband Networks blog tracks powerline broadband news and issues (also fiber to the home):
Community Broadband Networks
So far there have been about 100 power line broadband posts over the last year; you can use the built-in search function to find them. A.B.
Adelaide, Australia is already using its' street light infrastructure to support a municipal wireless network ("citilan") in the central business district:
Community Broadband Networks:
"City of Adelaide to offer wireless broadband downtown"
MuniWireless.com:
"Adelaide hotzone is up and running"
An added benefit is a virtually guaranteed job after college and very useful experience.
I went to Officer Candidate School (OCS) after college so I didn't have the benefit of the military paying my way.
After the Navy, I went back to grad school. After grad school, I found corporate recruiters really liked ex-military officers with technical backgrounds. Corporate recruiters were more interested in my ex-military classmates and I than the rest of the class.
Given the esteem in which our military is held by the public these days, I would expect this to be even more true.
It also helps to serve in an elite unit (submarines, aircraft, commandos).
Design Your Own Home(R) 3D WalkAround(TM) http://www.abracadata.com/dyoh3dwalkaround-macinto sh.html
It's by the same people, Abracadata, that did the older program you mentioned.
We used it a year or two ago running system 9.x, so I would expect it would work just fine with OS X in Classic.
I think it's just what you're looking for.