Slashdot Mirror


What's Frying the Electrical Systems On BART Trains? (ieee.org)

Tekla Perry writes: Earlier this month, BART engineers shut down a substation in hopes that the closure would quiet the power surges that were frying the electrical propulsion equipment on BART cars -- a peak of 40 in just one day in February. The shutdown seemed to solve the problem, but BART officials weren't sure they'd really found the answer. Yesterday, the power surges popped up again, on an entirely different section of tracks, damaging 50 cars before BART closed off that section, rerouting passengers onto buses. Track inspections yesterday revealed nothing, and BART reports that it has reached out to experts around the country and asked them to fly in and help solve the mystery. Do you have a theory? Note: BART is the 5th-busiest heavy-rail rapid transit system in the U.S.

40 of 250 comments (clear)

  1. Hackers ruining our infrastructure by fragMasterFlash · · Score: 4, Insightful

    Which nation-state is sponsoring the hacking crew that will inevitably be blamed for this issue?

    1. Re:Hackers ruining our infrastructure by flopsquad · · Score: 3, Funny

      Which nation-state is sponsoring the hacking crew that will inevitably be blamed for this issue?

      Well dammit we'd know already if they hadn't gone dark with all the unbreakable encryptions on their iPhones!

      --
      Nothing posted to /. has ever been legal advice, including this.
    2. Re:Hackers ruining our infrastructure by frovingslosh · · Score: 4, Funny

      Of course it's the Chinese, but we will never know for sure until we make out computers even less secure by forcing Apple to build in back doors.

      --
      I'm an American. I love this country and the freedoms that we used to have.
  2. I-squared-L? by swschrad · · Score: 4, Interesting

    could there perhaps be enough inductance in the multi-motor systems that it is generating its own connect/disconnect/connect surges? try isolating those DC motor controllers from both the track and the motors with some diode stacks and snubber caps.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?
    1. Re: I-squared-L? by WarJolt · · Score: 2

      Oh come on. It couldn't possibly be that simple. I write software for a living and even I know to look for back emf first.

    2. Re:I-squared-L? by Archfeld · · Score: 5, Insightful

      Unless they have recently upgraded a mass of cars or track, why does it suddenly start happening now ? The system has been in use and fairly reliable for a long time. I have no proof, but the suspicious borderline paranoid inside me is screaming that someone is hacking at the electrical infrastructure that feeds the Bart system, and the problem lies outside their direct observation, and is likely with PG&E's supply system to Bart. PG&E has demonstrated the disregard for maintenance, monitoring, and the security incompetence in the last few years to allow for something like this. It will likely take some outside support for Bart to prove the failures don't lie inside their infrastructure to get PG&E to even begin to look at their own systems.

      --
      errr....umm...*whooosh* *whoosh* Is this thing on ?
    3. Re:I-squared-L? by FirstOne · · Score: 2

      Couple that with time and mechanical wear.. The bionding between rail segments is wearing out, (and/or being stolen for scrap metal value.) A few intermittant rail to rail connections based on rail movement, (as when a fully loaded train passes over), would inflict a world of hurt on the 1000volt DC electronics.

    4. Re: I-squared-L? by hey! · · Score: 5, Funny

      I had a boss once who hired a consultant and was angry when the consultant told him to do the obvious thing.

      "You know what a consultant is?" he groused. "Someone who borrows your watch to tell you the time."

      I thought about this for a moment. "Yeah, but what's he supposed to do if you're standing there with the watch on your wrist and you don't know what time it is?"

      --
      Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
    5. Re:I-squared-L? by BronsCon · · Score: 3, Interesting

      On one hand, if the (looks like 000 gauge) conductors connecting the rails are being stolen, that could post a problem. On the other hand, since there are 4 paddles on each side of each car (as the 3rd rail can be on either side, depending on which side the platform is on at the next station), each car maintains contact with both rails as it transitions from one to the next. Likewise for the wheels, making the ground connection for each car. Coupled with the fact that cars share power, a 10 car train will be in contact with 4 or 5 rail pairs at any given moment, a majority of the bonding conductors would have to be missing or severely damaged to cause even a minor power issue. Cars are sized such that when the front of the train is crossing a rail threshold, the rear of the train is in the middle of a rail, so good contact is guaranteed at all times unless the first or last car is severely damaged and missing at least two paddles on the 3rd rail side. Even then, the likelihood of damage is minuscule.

      It is almost certainly a supply problem.

      --
      APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
    6. Re: I-squared-L? by geoskd · · Score: 3, Informative

      Oh come on. It couldn't possibly be that simple. I write software for a living and even I know to look for back emf first.

      True, but as the article mentioned, the system was designed for 1/20th of the current passenger load. I suspect that the problem actually lies in the way these cars are slowed. The obvious (intelligent) way to do it would be to induce back emf into the supply, and put some of the energy back into the system in the form of increased mains voltage. In small amounts (one car at a time), this would be a relative non-issue. In larger amounts, an especially unlucky synchronization of cars all slowing simultaneously could, in theory, overload the system and cause massive transients.

      The articles also mention that the cars have both DC and AC motors, but only the DC motors are getting cooked. This leads me to expect that the problem has been building up for a while, but has been below the threshold of damage to the motors until recently. The fact that it is the thyristors (used to rectify the AC power for DC use) which are failing, tells me that both the AC motors and the DC motors have been receiving severe overloads, but the Thyristor was simply the weakest component and has been failing first.

      If it turns out that the back emf is the culprit, the solutions are not simple. In effect, they need to find somewhere for that energy to go other than back into the supply network. Any option they go with is either going to A) significantly reduce the efficiency of the system or B) require additional expensive hardware be installed onto every car.

      --
      I wish I had a good sig, but all the good ones are copyrighted
  3. You can't defer maintenance forever by hawguy · · Score: 5, Insightful

    BART already tweeted the reason behind the breakdowns:

    From @SFBART:
    BART was built to transport far fewer people, and much of our system has reached the end of its useful life. This is our reality.

    BART has been continually expanding while deferring maintenance on the rest of the system, and that policy has finally come home to roost -- much of their infrastructure is over 40 years old and they can't defer maintenance forever. But by continually expanding, they've made themselves too big to fail (and they've gotten more counties on the hook to keep the service running), so they'll get bailed out one way or another.

    1. Re: You can't defer maintenance forever by Anonymous Coward · · Score: 2, Informative

      Yes, just think if actual personal automobile costrs were directly born by the users, they would run screaming.

      Instead it gets widely subsidized in invisible ways.

    2. Re:You can't defer maintenance forever by bill_mcgonigle · · Score: 4, Insightful

      BART already tweeted the reason [nytimes.com] behind the breakdowns:

      That's not a reason - it's a complaint (and a beg for more money). Maybe they do need more money, but if the cause of the surges remains a mystery, then by definition they don't know the reason.

      I'm sure they don't have any SCADA systems exposed to the Internet (right?), but "old stuff" is just a guess. And there are some good "old stuff" guesses elsewhere on this story.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
    3. Re:You can't defer maintenance forever by reve_etrange · · Score: 2

      BART is about equally priced to driving for a single person, and significantly more expensive for multiple people (i.e. versus carpool). People use it because, despite it's various frustrations, it's genuinely more convenient than driving, not because it's less expensive.

      --
      .: Semper Absurda :.
    4. Re:You can't defer maintenance forever by rossz · · Score: 2

      When I could BART to work I did so because I could sit back and read instead of gritting my teeth in traffic. The trip time was about the same, but it was no longer wasted time.

      --
      -- Will program for bandwidth
    5. Re:You can't defer maintenance forever by Princeofcups · · Score: 2

      BART is about equally priced to driving for a single person, and significantly more expensive for multiple people (i.e. versus carpool). People use it because, despite it's various frustrations, it's genuinely more convenient than driving, not because it's less expensive.

      BART is the only way to get across the Bay during rush hour without waiting at least 2 hours to drive across the bridge. That's NOT an exaggeration.

      --
      The only thing worse than a Democrat is a Republican.
    6. Re:You can't defer maintenance forever by DarkSabreLord · · Score: 4, Informative

      It's not just that - BART was simply never meant to be operating on the scale it does today. When BART was built, the creators envisioned a system that would serve about 100,000 people per week and choke points such as the Transbay Tube were built accordingly. Naturally, as the population increased, upgrades had to made. This worked for a while, but eventually lack of funding for serious overhauls caught up with the
      constantly increasing ridership. Maximum capacity is heavily influenced by the fact that sections like the tube are single line, with no easy way to expand to double or triple. BART could theoretically be a 24/7 system, but as things stand now their engineers need every minute of the nightly downtime they have to service a rapidly aging rail system.

      The rails already in place are almost at capacity, with a train crossing over them every 2 minutes. With the tech booms of the last decades, there's been an even bigger spike in these numbers. Over the last decade alone, passenger alightings at some stations have more than doubled. On busy days, the BART system now serves 25 times more riders than originally envisioned. There's some money for additional trains, but that can only do so much. Eventually, we are going to need to spend money on either more parallel tracks, cars, and bigger platforms or just a new system altogether.

      Their administrators are simply being realistic about the situation we're in

    7. Re: You can't defer maintenance forever by Anonymous Coward · · Score: 3, Interesting

      Conservatives hate talk about externalities. Their so called free market relies on you not noticing, and God forbid holding them accountable for, pollution, congestion, oil subsidies and tax breaks, a huge military devoted to guarding overseas private assets (oil wells and shipping lanes), poisoning of water supplies, seizure of land for private gain, mandatory insurance at predatory prices (which is somehow not ok with healthcare, except for the predatory pricing part), inefficient use of resources (automobiles in daily service won't last 40 years), and the list goes on and on. The ones who demand you pay and pay and pay for every last thing never actually want to pay their own way, but they sure like to make you think they do.

      But they'll complain mightily about public transport being subsidized, particularly when it involves things like rail systems, and especially when it's because rail systems tend to cause good externalities like reduced traffic congestion, better energy efficiency, etc. After further review, busses still suck though.

      I love my car, but I live in the South where they make sure public transport is slow, inconvenient, and only for poor and minority people. It's kind of alien to me when I visit a place where people from all walks of life use it. I've discovered I like not having to use a car for everything.

    8. Re:You can't defer maintenance forever by Anonymous Coward · · Score: 2, Informative

      "BART is about equally priced to driving for a single person, and significantly more expensive for multiple people (i.e. versus carpool)."

      If youre driving into the city, this is not true at all unless your employer has free parking. Most of the cost of diving into the city is paying for parking, which is anywhere from $30 - $80 a ~day~.

      The vast majority of BART ridership and hell-like experience is into the city in the morning and out again in the evening...if youre traveling from any other BART stop to any other, its not that bad at all.

    9. Re: You can't defer maintenance forever by Aaden42 · · Score: 2

      Nope. Just on a tangent.

  4. I suspect ... by PPH · · Score: 4, Interesting

    ... some sort of interaction between filter/power factor capacitors in the new substation and the inductive reactance of the tracks and distribution system. Transients caused by the inherent imperfect third rail to car contacts causes a ringing (oscillation) in the system which, with the new substation on line, happens to be on or near a frequency that some of the rolling stock motor controllers don't like.

    Throw some power quality analyzers on various sections of the track and watch the system's transient voltage response with power sources in various configurations.

    --
    Have gnu, will travel.
    1. Re:I suspect ... by 93+Escort+Wagon · · Score: 5, Funny

      Transients caused by the inherent imperfect third rail to car contacts causes a ringing (oscillation) in the system ...

      Oh, sure - blame the homeless.

      --
      #DeleteChrome
  5. vitrification of grounds and more rain by Anonymous Coward · · Score: 5, Interesting

    So the A & B cars having inductions motors seem to be fine, while the problem seems to be confined to the C cars having the DC motor. That's one difference.

    Also, what else has changed? Take a look at wunderground to see that the Bay Area is having a wet season.

    Why would the C cars have been mostly fine all along and having trouble now?

    So there's charge building up in the DC motors that they can't handle and that makes them blow out. The charge has nowhere to go. What controls the flow of charge? Grounding. What can go wrong with grounding? Good grounds can go bad when a lot of discharge causes the sand in the soil to vitrify (melt into glass) after discharges and lightning strikes have been shooting through it for decades. Better grounds can unexpectedly form when more highly conductive paths form up. The AC induction motors will suffer a power loss but can handle the charge jumping back & forth in unexpected ways, while the DC motors can't.

    Add it all up. This has to be a grounding problem aggravated by the wet season, and an underlying assumption that once you sink a ground it's good forever. It isn't.

    1. Re:vitrification of grounds and more rain by thegarbz · · Score: 3, Interesting

      Throw together something to actually log what is going on. Even something like this, with the right voltage divider plugged into a little rugged PC like this or similar would probably work and you could create the logging code with Labview.

      You're in university or work in research aren't you? In the rest of the world we just hire off the shelf logging equipment specifically designed for the task and move on.

    2. Re:vitrification of grounds and more rain by thegarbz · · Score: 3, Interesting

      This has to be a grounding problem aggravated by the wet season

      Grounding problems are the cause of problems in the rail cars for many different reasons. My favourite is when an Australian train company decided to go green and implement regenerative breaking on all their equipment without any forethought. After purchasing an entire fleet of fancy new green trains they found the power goes out at the train station every time the train approaches.

      But really this is not helped by the fact that ground forms an important part of the electrical path in a rail system, and not just a safety mechanism as in most other cases.

  6. control computers probably spygrading to Windws 10 by raymorris · · Score: 3, Funny

    Perhaps the newest computers controlling the system have forcibly "upgraded" themselves to Windows 10.

  7. Real Time Monitoring by BoRegardless · · Score: 5, Insightful

    I'm truly surprised that they don't have intensive real time monitoring with sensors through their whole system.

    Proper engineering and maintenance of such a critical system demands it.

    1. Re:Real Time Monitoring by Hans+Lehmann · · Score: 5, Insightful

      Monitoring of what? Sensors through the whole system that measure what? What exactly is this proper engineering that you claim they should have done? Armchair engineers always seem to have perfect hindsight.

      --
      09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
    2. Re:Real Time Monitoring by BigU+03C0mpin · · Score: 2

      If you're surprised that BART is lacking something remotely logical and/or basically requisite you don't understand BART or the non-Tech Industry side of the Bay Area.

      BART only funds repairs, no technical improvements what-so-ever. The most recent redesigns are ways to fit more standing people in each car since it raises rider count. In fact, the only technological changes I can think of having happened to BART in the 25 years I lived in the Bay Area was the upgrade to the Clipper Card system.

      Bathrooms in stations? All the underground ones, particularly in SF are closed permanently. Above ground stations outside of SF, if it's in service it might be open. It will be one of the most awful public bathrooms you've been in. I honestly have found nicer solar and pit toilets in the California back country than I've ever found in a BART station unless the station was a new expansion station.

      Anyone with half a brain can see that zero money has gone into BARTs technical budget in a long time. From my passenger perspective, it's been at least a quarter century.

    3. Re:Real Time Monitoring by thegarbz · · Score: 3, Interesting

      I'm truly surprised that they don't have intensive real time monitoring with sensors through their whole system.

      Outside of a power generation company looking at maybe one or two substations away from their plant you'll find power infrastructure monitoring to the level that could aid predictive maintenance is non-existent. Even in utilities it's non-existent. Even in companies that don't run their equipment into the ground it's non-existent.

    4. Re:Real Time Monitoring by hughbar · · Score: 2

      Actually, the Tube in London is ancient (I'm not going to say 150 years old, because it was steam at the start) and still running, with problems. It's also over-capacity, to the extent that the entrance barriers sometimes need to be closed at some stations. So BART is a child or an adolescent by comparison. Actually, maybe the problems are problems of adolescence!

      --
      On y va, qui mal y pense!
  8. Re:control computers probably spygrading to Windws by freeze128 · · Score: 3, Interesting

    I visited the Bay area in 1987, and I remember seeing the ticket sales machines having the big blue IBM logo on them. I bet the controllers run OS/2 warp.

  9. Eels on a train by sjames · · Score: 5, Funny

    It all started 2 years ago when a student majoring in EE took an exotic canoe trip on the Amazon. One day the canoe capsized while he was studying and his book sank to the bottom. Thee eels read voraciously and learned about series parallel wiring of batteries. An idea was born.

    And so now we have Electric Eels on a Train!

    1. Re:Eels on a train by khallow · · Score: 2

      You spilled it too early, so now there will be copycat movies hitching a ride on your blockbuster. Prepare for Lampreys on a Barge and Rugrats in a Car. Coming to theaters near you!

  10. perhaps someone is testing by MrKaos · · Score: 3, Interesting

    a cyber attack.

    --
    My ism, it's full of beliefs.
  11. Re:Solar flares by kenwd0elq · · Score: 4, Interesting

    My "go to" excuse for any electronic problems is "sunspots, or stray cosmic rays". However.....

    The Sun has been very quiet for the last several weeks, and Solar Cycle 24 is on a steep downward trend. I expect that we'll begin an extended Solar Minimum by the end of 2016 which may last 3-4 years (the "average" is 2 years, and the last Solar Min was nearly 3). I also expect that the next few solar cycles will be fairly quiet. Perhaps not Maunder Minimum quiet, but probably Dalton Minimum quiet, or nearly so.

    You should visit www.spaceweather.com periodically to keep up to date on this.

    Also; it's going to get somewhat chilly by and by.

  12. Re:Who stands to benefit? by Z00L00K · · Score: 4, Interesting

    Considering that it's aged - don't rule out aging wire insulation that no longer does its job.

    --
    If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
  13. Re:How is BART supposed to update trains... by Princeofcups · · Score: 5, Insightful

    ...when all their money is going to high salaries and benefits for union employees?

    Over 200 BART employees earned over $200,000 a year in total compensation...

    Those look mostly like executives, which are definitely NOT Union positions. The Unions are the ones trying to get some of that executive salary down to the real workers.

    --
    The only thing worse than a Democrat is a Republican.
  14. I've got a theory by maroberts · · Score: 2

    It must be bunnies, bunnies, it must be bunnies! /Buffy

    --

    Donte Alistair Anderson Roberts - hi son!
    Karma: Chameleon

  15. Re:North Korea? by eric_harris_76 · · Score: 2

    If these sorts of mystery problems go away after the North Korean regime collapses or is overthrown -- next year, next decade, whenever -- we'll know.

    Not that knowing will help us much now. Or then, for that matter.

    --
    There's no time like the present. Well, the past used to be.