What's Frying the Electrical Systems On BART Trains? (ieee.org)

Tekla Perry writes: Earlier this month, BART engineers shut down a substation in hopes that the closure would quiet the power surges that were frying the electrical propulsion equipment on BART cars -- a peak of 40 in just one day in February. The shutdown seemed to solve the problem, but BART officials weren't sure they'd really found the answer. Yesterday, the power surges popped up again, on an entirely different section of tracks, damaging 50 cars before BART closed off that section, rerouting passengers onto buses. Track inspections yesterday revealed nothing, and BART reports that it has reached out to experts around the country and asked them to fly in and help solve the mystery. Do you have a theory? Note: BART is the 5th-busiest heavy-rail rapid transit system in the U.S.

Comments

Hackers ruining our infrastructure

[fragMasterFlash]

Which nation-state is sponsoring the hacking crew that will inevitably be blamed for this issue?

Re:Hackers ruining our infrastructure

[frovingslosh]

Of course it's the Chinese, but we will never know for sure until we make out computers even less secure by forcing Apple to build in back doors.

I-squared-L?

[swschrad]

could there perhaps be enough inductance in the multi-motor systems that it is generating its own connect/disconnect/connect surges? try isolating those DC motor controllers from both the track and the motors with some diode stacks and snubber caps.

Re:I-squared-L?

[Archfeld]

Unless they have recently upgraded a mass of cars or track, why does it suddenly start happening now ? The system has been in use and fairly reliable for a long time. I have no proof, but the suspicious borderline paranoid inside me is screaming that someone is hacking at the electrical infrastructure that feeds the Bart system, and the problem lies outside their direct observation, and is likely with PG&E's supply system to Bart. PG&E has demonstrated the disregard for maintenance, monitoring, and the security incompetence in the last few years to allow for something like this. It will likely take some outside support for Bart to prove the failures don't lie inside their infrastructure to get PG&E to even begin to look at their own systems.

Re: I-squared-L?

[hey!]

I had a boss once who hired a consultant and was angry when the consultant told him to do the obvious thing.

"You know what a consultant is?" he groused. "Someone who borrows your watch to tell you the time."

I thought about this for a moment. "Yeah, but what's he supposed to do if you're standing there with the watch on your wrist and you don't know what time it is?"

You can't defer maintenance forever

[hawguy]

BART already tweeted the reason behind the breakdowns:

From @SFBART:
BART was built to transport far fewer people, and much of our system has reached the end of its useful life. This is our reality.

BART has been continually expanding while deferring maintenance on the rest of the system, and that policy has finally come home to roost -- much of their infrastructure is over 40 years old and they can't defer maintenance forever. But by continually expanding, they've made themselves too big to fail (and they've gotten more counties on the hook to keep the service running), so they'll get bailed out one way or another.

Re:You can't defer maintenance forever

[bill_mcgonigle]

BART already tweeted the reason [nytimes.com] behind the breakdowns:

That's not a reason - it's a complaint (and a beg for more money). Maybe they do need more money, but if the cause of the surges remains a mystery, then by definition they don't know the reason.

I'm sure they don't have any SCADA systems exposed to the Internet (right?), but "old stuff" is just a guess. And there are some good "old stuff" guesses elsewhere on this story.

Re:You can't defer maintenance forever

[DarkSabreLord]

It's not just that - BART was simply never meant to be operating on the scale it does today. When BART was built, the creators envisioned a system that would serve about 100,000 people per week and choke points such as the Transbay Tube were built accordingly. Naturally, as the population increased, upgrades had to made. This worked for a while, but eventually lack of funding for serious overhauls caught up with the
constantly increasing ridership. Maximum capacity is heavily influenced by the fact that sections like the tube are single line, with no easy way to expand to double or triple. BART could theoretically be a 24/7 system, but as things stand now their engineers need every minute of the nightly downtime they have to service a rapidly aging rail system.

The rails already in place are almost at capacity, with a train crossing over them every 2 minutes. With the tech booms of the last decades, there's been an even bigger spike in these numbers. Over the last decade alone, passenger alightings at some stations have more than doubled. On busy days, the BART system now serves 25 times more riders than originally envisioned. There's some money for additional trains, but that can only do so much. Eventually, we are going to need to spend money on either more parallel tracks, cars, and bigger platforms or just a new system altogether.

Their administrators are simply being realistic about the situation we're in

I suspect ...

[PPH]

... some sort of interaction between filter/power factor capacitors in the new substation and the inductive reactance of the tracks and distribution system. Transients caused by the inherent imperfect third rail to car contacts causes a ringing (oscillation) in the system which, with the new substation on line, happens to be on or near a frequency that some of the rolling stock motor controllers don't like.

Throw some power quality analyzers on various sections of the track and watch the system's transient voltage response with power sources in various configurations.

Re:I suspect ...

[93 Escort Wagon]

Transients caused by the inherent imperfect third rail to car contacts causes a ringing (oscillation) in the system ...

Oh, sure - blame the homeless.

vitrification of grounds and more rain

[Anonymous Coward]

So the A & B cars having inductions motors seem to be fine, while the problem seems to be confined to the C cars having the DC motor. That's one difference.

Also, what else has changed? Take a look at wunderground to see that the Bay Area is having a wet season.

Why would the C cars have been mostly fine all along and having trouble now?

So there's charge building up in the DC motors that they can't handle and that makes them blow out. The charge has nowhere to go. What controls the flow of charge? Grounding. What can go wrong with grounding? Good grounds can go bad when a lot of discharge causes the sand in the soil to vitrify (melt into glass) after discharges and lightning strikes have been shooting through it for decades. Better grounds can unexpectedly form when more highly conductive paths form up. The AC induction motors will suffer a power loss but can handle the charge jumping back & forth in unexpected ways, while the DC motors can't.

Add it all up. This has to be a grounding problem aggravated by the wet season, and an underlying assumption that once you sink a ground it's good forever. It isn't.

Real Time Monitoring

[BoRegardless]

I'm truly surprised that they don't have intensive real time monitoring with sensors through their whole system.

Proper engineering and maintenance of such a critical system demands it.

Re:Real Time Monitoring

[Hans Lehmann]

Monitoring of what? Sensors through the whole system that measure what? What exactly is this proper engineering that you claim they should have done? Armchair engineers always seem to have perfect hindsight.

Eels on a train

[sjames]

It all started 2 years ago when a student majoring in EE took an exotic canoe trip on the Amazon. One day the canoe capsized while he was studying and his book sank to the bottom. Thee eels read voraciously and learned about series parallel wiring of batteries. An idea was born.

And so now we have Electric Eels on a Train!

Re:Solar flares

[kenwd0elq]

My "go to" excuse for any electronic problems is "sunspots, or stray cosmic rays". However.....

The Sun has been very quiet for the last several weeks, and Solar Cycle 24 is on a steep downward trend. I expect that we'll begin an extended Solar Minimum by the end of 2016 which may last 3-4 years (the "average" is 2 years, and the last Solar Min was nearly 3). I also expect that the next few solar cycles will be fairly quiet. Perhaps not Maunder Minimum quiet, but probably Dalton Minimum quiet, or nearly so.

You should visit www.spaceweather.com periodically to keep up to date on this.

Also; it's going to get somewhat chilly by and by.

Re:Who stands to benefit?

[Z00L00K]

Considering that it's aged - don't rule out aging wire insulation that no longer does its job.

Re:How is BART supposed to update trains...

[Princeofcups]

...when all their money is going to high salaries and benefits for union employees?

Over 200 BART employees earned over $200,000 a year in total compensation...

Those look mostly like executives, which are definitely NOT Union positions. The Unions are the ones trying to get some of that executive salary down to the real workers.