Detroit's Emergency Dispatch System Fails

dstates writes "For most of Friday, police and firefighters in Detroit were forced to operate without their usual dispatch radio when the emergency dispatch system failed. The radio system used for communication between 911 dispatchers and Detroit's police, fire and EMS crews went down around 5:30 a.m. Friday morning, causing a backlog of hundreds of calls and putting public safety at risk. Michigan State Police allowed Detroit's emergency system to use the state's communication towers, but access was restricted to top priority calls out of fear of overloading the State system. More than 60 priority-1 calls and more than 170 non-emergency calls were backed up. With no dispatch to communicate if something went wrong and backup was needed, police were forced to send officers out in pairs for safety concerns on priority-1 calls. Detroit's new police chief, James Craig, says he's 'appalled' that a redundant system did not kick in. The outage occurred only days after Craig took office. The $131 million Motorola system was installed in 2005 amid controversy over its funding. Spokesmen for Motorola said parts of the system were regularly maintained but acknowledged that backup systems had not been tested in the past two years. They said the problem was a hardware glitch in the link between dispatch and the individual radios. As of 9 p.m. Friday, a Motorola spokesman said the system was stable and the company would continue troubleshooting next week."

Comments

well

[Anonymous Coward]

Knowing their police force, the 911 outage might have saved some lives.

Duhhhh....

[Mr. Dop]

...its Detroit! Michigan is the only reason why both California and Florida dont fall off into the Ocean. It sucks that much.

Expected

[girlintraining]

Detroit has had massive funding and infrastructure problems for some time now. It's a dying city with much of the suburbs either abandoned, being reclaimed by nature, and generally being both in appearance and substance as a 3rd world country. It's so bad it has gotten national attention -- an emergency financial planner was sent in to try to right their budget, with limited success.

You can't judge Detroit the same way as you could, say, Chicago. They're no longer really part of the first world. This wouldn't be news if it happened in Afghanistan, for example. It's a sad state of affairs, but this is the inevitable result of a slide into the third world... our bridges and other key infrastructure is also rotting. Detroit is just foreshadowing what will happen to many of our cities over the next 15-20 years as our economy continues to slide into the ocean of wealth inequity.

Re:Expected

[sjames]

The thing is though, all of TFAs indicate that the city had a valid contract with Motorola to maintain the system including routine testing. In spite of that, no testing happened. While your observations may have bearing in general, in this instance it seems like a well known vendor with a (perhaps undeserved in retrospect) good reputation is the source of the problem.

Re:Expected

[instagib]

Good analysis. The mentioned wealth inequity, the poverty, crime, lack of education, and corruption result in a vicious, almost unbreakable circle. It's like a real life experiment and illustrates why 3rd world countries mainly stay the way they are.

Re:Expected

[mjpollard]

If you're talking "suburbs" within the Detroit city limits, then yes, I agree with you. (I went by my grandma's old house in northern Detroit a while ago -- the 7 Mile/Gratiot/Hayes area, for the natives among us -- and "reclaimed by nature" doesn't begin to describe it. I nearly wept at the sight as the memories of my brother and I playing in the back yard when we were kids surfaced.) Most in the Metro Detroit area, however, know "suburbs" as the cities and towns outside the city limits, cities such as Grosse Pointe, Royal Oak, Southfield, Dearborn, etc., all of which are alive and thriving.

Re:Expected

[Anonymous Coward]

That's what happens when you build a colossal public sector and a wealth transfer system that punishes productivity and rewards doing nothing. Eventually the productive members of the society decide to pack their things and you're only left with the socialist bums complaining how their shitty life is the fault of banks and corporations. Trying to force income equity is a sure way to destroy even a prosperous economy.

Re:Expected

[ttucker]

Except again, from TFA, the city of Detroit was paying an enormous sum of money to a reputable vendor to maintain the system. How does that coalesce with this third world, wealth inequality theory?

Re:Expected

[DigiShaman]

Globalism. This is what happens when it's cheaper to move automotive manufacturing overseas only to be compounded further by the unions squeezing out all the profits and stonewalling the change that's necessary to survive. It's one giant death spiral that was enviable. We've had ghost towns in the past, there's no reason to think we wont have them again in the future. Hell, Midland TX (currently a boomtown) might be the next one should all the fracking stop via legislation.

Re:Expected

[Rockoon]

While Detroit hasnt had a Republican administration since January 1962, its still hard to conclusively blame the problems on the left.

We'll get a larger sample size soon enough.

Re:Expected

[Anonymous Coward]

different Motorolas -- it split up, the part Google got is the phone making one, but the radios one is Motorola Solutions

Re:Expected

[Anonymous Coward]

Shhh, it's all about wealth inequity. Everything is. Didn't you get the memo? It turns out that the secret to happiness in life is worrying more about other peoples' bank accounts than you do about your own.

Re:Expected

[theskipper]

I think there are subtle differences in interpretation when discussing wealth inequity.

Wealth inequity won't ever go away and in itself isn't bad. If you work harder than your neighbor then your net worth should eventually be greater. That's the American Way (tm).

The problem is when you reach a tipping point where 1% of the population owns 30%+ of the gross net worth of a country. Because of that overwhelming wealth, it results in the owners having a huge amount of influence in the political and legal processes. Meaning they can fund lobbyists to bend lawmaking to their wishes, and they can afford the absolutely best legal representation after breaking laws (i.e. powerful enough to be above the law).

That's seems to be crux of the wealth inequity argument imho. Viewing it as some type of "jealousy" by your viewpoint, and its systemic effect on the other (the OP).

Re:Expected

[iggymanz]

what is this "wealth inequity", what is this "unbreakable circle". there are plenty of minorities who work hard and are able to secure a good living in this country from zero even with very poor english skills and literacy..... but not the ones in Detroit. in Detroit you have a bunch of adult babies who only can whine and suck on the government teat while family structure breaks down and criminals make the place a ghetto. It's long past time to say "oh it's discrimination" or "oh they're a poor persecuted minority", those people need to get off their dead asses and make something of themselves.

Re:Expected

[FatdogHaiku]

Maybe it should be Motorola, Solutions?
Actually, I'm betting this was a failure at the local level, a couple of techs or more likely a middle manager that spreads their people too thin...

Re:Expected

[rtfa-troll]

Except again, from TFA, the city of Detroit was paying an enormous sum of money to a reputable vendor to maintain the system. How does that coalesce with this third world, wealth inequality theory?

Paying for things which don't get delivered is exactly how the third world manages to stay the way it is. Do you think the people there are lazy or genetically incapable or something? Basically what creates first world countries is a large group of people who are well enough paid and educated to understand what needs to be done and make sure it happens whilst at the same time not being rich enough to cut themselves off from the society and so having to care that everything gets done. These are exactly the people, in the middle, who are disappearing as wealth inequality increases.

The temptation of being paid to do something and not actually doing it is a standard thing that has been known for years. It's only when we get to the "full outsourcing" that our MBAs are so fond of that anyone would even consider getting rid of or selling off the people responsible for measuring and ensuring that the things that are paid for actually get delivered.

Re:Expected

[girlintraining]

Globalism. This is what happens when it's cheaper to move automotive manufacturing overseas only to be compounded further by the unions

Okay. Pardon the french, but I'm gonna have to ask you to take a step back, and literally go fuck yourself. Every major industrialized country except the United States has a labor party, and strong unions. The wealth gap in every other G20 country is significantly less than here in the United States. If your argument had even the slightest rootings in reality, the story would be very different. Unions had nothing to do with this; Rapid deregulation brokered by large corporations and a cozy relationship with Congress did. Unions have exactly dick to do with this -- it's just propaganda pushed out by Fox News and rabid conservatives who think profits are people too. Unions act as an economic stabilization force -- they cool the fluxuations in unemployment, wages, etc. Even Adam Smith in Wealth of Nations said Unions were a necessary part of a capitalist economy -- and he said the same about health care and unemployment insurance. He then went on to provide examples of how the long term growth of an economy improves with such policies; But they are usually not implimented because of short term focus on profit. He also advised governments to step in and create public works projects during periods of higher unemployment, back-filling the natural boot/bust cycle of capitalism, and then increasing taxation during periods of economic prosperity as an investment into the next cycle.

As much as many conservatives think they have a handle on what capitalism really is and what's best for it, they have a remarkable lack of education on the positions of its strongest supporters. It's unfortunate, really; If they weren't so fiscally irresponsible with their short term thinking and focusing on things like reducing government spending during a recession, etc., we wouldn't be stuck in these "stagflation" situations where inflation rises but unemployment remains constant. Such an (unpredicted) economic stall-out was first observed during the Reagan administration courtesy of "trickle down" economics. Its successor has resulted in the longest period of elevated unemployment in American history. And none of this has dick to do with Unions.

It's one giant death spiral that was enviable.

The death spiral isn't actually a spiral so much as a cycle. Deregulation leads to market crashes, which lead to regulation, which lead to market crashes, which lead to deregulation... Capitalism itself is fundamentally and systemically unstable, especially in its pure form. This is why almost every major world economy is a hybrid of socialist and capitalist policy, and any divisions between the two are largely arbitrary and based more on the political beer googles of the person assigning labels than what the economy is in actuality.

Re:Expected

[DigiShaman]

You don't have a clue. Sorry, but you don't. Unions no longer serve this nation in a global economy. They did when we didn't have OSHA standards, but that's about it. It's now swung so far the other direction as to become parasitic. Unions are nothing more than a protection racket with their pimp-lords funneling money into politics.

Also, we are headed out of stagflation and into inflation just as it happened in the 1980s. It's inflation that causes a rift in wealth inequality. But idiots on both sides of the isle want to spend money we don't have. Republicans spend on defense policing the world while Democrats spend domestically on stupid-ass projects and programs that don't do anything constructive or cohesive for our own national self-interests; only theirs (the Democrat party). But what the hell, they can just print more money and push the hot potato of debt further down the line. Why not? They get first spending rights to all new wealth created thus diminishing your own savings. That's assuming you even have any. It's like a sideband form of taxation in effectiveness without actually being a "tax" in the true meaning of the word.

In theory, both "trickle down" and "trickle up" policies will work in so much as they move money. It's the momentum that drives the economic engine. Now personally, I subscribe to the "trickle down" approach. However, even I will settle for "trickle up" rather then complete gridlock. Again, so long as it's based on sound policies (which they rarely are). The problem we face is a "trickle out". This nation is hemorrhaging wealth and opportunity. That's globalism. That's what happens and will continue to happen until a certain equilibrium is reached. Normally this kind pace happens at a steady rate. As you should know, the Internet changed all that. It was a shock to America's status quo of spending, taxing, and inflation. Now we are arguably in a financial death spiral with the BRIC nations so desperate to leave the US Dollar as the worlds reserve currency.

So what's the solution? There is none. Just watch it burn. Germany and Argentina have a lot to say about hyperinflation. Just ask them!

Re:Expected

[RubberDogBone]

Wait wait wait. Perhaps you hear the Motorola name and think this is a vendor with a proud and respected name and a great reputation. And that's true. 30 years ago.

The Motorola Solutions of here and now is notorious among Public Safety agencies for installing crap equipments, failures like this where the failover doesn't work, system problems, interface problems, junk radios, and all the while overcharging for all of it. Even when they low-bid something, they can be counted on to deliver a solution that doesn't actually, you know, work, and then negotiate extra fees to fix it. And fix it again, and again, Because by then, the agency has already spend X+Y dollars on it so they can't throw it away and get Kenwood or Harris in there. Besides, bad things happen when Moto loses contracts. Like it drops dead suddenly and mysteriously and doesn't come back for a long time. Sabotage? No. They just quit with the zero-day hot fixes.

Chiefs of Police and Fire and other agencies know this stuff goes on but good old /\/\ has lots of lobbyists and sales weasels throwing dinner and junkets at anything that breathes and can vote for or endorse a Motorola contract. If the local Chief doesn't want to play, then they go for the city managers, the city council people, the Mayor, the dog catcher, anyone they have to. They totally act like a big defense contractor except on a local level where the local yokels have no response this sort of action except to vote for Team /\/\ every time.

So in other words, the crap that happened to Detroit is FAR from the only similar situation where the system dies. NY has had problems, San Francisco, DeKalb County GA, Denver, systems all over the place. It happens so often that it's considered normal Moto behaviour and THAT'S scary.

Where are the other vendors? They're out there bidding too. But Motorola is sort of the penis enhancement of radios where "all the BIG GUYS have Motorola so you KNOW you want to have it too, don't you?" is what the sales reps say. You know that big city you want to be like? Yeah? They have Motorola. So sign right here. No no, that's not a contract change. We just corrected a mistake for you. Just sign it.

Re:Expected

[JBMcB]

Unions had nothing to do with this; Rapid deregulation brokered by large corporations and a cozy relationship with Congress did.

Sorry, in regards to the automotive industry, you have absolutely no idea what you're talking about. The Japanese, German and Korean car companies ATE THEIR LUNCH in the 80's, 90's and 2000's. That's what tanked the US car companies. Know how many foreign car companies have unionized factories? ZERO. OK, Toyota had ONE - NUMMI - that was a joint venture with GM (which was the only reason it was unionized) It's closed now, and they moved production to a non-union plant.

The union can be an important and positive force in the labor market. However, the auto worker unions have gone completely bonkers over the last 50 years. A couple years ago there was a minor scandal when Chrysler fired a dozen or so line workers for drinking and smoking weed on their lunch breaks. The UAW went to bat for these losers and got them reinstated. Got that? It's the UAW's policy that you can ingest legal and illegal intoxicating substances, then go work with heavy machinery, and it's perfectly OK.

I have dozens of stories about the self-destructive behavior of the UAW if you'd like more. Here's a quickie: the UAW retreat and conference center that costs the UAW $4 million a year. Good use of dues, right?

http://www.reuters.com/article/2011/09/22/us-usa-autos-union-property-idUSTRE78L29Y20110922

Re:And...

[djdanlib]

Last time we tried that, we got Australia.

Should have turned to the HAMs!

[blocsync]

I live in Florida, and when weather gets bad it can destroy critical communications equipment (including redundant systems). One thing I've seen done in the past is pushing communications through Amateur radio operators. Who (unlike the name would have you believe) are EXTREMELY professional and they tend to be able to very rapidly deploy communications equipment from the inner cities all the way out to the rural areas. Some of their equipment is capable of city and state coverage, but some of them can also establish international communication on a moments notice. This would have been a good fail-over for the lower priority calls. Just my 2 cents... http://www.arrl.org/ares has some info on the group I'm referencing.

Re:Should have turned to the HAMs!

[Technician]

A Motorolla trunked radio system consists of a pool of repeaters. A single repeater is placed into the control for the repeater site. On failure of the computer, all the repeaters switch over to a failover mode and become simple repeaters. All talkgroups vanish and all communications are shared by the repeater the radios have been assigned to. If properly assigned the dispatch and patrol should be all on one repeater. A second radio in the dispatch should be on another repeater along with top level "staff" communications.

Upon failure of an entire radio site due to power failure, antenna catastrophic failure, etc, then the backup site should kick in.

A poorly managed radio system will have radios assigned to repeaters at random. I have seen this, so when failover happens, teams can't talk to each other as they are not on the same repeater and other services are blended in so some patrols and some fire may be on the same repeater and some may be on another with no communication between repeaters.

This poor management happens when the person laying out the system is suddenly downsized and someone without knowledge of the plan has to add or replace radios and have no idea want the default radio assignments should be so they are assigned at random.

Re:Should have turned to the HAMs!

[civiltongue]

Do police officers' radios work on ham radio frequencies, or could thousands of ham radios actually be distributed to them in short order?

No to both. They'd both be illegal. What happens is that some ham operators work at the dispatch center, while others ride in patrol cars. It worked this way very effectively when L.A.'s Valley area transmitters all went down a few decades ago. A similar arrangement occurs when a major hospital's internal telephone system crashes, which happens every so often. Ham radio support groups are equipped and drilled for these and other comm emergencies.

Re:Should have turned to the HAMs!

[Mr. Freeman]

Nonsense. HAM radio operators are less relevant today than they have ever been. HAMs provided useful emergency communications two decades ago, but no longer. Nowadays, cellular providers truck out COWs (Cellular On Wheels. i.e. mobile cell sites) within hours of a disaster, or even preemptively if a disaster is expected. After a disaster, you'll see people talking on Nextels, not relaying messages through HAM operators. Sure, the red cross will accept volunteer radio operators, but only because policy dictates that they do so. They'll be told to set up useless radio stations and relay worthless information until they get bored and leave, or they're told that they're no longer needed.

If you want the truth about HAM radio involvement in emergency communications then you need to talk to HAMs that have actually attempted to participate in such activities within the last decade. Organizations like ARRL are going to talk up how important HAM radio is because that's the point of the organization. They'll tell you that some number of HAM radio operators participated during some disaster, but they won't tell you that the only traffic sent over those nets consisted of nothing more than periodic radio checks. They won't tell you that all HAM radio stations were set up in tactically unimportant locations. Half the time they're in areas which don't need a radio operator and the other half of the time they're set up next to a fully-functional radio truck which doesn't need any kind of HAM support.

In the situation submitted by the OP, HAM radio wouldn't have helped anyway. HAMs can only practically set up fixed locations, which are already served by landlines or personal cell service. A mobile solution would involve putting a HAM radio operator into every single squad car, which would be impractical for obvious reasons.

Re:Should have turned to the HAMs!

[Anonymous Coward]

After a disaster, you'll see people talking on Nextels, not relaying messages through HAM operators.

Might want to check on that, there are no more "Nextels", as the iDEN network was shut down on June 30th (though there is a regional provider in the southeast (SouthernLinc) that still supports the iDEN technology). Amateur radio does not rely on the cellular/landline/satellite communications infrastructure, and in a real disaster would be available instantly to assist with emergency communication needs.

HAMs can only practically set up fixed locations, which are already served by landlines or personal cell service.

Not sure where you are getting this, however Amateur radio operators are able to operate mobile as well as fixed locations. And during a disaster, there is a real possibility that landlines and personal cell services could be disrupted. There are a number of HAM organizations that operate repeaters throughout the country that provide mobile HAMs extended communications areas. Perhaps a HAM in every squad might not be practical, however claiming that Amateur radio for emergency communications isn't relevant simply isn't true.

Re:Should have turned to the HAMs!

[blocsync]

Actually, I don't need to speak to "HAMs that have actually attempted to participate in such activities within the last decade." As I am one. I'm not sure what bad experience you've seen with what sounds to have been a poorly organized net. However, it does not describe the entire community and it definitely doesn't apply to all situations. I don't think I've ever heard an actual "radio check" on a live emergency net either. Net Control tends to get very annoyed about low priority traffic like that.

Perhaps you're speaking only of a specific area or a specific group of HAMs, but I don't believe your comments apply everywhere.

Clearly in this situation, All Police/Fire/EMS/Dispatch personnel could have used Cell Phones to fill the void, but they didn't. There's a string of failures here, not just one system failing. My suggestion wasn't to replace their coms completely with HAMs, rather to use them in an organized NET to handle the lower priority calls, due to the concern over high volume on the state radio system.

I think people underestimate the degree to which people will volunteer and assist public services when called upon. You can criticize HAMs/Red Cross/etc... all you want for their failures, I'll judge them on their successes when few others are stepping up at all.

Re:Should have turned to the HAMs!

[brainboyz]

Someone's got a giant ignorant chip on their shoulder!

As an EmComm amateur radio operator, it's pretty obvious you've got no idea what HAM does in emergencies. We've stepped in when hospital phone systems fail, we've helped get health & welfare messages out of disaster areas when infrastructure isn't working for miles (call it unimportant but family outside the disaster zone sure appreciates it), we're actively involved on a mobile ad hoc basis when volunteer groups are deployed, and our equipment works despite cell towers being overwhelmed or out of power.

Are the end-all to communication? Hell no, we prefer when the cell and radio networks stay up. But, to say amateur radio is less relevant than ever is ignorant. We're a stop-gap until infrastructure can be replaced. After many major disasters those trucks can't get in, and it may be days before the roads are clear enough to get them in. Guess who's on the radio and organizing within an hour? The HAMs.

If needed my city does, in fact, have a plan for deploying amateur radio operators with units or trucks as needed. Ideally that won't be the case, but we've trained for it. The local sheriff was actually annoyed when we weren't activated during a power outage. He wanted the extra manpower to do useful, but non-priority, situational awareness tasks like checking neighborhoods and intersections for power. Cell towers were swamped, so non-comm volunteers would've had trouble dialing home.

Are our radios of limited use while infrastructure is alive and kicking? Sure. Do we train on redundant radios during non-emergency events? Yup, and yes, it seems silly to be using a handheld when the walkie-talkie works just fine at event-sized distances. But most of the city leaders are happy to have us.

Re:Should have turned to the HAMs!

[adolf]

Indeed, cell phones work for temporary problems.

I was once tasked with hauling gasoline up 12 flights of stairs to fuel a generator for a LEO repeater that was down due to a power outage caused by a ridiculous, one-in-a-few-lifetimes flood.

Yeah, I did pretty good at getting that done -- especially with the small, efficient Honda Inverter generator we had up there at first. But when the Honda died after a couple of days, and the gasoline-hauling sessions went from about 0.75 times per day to three times a day because the only available replacement generator was both huge and thirsty, nobody was harmed when the repeater would drop because I was delayed en-route*.

Indeed: They had other (simplex) channels they could talk on, and worst-case, every LEO had a cell phone.

*: What, you think it's easy getting from dry ground out-of-town to a repeater location that is in the river valley and surrounded by water? Nevermind a working place to buy gasoline to begin with. It was hard work getting that done, and the sleep deprivation didn't help any.

**: As an unreferenced footnote, it's not that any of this matters. It seems that Detroit suffered a central dispatch failure. And in that instance, depending on the pervasiveness of that failed system, it might have been impossible to look up a cell phone number.

Re:And...

[Anonymous Coward]

That turned out pretty well... It only took a few hundred years.

We should try it again. And put it on tv too.

Re:When seconds count, help is minutes away

[Lunix Nutcase]

Except the fault was Motorola's. But don't let facts get in your way.

So 230 calls ...

[wisnoskij]

That is a lot of communications.
I understand the need to worry about overloading state infrastructure with that many calls, and why just picking up a 50 cell phone could not have fixed this problem.

Re:When seconds count, help is minutes away

[wisnoskij]

Responsibility, it always slides down hill.

Re:My question

[Rockoon]

Because this is a very serious thing indeed. Somebody breaking into your house? 911

The 1911 isnt a very good home defense weapon. Shotguns are much better. The 1911 is for when you are out and about.

gun control?

[fazey]

For those of you who love gun control... this is why you should own guns.