Slashdot Mirror
New York City Has a Y2K-Like Problem, and It Doesn't Want You To Know About It (nytimes.com)
On April 6, something known as the GPS rollover, a cousin to the dreaded Y2K bug, mostly came and went, as businesses and government agencies around the world heeded warnings and made software or hardware updates in advance. But in New York, something went wrong -- and city officials seem to not want anyone to know. [Editor's note: the link may be paywalled; alternative source] New submitter RAYinNYC shares a report: At 7:59 p.m. E.D.T. on Saturday, the New York City Wireless Network, or NYCWiN, went dark, waylaying numerous city tasks and functions, including the collection and transmission of information from some Police Department license plate readers. The shutdown also interrupted the ability of the Department of Transportation to program traffic lights, and prevented agencies such as the sanitation and parks departments from staying connected with far-flung offices and work sites. The culprit was a long-anticipated calendar reset of the centralized Global Positioning System, which connects to devices and computer networks around the world. There has been no public disclosure that NYCWiN, a $500 million network built for the city by Northrop Grumman, was offline and remains so, even as workers are trying to restore it.
City officials tried to play down the shutdown when first asked about it on Monday, speaking of it as if it were a routine maintenance issue. "The city is in the process of upgrading some components of our private wireless network," Stephanie Raphael, a spokeswoman for the Department of Information Technology and Telecommunications, said in an email on Monday. She referred to the glitch as a "brief software installation period." By Tuesday, the agency acknowledged the network shutdown, but said in an emailed statement that "no critical public safety systems are affected." Ms. Raphael admitted that technicians have been unable to get the network back up and running, adding, "We're working overtime to update the network and bring all of it back online." The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover.
City officials tried to play down the shutdown when first asked about it on Monday, speaking of it as if it were a routine maintenance issue. "The city is in the process of upgrading some components of our private wireless network," Stephanie Raphael, a spokeswoman for the Department of Information Technology and Telecommunications, said in an email on Monday. She referred to the glitch as a "brief software installation period." By Tuesday, the agency acknowledged the network shutdown, but said in an emailed statement that "no critical public safety systems are affected." Ms. Raphael admitted that technicians have been unable to get the network back up and running, adding, "We're working overtime to update the network and bring all of it back online." The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover.
>> NYCWiN, a $500 million network built for the city by Northrop Grumman
For $500M you could have built and deployed the system ($40M?) and then put the rest ($460M) in a trust whose proceeds could have funded maintenance forever.
well, it's hard to compete with the bright lights and excitement of Sheboygan, but New Yorkers seem to make do.
Maybe it went something like this?
From what I had read earlier last week, the issue was affecting an emergency response network where they pay Northrop Grumman 40 million a year to maintain.
Maintaining it each year is as much as building it should have cost.
They pay 45 million a year in support for that network to Northrop Grumman. GPS being the root of that downtime should have been easily fixable. The GPS epoch that ended was the second one since it's origin in 1980. It was entirely predictable down to day dates minutes and they had 20 years to prepare for it. Hell they even have 20 or so or more satellites with atomic clocks whose sole purpose for being built is calculating the time.
I don't see how they are supporting their claim of the City trying to keep people from knowing about this. Just because the government isn't jumping up and down declaring "we failed!" doesn't mean they are actively trying to oppress people from reaching that conclusion.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
"The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover." I would say it raised answers not questions. The question, did NYC prepare for the GPS rollover, was answered a resounding and emphatic NO, they did not even try to prepare.
> waylaying [...] the collection and transmission of information from some Police Department license plate readers
good.
man, I feel like mold.
I didn't say this was absurd to have downtime. This isn't a planned outage for maintenance either. To be clear this isn't 1% down time. This issue began on Sunday the 6th, and the network has been down for at least nine or ten days. At this point the network has been down for about 3.6 % of the year, and that percentage is increasing.
What I said quite clearly, they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise. They had they designed the network properly would have been aware of this absolute unavoidable reality and been able to pre-emptively planned for and fixed the underlying causes.
They clearly did not, so the question becomes what exactly do they do for the 40 million dollar contract, if not maintain what they built and marketed as a safe alternative and reliable and viable critical information network.
I live in NYC, and have my whole life. It mostly sucks. Giuliani fixed a lot of it, but going downhill to the 70s again, fast
-- 73 de KG2V For the Children - RKBA! "You are what you do when it counts" - the Masso
Northrop Grumman designed and specced out the hardware. If they designed a network with a known death date due to a proprietary GPS it was a poor design. Even if it was an accident that they messed up and it was a human error in design, they still had years to research a replacement part to keep it viable, when they went and looked at the hardware in advance of the epoch change (what they get paid to do). This looks to me like they were caught flat footed (northrop grumman) with the failure to design/build/plan for the future and or a failure to maintain what they were paid to support operations viability and up-time of a network.
A big problem with GPS is so many 3rd party hw vendors made it proprietary-dirt-simple and "upgrading" to a new signal means gutting the existing logic, sometimes entirely. It's not a trivial upgrade whatsoever.
And yet every cheapie android phone out of China managed it just fine. Since the event's timing has been known down to the second since the GPS system came in to existence, most devices didn't even need an update, they left the factory ready for the event. It's not like it's a whole new protocol, it's actually the same protocol just with a counter rolled past zero.
I was involved with a similar project in a different city, and just the infrastructure upgrades to the shelters and towers ran about $20 million per site for about 20 sites. None of them had been designed to a sufficiently robust criteria originally. The networking bits were easily another $10 million per site from what I understood.
Heh. If its the 2.5Ghz spectrum cellular radio network I think it is, the base stations all use GPS-referenced 10MHz source oscillators to feed the radio stacks. If they did not choose the reference clock hardware wisely then I could see why its taking so long to get things back: they have to touch every base station.
It will be interesting to find out exactly what was in the contract. That will come out in the lawsuit,
You're assuming NG didn't throw in a binding arbitration clause together with their general Disclaimer of Warranty and Force Majeure covering unexpected situations such as a GPS Rollover as part of the client onboarding.
Actually the firmware doesn't need to upgradable, you just need to be able to set and save the current date occasionally to non volatile memory. The device could do this itself periodically after synchronising itself. Yes, there is a small risk of spoofed GPS signals causing an anomalous date to be saved but even there the risk can be minimised. Time more that x seconds since last recording, more that y seconds of continuously synchronised signals. For mobile GPS devices more that z km's of travel as well while switched on.
A running system shouldn't fail across a GPS epoch event. Given this is not the first epoch event all of the equipment should have been able to handle this. This can be simulated in QA testing prior to releasing the firmware image.
"The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover."
I'm no rocket scientist but seeing as how they're having massive problems due to the rollover I'd have to say no, they didn't.
Just cruising through this digital world at 33 1/3 rpm...
they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise.
It was no surprise. The people that were there way back when knew perfectly well about the problem, and they also knew perfectly well that they wouldn't be around to be blamed for it in 20 years.
It was easy for them at the time to make the decision to "acknowledge the problem" and to quietly pretend that someone else would fix it later.
All those people are long gone, and the people that came after them just kept kicking the can down the road until they ran out of road.
Just cruising through this digital world at 33 1/3 rpm...
they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise.
It was no surprise. The people that were there way back when knew perfectly well about the problem, and they also knew perfectly well that they wouldn't be around to be blamed for it in 20 years.
It was easy for them at the time to make the decision to "acknowledge the problem" and to quietly pretend that someone else would fix it later.
All those people are long gone, and the people that came after them just kept kicking the can down the road until they ran out of road.
Some of us realized we would still be around when the clock ran out. I watched others deal with DEC's date-75 problem. I fixed the Town of Hudson's Y2K problems in 1998, and later reported a Y2K bug in DEC's software. The next problem will come when the Unix 32-bit seconds counter overflows in 2038. There are people who are concerned with such things, but we don't get much attention until the crisis is upon us and it is too late for any of the easy solutions.
Since when does a union boss gets payment when a company conducts a project?
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
He probably has no idea how few money 40million actually are (his proposal).
Assuming a router or repeater costs $20 and a work hour $20 too, and making the optimistic assumption a worker sets up 4 routers per hour, then we have about $100 costs per hour (including the routers). So 40,000,000 / 100 is 40,000 work hours and 160,000 routers which covers a square of 400 x 400 routers. With about 100m distance from router to router that would be a 40km square, something like 25miles x 25miles.
If a worker is really that quick (and that cheap) it actually could work out :P but a high end/high quality router is most likely more expensive, and the grid/mesh is probably much smaller than 100m.
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
And yet every cheapie android phone out of China managed it just fine.
Mobile phones do not critically rely on GPS time for any function. Now the infrastructure behind them on the other hand critically require precise time sources for data synchronisation.
Your argument is like saying I survived just fine in the last blackout without a generator, why would hospital need one.
Don't make those arguments. They are anti-intellectual. And shame on the people who modded you up.
I really do enjoy it when I get modded Troll by people who have no idea. $40million? Ha we spent more than that on a wireless project at a large chemical plant. If I had to give a thumb in the air estimate I would have come at larger than $500million for city wide multipurpose wireless infrastructure.