Slashdot Mirror
New York City Has a Y2K-Like Problem, and It Doesn't Want You To Know About It (nytimes.com)
On April 6, something known as the GPS rollover, a cousin to the dreaded Y2K bug, mostly came and went, as businesses and government agencies around the world heeded warnings and made software or hardware updates in advance. But in New York, something went wrong -- and city officials seem to not want anyone to know. [Editor's note: the link may be paywalled; alternative source] New submitter RAYinNYC shares a report: At 7:59 p.m. E.D.T. on Saturday, the New York City Wireless Network, or NYCWiN, went dark, waylaying numerous city tasks and functions, including the collection and transmission of information from some Police Department license plate readers. The shutdown also interrupted the ability of the Department of Transportation to program traffic lights, and prevented agencies such as the sanitation and parks departments from staying connected with far-flung offices and work sites. The culprit was a long-anticipated calendar reset of the centralized Global Positioning System, which connects to devices and computer networks around the world. There has been no public disclosure that NYCWiN, a $500 million network built for the city by Northrop Grumman, was offline and remains so, even as workers are trying to restore it.
City officials tried to play down the shutdown when first asked about it on Monday, speaking of it as if it were a routine maintenance issue. "The city is in the process of upgrading some components of our private wireless network," Stephanie Raphael, a spokeswoman for the Department of Information Technology and Telecommunications, said in an email on Monday. She referred to the glitch as a "brief software installation period." By Tuesday, the agency acknowledged the network shutdown, but said in an emailed statement that "no critical public safety systems are affected." Ms. Raphael admitted that technicians have been unable to get the network back up and running, adding, "We're working overtime to update the network and bring all of it back online." The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover.
City officials tried to play down the shutdown when first asked about it on Monday, speaking of it as if it were a routine maintenance issue. "The city is in the process of upgrading some components of our private wireless network," Stephanie Raphael, a spokeswoman for the Department of Information Technology and Telecommunications, said in an email on Monday. She referred to the glitch as a "brief software installation period." By Tuesday, the agency acknowledged the network shutdown, but said in an emailed statement that "no critical public safety systems are affected." Ms. Raphael admitted that technicians have been unable to get the network back up and running, adding, "We're working overtime to update the network and bring all of it back online." The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover.
NYC Maintenance budget. You would assume they would switch out 5% of everything a year for a 20 year refresh cycle, but No.
Only a moron would want to live there.
FYI, if you split a quoted post into multiple quotes to respond to parent bullet points, this now trip's slashdot's lameness filter. /offtopic
she took a page out of Sarah Huckabee Sanders' book.
Every maintenance cycle has downtime. If they had double resources they could make a seamless transition, but some downtime for a major system overhaul is not really unheard of for any operation. Ideally you stagger it.
I'm sure things they didn't anticipate went wrong also, they always do, but to phrase this as if "any downtime" during a major firmware switchover (semi big operation done on a budget) is some big catastrophic fail is absurd.
There's a lot of top tier management that has to happen for an entire multi-agency fleet of networked devices to play nice, on any budget.
" The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover. " - Yeah, that's a question you ask when you LATER dissect what went wrong.
So no traffic tickets then???
"Ms. Raphael did not respond to questions about what the city had done to prepare for the rollover; she said that the city pays $37 million a year to Northrop Grumman to maintain and operate NYCWiN. A Northrop Grumman spokesman referred all questions to the city."
Maybe it went something like this?
to judge others by where they live or want to live is idiotic. I get tired of them trying to dictate to me, but all I want them to do is succeed from the nation. Again though, unlike them. Or you. Who am I to tell them where to live or how to live.
Anonymous comments are as pathetic as the anonymous "sources" that contaminate gutless journalism from the New York Time
They pay 45 million a year in support for that network to Northrop Grumman. GPS being the root of that downtime should have been easily fixable. The GPS epoch that ended was the second one since it's origin in 1980. It was entirely predictable down to day dates minutes and they had 20 years to prepare for it. Hell they even have 20 or so or more satellites with atomic clocks whose sole purpose for being built is calculating the time.
I don't see how they are supporting their claim of the City trying to keep people from knowing about this. Just because the government isn't jumping up and down declaring "we failed!" doesn't mean they are actively trying to oppress people from reaching that conclusion.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
"They pay 45 million a year in support for that network to Northrop Grumman" - Even so given the size/scale having 1% downtime for a major wide-net transition isn't "completely absurd" really... granted, they should have estimated better...
A big problem with GPS is so many 3rd party hw vendors made it proprietary-dirt-simple and "upgrading" to a new signal means gutting the existing logic, sometimes entirely. It's not a trivial upgrade whatsoever.
It's like writing new drivers for every single IOT device in an entire ecosystem. Yeah, it can be done, $40-45 million is about what that support ticket costs as a starting point.. but it's certainly not a trivial "reboot it" problem.
Saying they had "20 years to prepare" is kind of a major misnomer even if "sorta true" on some 60,000 feet level.
... should have been easily fixable...
Yes, should. I imagine the reason it isn't simple is because some of the platforms are out of support, others are locked down and the vendor has no interest in fixing them, some require a development tool that only runs on Windows 95, and they lost the source code for the rest. Then there's the ones they probably bricked while updating.
Note that I never morons where to live, I just said only morons live in New York.
It seems like they placed all of their non-critical systems on the wifi network. A temporary disruption causes inconvenience, but it's not catastrophic.
"The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover." I would say it raised answers not questions. The question, did NYC prepare for the GPS rollover, was answered a resounding and emphatic NO, they did not even try to prepare.
There's a bit of confusion here I think... There are tons of IoT devices, which may or may not have GPS problems of their own, but isn't the main issue just the network being down? Once that's back up, what remains of individual firmware rollouts for various types of devices can be addressed.
I'm assuming the NYCWiN has some sort of mesh topology? Maybe with scrambled GPS data it's now trying to reconnect to devices far away from where it actually is and is this failing to converge any routes? I'm a bit curious on the tech details here. Hope we learn more.
Hire a Linux system administrator, systems engineer,
Note that you calling anyone a moron, while you can't even write complete sentences and spell simple words like 'secede', is quite ironic.
> waylaying [...] the collection and transmission of information from some Police Department license plate readers
good.
man, I feel like mold.
>while you can't even write complete sentences and spell simple words like 'secede'
OP Anon didn't write that, Pgmrdlm did. Any moron could see that.
but all I want them to do is succeed from the nation.
I think you mean secede.
"but isn't the main issue just the network being down?" AFAIK no, it's not the main issue, that's a glaring symptom. This is a distributed WAN, it relies on GPS (somehow in there) for the mesh routing. It's not trivial.
You don't just flip a switch and the network suddenly communicates perfectly with outmoded end nodes. You need an IT guy to go out there to each one and set them up.
There's all kinds of dependencies, ordered operations, setups to redo, configs to restore and troubleshoot... all of this requiring man hours from limited men.
Duh. GPS devices are so standardized it's hardly an issue. Unless you purposely choose some proprietary system, it's as simple as reconnecting another device to an RS232 -or comparable- port. Antenna plugs are standard too. Even the sub-millisecond-precision 'tick' signal, sent by certain devices each second for exact time synchronization is standardized, but usually only scientist have a need for those. There is really no excuse at all to have your GPS unit not replaceable by *random brand* *random device*.
You happen to have a link to that contract, since apparently you've read it?
It's not unusual for a government contract to pay $X / year for the thing, with any changes or modifications of any kind requiring approval by the city, which means approval by several departments. The government is responsible for some things, such as providing locations to install the equipment, one contractor installs the utility power for the devices and keeps that running, another does the networking, a third provides the devices and their firmware.
A fix that involves 15 minutes fixing the code can involve months of bureacracy and coordinating who does what. Then one of the people at the city quits their job and you mostly start over with a new person who doesn't know what's going on.
Yes, Pgmrdlm is a moron also, in addition to "Note that I never morons where to live" AC. They can both be morons. Nobody asked where these inbred uneducated faggots wish to live, lol. Nobody cares.
It's whiny GOP Fox News faggot shit, not information of any use or value.
"The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover."
I think any question was answered the minute the network went down.
I didn't say this was absurd to have downtime. This isn't a planned outage for maintenance either. To be clear this isn't 1% down time. This issue began on Sunday the 6th, and the network has been down for at least nine or ten days. At this point the network has been down for about 3.6 % of the year, and that percentage is increasing.
What I said quite clearly, they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise. They had they designed the network properly would have been aware of this absolute unavoidable reality and been able to pre-emptively planned for and fixed the underlying causes.
They clearly did not, so the question becomes what exactly do they do for the 40 million dollar contract, if not maintain what they built and marketed as a safe alternative and reliable and viable critical information network.
There, FTFY. Older GPS devices? Not so much.
Sorry my math was confused. But my larger point stands;\ that ten days of a critical response network is not a prepared response. If it was possible that this was going to occur they should have spent months prior to alerting all parties and this news cycle would have been easily avoidable.
Clearly, The Machine is battling Samaritan again.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
What the hell kind of WiFi network depends on gps to function?
Maybe he just wants the nation to succeed? I mean, an admirable goal I guess?
Northrop Grumman designed and specced out the hardware. If they designed a network with a known death date due to a proprietary GPS it was a poor design. Even if it was an accident that they messed up and it was a human error in design, they still had years to research a replacement part to keep it viable, when they went and looked at the hardware in advance of the epoch change (what they get paid to do). This looks to me like they were caught flat footed (northrop grumman) with the failure to design/build/plan for the future and or a failure to maintain what they were paid to support operations viability and up-time of a network.
is the only possible explanation.
There is absolutely zero probability that a GPS time-source problem would cause the entire network to go down. None whatsoever. Either it is coincidence or incompetence.
My vote goes to incompetence.
A big problem with GPS is so many 3rd party hw vendors made it proprietary-dirt-simple and "upgrading" to a new signal means gutting the existing logic, sometimes entirely. It's not a trivial upgrade whatsoever.
And yet every cheapie android phone out of China managed it just fine. Since the event's timing has been known down to the second since the GPS system came in to existence, most devices didn't even need an update, they left the factory ready for the event. It's not like it's a whole new protocol, it's actually the same protocol just with a counter rolled past zero.
That's some pivot ya go there, Gunther.
No doubt everything you say is true. But, you stated that this should be easily fixable. I simply theorized that it isn't quite so easy to fix (or they would have done it), because of those poor decisions.
A good question here is: Should vendors be held to a standard of responsibility or is it the RFP writer's responsibility to ask for these things? If we decide to hold Northrop Grumman to a standard of responsibility, what is that standard and how is it enforced? What happens if all of the potential vendors turn out to be just as irresponsible?
Whole place should be nuked
Heh. If its the 2.5Ghz spectrum cellular radio network I think it is, the base stations all use GPS-referenced 10MHz source oscillators to feed the radio stacks. If they did not choose the reference clock hardware wisely then I could see why its taking so long to get things back: they have to touch every base station.
I pivot Fox News propaganda faggot shit right back to reality, yessir. Fuck the traitors.
It seems that if they had been at all pro-active they could have spent those months making the disruption not happen at all. It's not like every device has to wait until the rollover to be patched. They could have fixed this 5 years ago and not even skipped a beat when the rollover happened.
Lawyers write contract terms to predetermine these outcomes. NG has them, NYC has.. some... but I bet they're severely outclassed on that bent
When someone is paying you half a BILLION dollars, you are responsible for vetting the selected hardware against that sort of problem. When you're buying millions in hardware, you get to put things like that in the contract.
Given NG's line of business, none of this should have been new to them.
Northrop Grumman has an $299 unit fix.
When you're buying millions in hardware, you get to put things like that in the contract.
You get to, but you don't necessarily have to.
It will be interesting to find out exactly what was in the contract. That will come out in the lawsuit, if it's not somewhere publicly posted already.
Velociraptor = Distiraptor / Timeraptor
Since NG is also getting $45 million/year to maintain the network, it would be on them to put that in their contracts with any hardware bendors they worked with.
It will be interesting to find out exactly what was in the contract. That will come out in the lawsuit,
You're assuming NG didn't throw in a binding arbitration clause together with their general Disclaimer of Warranty and Force Majeure covering unexpected situations such as a GPS Rollover as part of the client onboarding.
Wow, that's amazing - you solved the problem by simply reading msmash's summary - you are like a technical savant. /sarcasm
The issue isn't the GPS hardware, it's the signal the satellites are sending and the software that is processing those signals.
Ken
It's the directed government answer to all media inquiries. And let's be honest: getting ambushed on the way into a building is a hit job, not journalism.
The issue isn't the GPS hardware, it's the signal the satellites are sending and the software that is processing those signals.
The satellites send a week number from 0 to 1023, which is a range of slightly less than 20 years. The GPS hardware needs to turn this into a date. You could build GPS hardware with firmware that assumes it will never be used _before_ the hardware was built, so the date must be between (starting date built into firmware) to (same + 1023 weeks).
This is fine if you assume your hardware breaks down within 20 years. If you assume it lasts longer, then the firmware must be updateable, so the start date can be updated every 19 years.
Uh, this is the second GPS date rollover since its inception in 1980. The first was in 1999, after 1024 weeks of operation.
There is no excuse for any device released or updated after 1999 to not account for this GPS glitch.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
Actually the firmware doesn't need to upgradable, you just need to be able to set and save the current date occasionally to non volatile memory. The device could do this itself periodically after synchronising itself. Yes, there is a small risk of spoofed GPS signals causing an anomalous date to be saved but even there the risk can be minimised. Time more that x seconds since last recording, more that y seconds of continuously synchronised signals. For mobile GPS devices more that z km's of travel as well while switched on.
A running system shouldn't fail across a GPS epoch event. Given this is not the first epoch event all of the equipment should have been able to handle this. This can be simulated in QA testing prior to releasing the firmware image.
"The problem has raised questions about whether the city had taken appropriate measures to prepare the network for the GPS rollover."
I'm no rocket scientist but seeing as how they're having massive problems due to the rollover I'd have to say no, they didn't.
Just cruising through this digital world at 33 1/3 rpm...
they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise.
It was no surprise. The people that were there way back when knew perfectly well about the problem, and they also knew perfectly well that they wouldn't be around to be blamed for it in 20 years.
It was easy for them at the time to make the decision to "acknowledge the problem" and to quietly pretend that someone else would fix it later.
All those people are long gone, and the people that came after them just kept kicking the can down the road until they ran out of road.
Just cruising through this digital world at 33 1/3 rpm...
Everyone making GPS devices should have known about this but now the Tom-Tom in my car thinks its noon once it gets a GPS lock.
According to their site, there's an update available.
I got an email from Garmin and did the required update and no problems.
It will be interesting to find out exactly what was in the contract. That will come out in the lawsuit,
You're assuming NG didn't throw in a binding arbitration clause together with their general Disclaimer of Warranty and Force Majeure covering unexpected situations such as a GPS Rollover as part of the client onboarding.
Force Majeure is in no way applicable to a man created event that has an 100% probability and known timeframe down to the second - 20 years in advance.
they have known (northrop grumman) about the epoch changing for 20 years. This shouldn't be a surprise.
It was no surprise. The people that were there way back when knew perfectly well about the problem, and they also knew perfectly well that they wouldn't be around to be blamed for it in 20 years.
It was easy for them at the time to make the decision to "acknowledge the problem" and to quietly pretend that someone else would fix it later.
All those people are long gone, and the people that came after them just kept kicking the can down the road until they ran out of road.
Some of us realized we would still be around when the clock ran out. I watched others deal with DEC's date-75 problem. I fixed the Town of Hudson's Y2K problems in 1998, and later reported a Y2K bug in DEC's software. The next problem will come when the Unix 32-bit seconds counter overflows in 2038. There are people who are concerned with such things, but we don't get much attention until the crisis is upon us and it is too late for any of the easy solutions.
This gives me lots of faith that we will have 0 problems in 2038 with the epoch rollover.
The next problem will come when the Unix 32-bit seconds counter overflows in 2038.
I can't wait to see the general havoc this causes, assuming I'm still alive to enjoy it.
Just cruising through this digital world at 33 1/3 rpm...
I realize that every byte of data has its cost when stored or transmitted, but can't anyone use a little bit of imagination when they're developing a technology which has moderate potential to expand widely or endure many years, and thereby design things accordingly?
Y2K, Y2038 (32-bit signed-integer seconds since 1970), IPv4 address depletion, GPS week number wrapping...
And how many character encodings were there before Unicode? And even after the Unicode 1.0 standard appeared, and anyone could see that representing even a core subset of Asian characters would go beyond 16-bit unsigned integers, languages like Java and C still could not actually put a Unicode character code point beyond BMP-0 in to the "char" type, and Windows had a quasi-Unicode BMP-0 16-bit "wide character".
And Windows, with its infuriating long-time low limits on things like file names, path lengths, computer names, etc.
And TCP/IP port numbers! I know a "connection" is a tuple, (remote IP, remote port, local IP, local port), which means any pair of IPs can have ~4B connections, but if you want to choose a listening port number for a service, the 16-bit port space seems rather crowded -- and it seems quite obvious that a 32-bit port space would greatly reduce collisions and possibility of port-exhaustion (in DOS or not-so-uncommon stupid socket TIME_WAIT scenarios). Kudos for IPv6, and surely IPv6 address space can be used creatively to compensate for the 16-bit port nonsense, but...lame!
And yet every cheapie android phone out of China managed it just fine.
Mobile phones do not critically rely on GPS time for any function. Now the infrastructure behind them on the other hand critically require precise time sources for data synchronisation.
Your argument is like saying I survived just fine in the last blackout without a generator, why would hospital need one.
Don't make those arguments. They are anti-intellectual. And shame on the people who modded you up.
Technically, the 32-bit signed integer that holds the Unix seconds value overflows and goes negative! It does not overflow to zero. Which means some systems will jump back in time to 1970 - 68 = 1902.
Note that Y2038 failures will start to manifest themselves when attempts are made to set timers past the overflow date. Therefore, failures will start to appear before the actual overflow date.
Any system calls, protocols and file systems that use 32-bit signed absolute time fields will also be impacted. Therefore, 64-bit systems are not immune to Y2038 due to inter-operability support for other systems and non-compliant 32-bit applications.
In particular, the automobile industry is going to be at high risk of hitting Y2038 failures as vehicles built today may still be on the road in 2038.
I note that Linux v5.0 introduced some Y2038 compliant fixes which removed get_timeofday(). This promptly broke a 3rd party kernel loadable module that relied on the now non-existent get_timeofday(). Therefore the Y2038 havoc has already started...
Also it is not possible to set a smartphone past the year 2036. So currently Y2038 on smartphones has been "fixed" by not allowing the smartphone to get too close to the year 2038!
Actually, why would the communications network even NEED GPS to function at all?
In 1977 as a trainee COBOL programmer our lecturer pointed out that data in our system was expected to outlast the aircraft we would be writing management reporting programs for and there was potential for problems in the year 2000 if we used 2 digit years. We were repeatedly warned that if we ever came across 2 digit years we were to raise the danger with our team leader. As late as 1997 I was meeting new program specs with 2 digit years. In 1998 and 1999 I made a fortune investigating systems for Y2K compliance. What was fun was that the client was a well run company and the code was all compliant.
so the question becomes what exactly [does Northrup] do for the 40 million dollar contract, if not maintain what they built and marketed as a safe alternative and reliable and viable critical information network.
well, my dad just retired from Northrup so im guessing it was his fault.
I half expect someone to decide the right thing to do is create a new era named AR (After Rollover) and start adding AR to some dates.