Passage of Time Solves PS3 Glitch

An anonymous reader writes "A quick update on the widespread PlayStation 3 glitch we discussed recently: as of last night (Monday, March 1st) the problem has resolved itself. I powered up my PS3 to find the clock was set to April 29th, 2020, but once I went into the system menu and set the date and time via the internet I got an accurate date. That seems to be the test of whether your PS3 is 'fixed' or not; Sony says you should be all set."

Comments

Sony's Official Announcement

[eldavojohn]

Here is the official press release I received in an e-mail this morning:

First Quadrennial Childhood Obesity Awareness Day Goes Off Without a Hitch!

We'd like to thank all our gamers for observing our compulsory First Quadrennial Childhood Obesity Awareness Day that we had planned many years in advance back when we made our first consoles. We hope all the children took the time to get outside and exercise. As always, Sony endorses moderation in game play and we feel that this surprise holiday away from the Playstation Network will help today's youth become more healthy and social.

Should we decide to surprise you with a second Quadrennial Childhood Obesity Awareness Day, it could happen March 1st of 2012. See you in 2012 (maybe)! Until then, remember to get plenty of fresh air and exercise!

Totally in control of the situation,

Muhammed Saeed al-Sahaf
Chief Sony Public Think of the Children Relations Officer

Re:Sony's Official Announcement

[Zenaku]

So instead, you are suggesting that they think 0x10 is the same as 10?

No, I am suggesting that person or group A developed a hardware clock that stores a set of values as binary-coded decimals, and person or group B, probably at an entirely different company, wrote firmware to read those values as binary numbers without verifying whether that was the correct format. Either someone didn't document it, or someone else didn't read the documentation.

I don't think that is much credit.

Hence my emphasis of the word little. It is certainly a more understandable path to failure than "Durrrr, leep yeers every 2 years i am a pogrammer, Yay!"

Re:I can't wait for April Fool's Day...

[Mongoose Disciple]

Is SONY going to make my PS3 explode?

No, but every twenty minutes a kitten will come out.

That'll be fixed in the next update though.

Sony is very lucky...

[fuzzyfuzzyfungus]

That this particular bug "fixes" itself in a relatively short span of time.

According to the documentation provided for PS3 linux, the clock that is embedded in the PS3 cannot actually be manipulated from under the hypervisor:

"Similar to a PC, a built-in real time clock (RTC) keeps the wall clock time for the PS3. The RTC is backed up by a battery and so ticks even if external power is removed. The RTC value can be read by a hypervisor call, but it can not be written. The RTC value monotonically increases and never rolls back. The PS3 Linux platform support uses the standard RTC userland interface /dev/rtc, The standard hwclock command can be used to manage the RTC. Since the RTC is read only, the PS3 Linux platform support maintaines a value in system flash memory that represents the difference between the hardware RTC value and the Linux RTC value."

I'm assuming that this read-only clock "feature" is in some way related to DRM, to keep people from playing tricks with expiration dates. Worst case scenario, it is impossible to modify the RTC without hardware tinkering. Had this not been a transient bug, that would have meant massive physical recalls. More likely, it is possible for sony-blessed firmware updates to modify the clock. However, Sony can only push those either through the internet, or on physical disks. Since the bug was preventing PSN logins, the internet option wouldn't have been automatically available(though, since the issue is transient, it now is again). They would either have to mail out upgrade disks to affected users, bundle the upgrade with future game releases, or make their customers go to some support site and burn their own upgrade disks. Gigantic pain in the ass.

So what were they supposed to do?

[Dr. Manhattan]

I'd love to hear an explanation from Sony about how, exactly, they managed to have this bug exist in the first place.

Well, apparently, it wasn't a Sony bug per se, it was a bug in one of the support chips.

Sony decided to be paranoid about time because of pirates. If you can hack the PS3 and change the date, then you can avoid expiration times and so forth. So if the hardware clock and software clock get out of sync, their DRM and such stops working. Considering the PS3 is the only major console that has not been hacked to the point of widespread piracy, keeping to this level of paranoia seems to have paid off for Sony's purposes.

As to Sony's "piss-poor handling of the entire incident", I'd like to know what, exactly, you think they should have done about it?

Seriously, I've just appointed you, _xeno_, to be CEO of Sony, and you just got a phone call. "Oh, crap, it's midnight GMT on March 1st, 2010, and all the older PS3 consoles can't play downloaded content or games with trophies or sign into the PSN!". What are you going to do? What orders do you give?

Re:So what were they supposed to do?

[_xeno_]

You may not be aware of this, but they have a Twitter account devoted to the PlayStation and a blog. Sadly, the blog doesn't record the time when the entries were posted, but you may notice the 13-hour stretch between "slim consoles still work" and - well, actually, if you follow the link, it basically reiterates that slim consoles are working.

Then complete silence until the 24-hour period ended, followed by a brief announcement that "hey, it works again!" and then completely ignoring that it ever happened. Instead they've posted several blog entries that conveniently knock the PSN outage way down the page.

Do they intend to fix this issue with a patch? Can they? Does it even matter? Who knows, they certainly aren't saying. All they've said is "oops, sorry" and, well, that's it. Not even a "we're still looking into this matter."

Of course, based on the vague "if we get new information we'll keep you posted," I get the impression that this isn't the fault of the people running the blog, it's that the PlayStation group themselves are simply not bothering to communicate. Maybe they're still looking into it, maybe they aren't, but the community managers apparently have no idea based on the weasel-wording on the blog. And that would be a problem that Sony should address.

But in any case, I still have to wonder: why in the hell does a reduced instruction set computer have a buggy leap-year function? Why the hell does it care what the human-readable date is? All it needs to do is keep track of "units of time since a known start point." Let the OS worry about what the human time is.

Re:So what were they supposed to do?

[Dr. Manhattan]

Then complete silence until the 24-hour period ended, followed by a brief announcement that "hey, it works again!" and then completely ignoring that it ever happened. Instead they've posted several blog entries that conveniently knock the PSN outage way down the page.

A major bug knocks out significant functionality in a console with an installed base in the tens of millions. Remember, you're the CEO of Sony, and you have to protect the value of your company. It would be criminally irresponsible if you were to rush out an untested fix. If that broke anything you'd be subject to lawsuits. Firmware updates are risky at the best of times.

Meanwhile, your engineers are telling you, "We've got a problem with the date that's screwing up DRM. On our special development consoles, it looks like once the date rolls over in less than 24 hours, the problem will go away. We've tested it on a handful of our customer-style consoles, and from what we can see it appears to be the case there, too. But there are seven 'Fat' models out there and in these few hours we can't test 'em all. Even once that's fixed, we can't absolutely guarantee all will be working after that."

So, you're careful about what you say, and you proceed with deliberate speed. The problem hasn't even been resolved for 24 hours yet! I strongly suspect that they are working on adding a fix to the next firmware upgrade - but that means they'll need to delay the next upgrade, add new tests to the regression tests and QA process, evaluate the fix on all nine models of PS3 (plus the two new slim models in the pipe), and then finally roll it out.

Every company with a substantial codebase and millions of customers is this slow, by necessity. It took Microsoft a week to get the Live network stable after the flood of new users back in December, 2008. A week later (i.e. seven times 24 hours), they gave away a free downloadable game as a further apology. And they still got sued over it.

I strongly suspect that Sony will release more information soon, and may offer a downloadable trinket as a further apology, too. But expecting a giant company to share technical information (that might be used in a lawsuit) in real-time is a bit much.