Sony Forgets To Pay For Domain, Hilarity Ensues

First time accepted submitter Dragoness Eclectic writes Early Tuesday, gamers woke up to find out that they couldn't log in to any Sony Online Entertainment games--no Everquest, no Planetside 2, none of them. Oddly, the forums where company reps might have posted some explanation weren't reachable, either. A bit of journalistic investigation by EQ2Wire came across the explanation: SOE forgot to renew the domain registration on SonyOnline.net, the hidden domain that holds all their nameservers. After 7 weeks of non-payment post-expiration, NetworkSolutions reclaimed the domain, sending all access to Sony's games into an internet black hole. Sony has since paid up. SOE's president, John Smedley, has admitted that the expiration notices were being sent to an "unread email" address.

Comments

Black hole?

[djupedal]

Hole in someone's head, maybe - after all, a simple spreadsheet to track something this basic or a reminder in a calendar with alerts with someone assigned to keep an eye on things would take care of things like this. They're lucky it wasn't held hostage...

Re:Black hole?

[Anonymous Coward]

You want to assign someone to keep an eye on things that can be fully automated? Is your hair pointy?

Re:Black hole?

[lgw]

Same thing happened to Turbine a couple years back: DDO, LotR, etc all down for exactly the same reason. You wouldn't think this would be that hard to get right, but chances are no one in dev at either company survived from the early days to when the problem happened, so the tribal knowledge was lost.

Re:Black hole?

[geekoid]

" simple spreadsheet to track something"
that is the bane of corporations. Important info sitting in a spreadsheet, somewhere.

Re:Black hole?

[Anonymous Coward]

When I fire someone, I redirect their email to their supervisor. It's right there in their employment contracts that their work email address and any correspondence are the property of the company (as if that wasn't obvious, but CYA applies). For things like this we have title addresses like dnsadmin@example.com, noc@example.com etc. which are broadcast to several staff responsible for the management of such affairs.

Also payments such as these are lodged in our recurring expenses ledger and paid by accounts payable. You can't incur recurring expenses here without making a ledger entry as the account would not get paid thus the domain name would never have been registered. I guess if you were a total dick you could try and sneak a recurring expense invoice past AP as an NRE, though I kind of hope our AP people are clueful enough to catch shenanigans of that sort.

Re:Black hole?

[theskipper]

There's no law per se, but there is a recent ICANN requirement called "Whois Accuracy Data Specification". It requires registrars to contact the registrant and click an emailed link as validation that their whois info is correct. The domain can be suspended if the validation isn't done within 15 days.

The intent is good but the implementation is pretty mindboggling. They're expecting every owner of a domain name to check that the email sent to them is not a phishing attempt...how that's supposed to work reliably is anyone's guess.

So, yeah, owners are supposed to verify to the registrars that the info is accurate which you could say is "ICANN's law". But not legally. Here's one of many articles that goes deeper into the issue:

http://blog.easydns.org/2014/0...

Re:Black hole?

[raftpeople]

Good point, I better print the email reminder and place that in the three ring binder that sits behind my desk.

Re:Black hole?

[Qzukk]

This is apparently my president's nightmare because he will call me at midnight and ask me when our domains and SSL certs expire.

Re:Black hole?

[LordKronos]

Actually, 10 years is the max registration. And that's exactly what I do. Throwaway domains that I'm experimenting with might only get a year or 2, but once anything becomes important to my business, it gets renewed for 10 years. The same is true for my personal domain. And every couple years I go through and bump it back up to the max. I'd literally have to go 10 years without remembering to renew a domain before one would expire. I can't see why any business would do otherwise.

Re: Black hole?

[pla]

here's the law. you want me to do any of your other homework for you?

Not the GP, but yeah, I do - Can you explain what an anti-domainsquatting law that specifically deals with trademarks and identity theft, and absolutely nothing to do with simply giving fake info to a registrar, has to do with your original claim that giving ACCURATE contact info counts as US law?

Now, ICANN can enforce its policies on the registrars themselves, simply by virtue of the fact that a registrar requires ICANN's continued blessing to operate. But the only recourse they have about (non-identity-stealing) fake registration info comes down to taking the domain away from you. For someone like Sony, that might look like an end-of-the-world scenario. For someone who just wants a named place to stick stuff online for my own personal use? Meh, worst case, I've lost $10-$15 and I have to wait three days for a new domain to propagate (and not always even out the money - Much to my surprise, I actually had GoDaddy refund me when I flatly refused to send them a photocopy of my license, three months into a registration).

Re:Black hole?

[Nevo]

Yes, somebody should absolutely be assigned responsibility to keep up with things like this. Because when no one's assigned the responsibility, well, then you get things like domains expiring.

Re:Black hole?

[rhodium_mir]

Network Solutions offers kinda-sorta-100 year registration. Technically it's just a ten year registration that they automatically renew for you every ten years, but it still would've saved Sony a lot of trouble in this case.

Network Solutions price chart

20 and 100 Year Domain Registration Service - If the domain name registry of a particular third level domain does not provide for an initial registration term of 20 or 100 years, then Network Solutions will register your domain name on your behalf for the maximum term available at the respective registry, and as long as your domain name is registered with us, we will continue to add additional years to your registration on an annual basis up to the total of 20 or 100 years, depending upon the term you select from the date of purchase.

Re:Black hole?

[TitusGroan8856]

have you heard of automated systems breaking? this is why someone should be assigned and responsible for it.

Re:Black hole?

[Antique Geekmeister]

I'm afraid that the current "whois" practices were deliberately set up to allow plausibility deniability, to protect the domain owners from being actually reached by the spammers and numerous sales people or lawyers with cause to contact domain owners. The domain vendors benefit from this: they can follow the letter of the law, but not actually support contacting the domain owners to handle criminal or abuse behavior, and wait for days, weeks, or years while lawyers collect the evidence and chain of repeated contact failures before a court order can be obtained.

In the meantime, they're collecting the registration fees, in bulk, for the relevant domain and all the related domain names. The current system is a critical revenue stream, which the domain and SSL key vendors have no need or desire to encumber by enforcing legitimate contact information.

Re:Black hole?

[ArcadeNut]

Or....... just buy the domain for 100 years... it's not like Sony wouldn't make the registration fees back and they would never have to worry about it again....

Re:Black hole?

[DigiShaman]

I've fucked up and forgot when an SSL cert was about to expire. I found out the next morning when their iPhones could no longer access the Exchange server. Shit happens. This time I include SSL, Domain, and Server hardware warranty expiration notices scheduled way in advanced in my calendar as an event.

Re:Black hole?

[ketomax]

I bought mine for 10 no problems.

Re:Black hole?

[ArcadeNut]

Network Solutions offers it:

http://www.networksolutions.co...

Re:Black hole?

[pushing-robot]

Because something that has to be done every year gets done every year, like taxes.

Something that has to be done every 10+ years is a lot more likely to get lost and forgotten. Sure, you could set a reminder...but where? Staff get replaced, calendars get replaced, software gets replaced, computers get replaced, offices get cleared out, and the people who trained the current employees weren't even around themselves the last time it needed to be done.

It's like the hundred year $DISASTER, which kills hundreds and causes billions in damage simply because it's so rare. If it happened every year, damages would paradoxically go down because building codes would improve and the public would be better prepared.

Re:Black hole?

[Opportunist]

Do you have any idea how that's gonna look on your quarter report? Not to mention that you might not be around in 10 years, let alone 100, so why bother paying for the domain so far in advance?

Welcome to corporate thinking!

Re: Black hole?

[mwvdlee]

Just to add my 2c. A while ago I was working on a project which could use data in WHOIS records. Ultimately this failed because the data is very unreliable and mostly unavailable, but I did come accross some laws.

Seems the U.S. is pretty much the only country that has a law on this, and it just says that it is illegal to have inaccurate information in a WHOIS record if and only if you're using that inaccurate information to scam people. So basically you can use inaccurate information all you want but if you're conficted of scamming people online, the use of inaccurate WHOIS information can be used to add some additional jailtime.

Re:Black hole?

[cryogenix]

100 year registration is dirt cheap compared to what happened as a result of it expiring :)

Re:Black hole?

[cryogenix]

It wouldn't hurt to have a distribution group for this and then make yourself and others a member of the group, even if it's your boss. Best case scenario, he gets the alert and says, Bob, did you see the alert about... Already took care of it this morning. Good man.

Re:Black hole?

[lgw]

Wow, that's a non-sequitur (and, BTW, like most things on Slate, the argument fails: the SCOTUS ruling explicitly hinged on the fact that Hobby Lobby was a closely held corporation, and thus no different from a partnership It was not a broad ruling applicable to corporations in general, where the linked argument might have been relevant.)

Re: Black hole?

[Talderas]

bigtits.com?

It's obviously an avian enthusiast website. What else would something think it be?

Re:Black hole?

[FatLittleMonkey]

You want to assign someone to keep an eye on things that can be fully automated?

It's a basic accounts payable function, so yes.

Someone has to have authority to maintain and modify the automated payment schedule, otherwise either anything can be added/removed or nothing can be added/removed. Moreso, someone within their accounts payable department should be specifically responsible for all these particular kinds of payments: trademark fees, site ownership fess, official registrations, patent renewal fees, etc. That person should have lost their job today.

I'm a bookkeeper, I would lose my job if my employer's domain was cut off because a) I didn't pay the account, and b) my excuse was, "<shrug>I don't read that email account any more."

Hell, SOE is a large enough corporation in its own right that they should have someone whose sole job is to make sure contact information at vendors is up to date in response to structural changes and staff movements, as well as checking email accounts for employees who are no longer at the company or divisions that no longer exist to ensure that critical info isn't being lost. And an entirely separate person (but working in the same section) whose job is to make sure SOE's internal contact & billing information is updated for all of the major clients, to ensure that SOE itself is also sending accounts to the right companies/divisions and contacting the right people at those companies, to prevent SOE from causing similar embarrassments to SOE's major clients/partners/etc.

And Sony Corporation (the parent) should have at least one person whose sole job is to liaise between those "Contact Managers (internal/external)" in the different subsidiaries and major divisions of Sony, to ensure that the whole group is up to date.

And that person should have lost their job too.

Seriously, fuck that guy.

Re:ring ring

[Zaelath]

I immediately thought this too, but you try ringing one of these corporations and see how far you get.

7 weeks?

[MrLogic17]

Wow, giving the company 7 weeks before Network Solutions took the site down? That's going way above & beyond. The average luser like me would be taken down the day of expiration.
 

Re:7 weeks?

[rmdingler]

That's the problem with too much leeway.

The procrastinator in us invariably assumes there is seemingly an infinite amount of time to take care of this... by the time they send the We really fucking mean it this time! notice, well hell, they've been crying "wolf" so long it doesn't mean anything.

And to be fair, didn't the nerdtastic Mecca website herself forget to renew a certificate recently?

Black hole?

[Bovius]

This sort of lapse has happened in every company I've worked in, big and small, when the person formerly responsible for this kind of thing leaves the company and someone else has to pick up their responsibilities. Sloppy, unorganized? You betcha. Also what I've come to expect.

Re:ring ring

[rudy_wayne]

you try ringing one of these corporations and see how far you get.

Exactly. Unless you know someone or have some inside connections, it is virtually impossible to contact someone, who actually knows something, using publicly available information. And I'm sure that NetworkSolutions really doesn't want to spend time calling everyone who lets their registration lapse.

The real problem is that Sony couldn't be arsed to register the domain names using a working e-mail address that actually goes to the person at Sony who is responsible for such a thing.

Not the first time!

[rstanley]

I long for the good ole days when they actually send out paper invoices in envelopes! ;^)

And from the archives:

"In December 1999, Microsoft forgot to renew the domain name Passport.com,
and so rendered its Hotmail service partially crippled. A Linux
programmer, Michael Chaney, paid the $35 fee and promptly handed over
ownership to Microsoft."

It happened again in 2003:

http://www.theregister.co.uk/2...

Will they ever learn? ;^)

Re:Not the first time!

[zephvark]

I long for the good ole days when they actually send out paper invoices in envelopes! ;^)

You actually still look at your paper mail? I tend to assume it's all just spam. Then again, I tend to assume that of my email, too. What was the last year we had a communications system that had more signal than noise? It seems to have been a while.

Re:ring ring

[perpenso]

]The real problem is that Sony couldn't be arsed to register the domain names using a working e-mail address that actually goes to the person at Sony who is responsible for such a thing.

Not quite, it should be a special purpose email like domain_registration@sony.com rather than an employee email. However the special purpose email should forward to those responsible, involved or overseeing the particular thing. The special purpose email should not be something that someone is supposed to log in to.

Re:Sony playing catchup with Microsoft

[bad_fx]

How on earth do you figure this is a "blow" in the console war? Are you suggest that Microsoft was somehow behind this? Or is everything that gets reported on and is related to Playstation\Xbox now some sort of insidious plot to discredit one or the other console?

In reality it sounds like pure incompetence at Sony (and the same in the story you link about Microsoft) and I think when many people are affected by this sort of thing it's fair enough that it's covered on tech sites. It doesn't have to be part of some 'console war' that you parenthesize with apparent disdain while at the same time perpetuating the idea.

An "unread email address"??

[Rone]

If the address was unread now, it must have been monitored originally.

What are the chances that the original recipients were RIFed at some point to goose the quarterly numbers?

Re:An "unread email address"??

[scsirob]

Must have been "support@sony.com"

Re:oops

[Anonymous Coward]

I want more DRM. It is so nice to see the things I paid for just stop working like that. DRM FTW!!!

"Hilarity Ensues"

[steelfood]

Hilarity Ensues

You keep using that word. I do not think it means what you think it means.

Now, if some domain squatter had taken over the name the moment the domain expired, that would be funny. Giving them 7 weeks is just ... well, sad.

Re:Special email addresses ...

[msauve]

It still requires tracking and making changes. It's easier to change the local email system than a registrar's database, but in either case, updates must be made to be effective. With 10 year registrations available, there's no guarantee that former_group_members@example.com is much better than former_employee@example.com, especially in fast moving industries. If company X acquires company Y, dns@y.com is apt to be forgotten, too.

You're suggesting a tactical solution to a process issue. Better to have the responsible group track and update necessary renewals on a regular basis, instead of depending on notifications from external parties being received.

Yet another reason to insist on software freedom

[jbn-o]

Early Tuesday, gamers woke up to find out that they couldn't log in to any Sony Online Entertainment games--no Everquest, no Planetside 2, none of them.

Could the users have used another server to connect with each other? Or is this a case of DRM ("Digital Restrictions Management", when properly viewed from the perspective of its effect on the users) and, more generally, nonfree software restricting users from running the games with other people?

Re:ring ring

[Anonymous Coward]

According to this article:

On Twitter, however, SOE President John Smedley suggested that the company had failed to pay its website bills.

"The payment notifications went to a junk email box," Smedley tweeted, adding, "Someone left and it got caught in the replacements junk filter. Simple as that. Embarrassing as that. No point dodging."

"DNS problems could take up to 48 hours to resolve," he wrote, adding, "We are really really sorry on this one folks. Embarrassing and preventable. We screwed up."

Also human

[Sycraft-fu]

Anyone on Slashdot who gets smugly superior about this and how "stupid companies are" is just being a hypocrite. We have ALL forgotten things in our lives. We've all forgotten an event we were supposed to be at, a bill we were supposed to pay, something we were supposed to bring with us. It happens.

What's more, everyone has been in a situation where something didn't happen because they, and everyone else, assumed someone else was going to deal with it. You don't go and check on everything that ever happens around you or involving you, you mentally categorize things you are and are not responsible for and ignore the latter.

So ya, companies, which are made up of people, can fuck up too. It's amusing, but perfectly normal.

Re:Also human

[Anonymous Coward]

Thats why you build in redundancies.

Companies are STUPID because they've gotten hooked on the idea of "employee efficiency" to the point that employee efficiency is being negatively impacted. In the past, when a mistake was made, you could easily nail multiple employees simply because they were supposed to be watching/covering one another. If one (or more) screwed up, it meant the others weren't doing their job so they all got punished. It cost a lot more in payroll, but it made sure the job got done, on time, correctly (as far as procedures were concerned). Nowadays, GM can't even find ANYONE to pin the blame on for the ignition switch recalls.

So yeah, companies can fuck up too. But when you can't even find someone within the company you can point to say "that person is the one who fucked up", what does that say about the company?

Re:Also human

[Opportunist]

This. A billion times this.

Corporations are stupid for simply assuming that people are automatons. You come to work and do it flawlessly, always following the ISO 9001 standard. Yeah. Sure. And monkeys fly out of my butt.

People are people and people are making mistakes. Always. Every single day. Anyone in security learns that VERY quickly. And he also learns quickly that you cannot trust humans to be flawless. Not because people are stupid but because people are NOT automatons and make mistakes. Yes, even (actually, especially) if doing the same job for ages. Show me a person who makes no mistakes and I show you a person who does no work!

Security is FINALLY starting to get wise and build systems that are tolerant of human error. Let's see how long it takes 'til the rest of the system catches on.

Re:Also human

[Sockatume]

The whole point of having a corporation (or any other sort of team for that matter) is that you find ways to be less failure-prone than you are as individuals. You have to do this to offset the fact that a failure of the group affects every member - the cost is multiplied.

come on...

[Charliemopps]

come on guys.. There's lots of reasons to hate on SOE. Hell, I haven't bought an SOE product in 10yrs because of the Foglok fiasco... I was actually banned from their forums for a few months back in the day for suggesting they didnt exist, only later find out I was right. The title of the freek'n thread to announce the disappointment was "CharlieMopps was right, not a troll, there are no frogloks!!!" (paraphrased, the threads been deleted for some time now) If you don't know what thats about you've no reason to hate on SOE. Ok ok, I'm just tryning to point out I have no love for them...

Anyways... Managing a domain is a pain in the ass. I've worked in a few places with large website, I'm sure a few of you have. Maintaining that domain registration is deceptively difficult. Think about it as if you were the one in charge of it.

You tell your staff "Register out domain!"
They go off and come back "well, it appears we can register it for anywhere from 1yr to 5yrs, which you would like?"
You say "5yrs of course!"
They tell you "how would you like it billed? We can pay it one time now... or put it on the company credit card?"
You say "The company card of course! It will renew!"
***5yrs later your site goes down***
How could this happen?!?! An in-depth review shows that the entire team you assigned to take care of that task has either moved on or transfered elsewhere in the company. Doh! Even worse, credit cards only last for 5yrs before they are canceled and reissued, you were doomed from the start. All the phone numbers you gave them were moved, the people gone, and those that answered barely knew what a domain was in the first place. You're biggest fault was apparently setting the renewal so far out. If you'd set it for 1yr at least you could have a repeating process for people to get use to as newhires rolled in and out.

But wait! There's a "contracts" department that should have cought this!
Well "contracts" kind of sorts things in order of importance by cost and that domain registration cost what? $20? So that out it between free Twinkie Friday and the new coffee pot... not really on their radar.

As many times as I've seen this happens it still baffles me to this day why there isn't a service that went something like "$10k per year and you'll never have to worry about any of your domains... ever... pay us, we take care of it"

anyways, whatever... point is, it's not as simple as it appears on the surface.

As we suspected

[HangingChad]

SOE's president, John Smedley, has admitted that the expiration notices were being sent to an "unread email" address.

The same one used for customer service inquiries.

Happens all the time...

[RoloDMonkey]

I have been doing web work for a decade, and I can tell you this happens all the time. In fact, older employees in marketing have told me horror stories about 800 numbers and mailing addresses that were never set up, misprinted, or never updated.

I always tell clients that they should set up emails that describe the job/function, like marketing@example.com and webmaster@example.com, and make sure that those emails go to a distribution list that goes to at least two people.

You wouldn't believe how often critical accounts and webforms are only accessible with the email addresses of Sally the Secretary or William the Webmaster. When they leave, no one knows there is a problem, until it is a big problem.

It's still not working!!

[NotDrWho]

I'm still not able to log on to Star Wars Galaxies.

Meh...

[SanDogWeps]

EVE Online still holds the record for most epic fallout from not paying a bill...