Slashdot Mirror
Sony Forgets To Pay For Domain, Hilarity Ensues
First time accepted submitter Dragoness Eclectic writes Early Tuesday, gamers woke up to find out that they couldn't log in to any Sony Online Entertainment games--no Everquest, no Planetside 2, none of them. Oddly, the forums where company reps might have posted some explanation weren't reachable, either. A bit of journalistic investigation by EQ2Wire came across the explanation: SOE forgot to renew the domain registration on SonyOnline.net, the hidden domain that holds all their nameservers. After 7 weeks of non-payment post-expiration, NetworkSolutions reclaimed the domain, sending all access to Sony's games into an internet black hole. Sony has since paid up. SOE's president, John Smedley, has admitted that the expiration notices were being sent to an "unread email" address.
LOL
Hole in someone's head, maybe - after all, a simple spreadsheet to track something this basic or a reminder in a calendar with alerts with someone assigned to keep an eye on things would take care of things like this. They're lucky it wasn't held hostage...
Is it that hard to pick up a phone?
Wow, giving the company 7 weeks before Network Solutions took the site down? That's going way above & beyond. The average luser like me would be taken down the day of expiration.
Looks like Sony's playing catchup with Microsoft. Honestly, this is one of the lamest blows in the so-called 'console war' to date.
http://crummysocks.com
I went to a training session for our new $50k accounting system. They had forgotten to renew their own license for the training classroom. Took an extra hour to get their tech in there to get it fixed. Yup, should have got up and went home at that point.
sigh
We bought it cause it was industry specific (well focused at least) and by a small company that only did this for 20 years. Next year they are bought by a national company and instead of being 1 of 200 customers now we were 1 of 20000 on a minor product. Not exactly the same experience :/ Naturally, followed a couple years later by a purchase from a multinational software company :(
I programmed videogames for 17 years, worked on all of the Sony consoles except the PS4.
They're dumber then a bag of hammers over there. I'm glad I'm out.
This sort of lapse has happened in every company I've worked in, big and small, when the person formerly responsible for this kind of thing leaves the company and someone else has to pick up their responsibilities. Sloppy, unorganized? You betcha. Also what I've come to expect.
This is why you don't directly use employee email addresses for certain business activities. These activities get their own emails which forward to whoever the responsible person or persons are. Ex. domain_registration@sony.com. Note "forward to", these would not be standalone email addresses that someone has to log in to.
I long for the good ole days when they actually send out paper invoices in envelopes! ;^)
And from the archives:
"In December 1999, Microsoft forgot to renew the domain name Passport.com,
and so rendered its Hotmail service partially crippled. A Linux
programmer, Michael Chaney, paid the $35 fee and promptly handed over
ownership to Microsoft."
It happened again in 2003:
http://www.theregister.co.uk/2...
Will they ever learn? ;^)
I used to work @ sony playstation. Yes I spelled it right. they dont deserve the capitalization as it would show some form of respect..
moving past that,
sony is a bunch of teenagers with money..
what would happen if someone had trolled this action and somehow managed to pick it up before sony pulled their heads out.
Moving forward, this should be an indicator of how much sony staff and its associates care about you. They care about the money, in this instance it wsa not the public out cry, but more an alert that sony was not making any money off their "cool" portal thingy, once it was discovered, validated and assessed for impact, then action was made.
Whats an even bigger shame, if you dont wear a suit they wont pay you enough to live any where reasonably close to the office..
bt i could go on, but wont..
peace be with you all,
If the address was unread now, it must have been monitored originally.
What are the chances that the original recipients were RIFed at some point to goose the quarterly numbers?
And then I realized I never use PSN, but just play games I have discs for on my PS4.
Second Son rocks. It's like Seattle, but more.
What GTA should have done for GTA: Emerald City but they were too chicken to mess with the best!
-- Tigger warning: This post may contain tiggers! --
Well, dinosaurs did die out.
Hilarity Ensues
You keep using that word. I do not think it means what you think it means.
Now, if some domain squatter had taken over the name the moment the domain expired, that would be funny. Giving them 7 weeks is just ... well, sad.
"If a nation expects to be ignorant and free in a state of civilization, it expects what never was and never will be."
Isn't a people problem, or a company problem, or a technical problem. Like the majority of company fuckups it a procedural problem. There is no documented procedures for anyone to follow, and it's a SPOF in a service.
.. abnormal for this company? .. the company that Thumbed their Nose at the iPod because they had the Walkman? .. the company that Thumbed their Nose at the iPhone because they had dial up? .. the company that sets new standards for giving away business like Christmas Gifts ? .. it's no Wonder Steve Jobs loved this company.. wouldn't you? in Steve's position
"SOE's president, John Smedley, has admitted that the expiration notices were being sent to an 'unread email' address"
You mean the e-mail address belonging to the IT department you banished last year? :D
I think this comic is incredibly apropos:
http://www.wastedtalent.ca/nod...
Could the users have used another server to connect with each other? Or is this a case of DRM ("Digital Restrictions Management", when properly viewed from the perspective of its effect on the users) and, more generally, nonfree software restricting users from running the games with other people?
Digital Citizen
I work as a sys admin at a medium sized medical research institute, one of the things I made sure to do was to add nagios scrips to throw alerts for important licenses, certs, and domain names. I'm not sure why an organization as huge as Sony Online would not have added these kinds of checks to whatever monitoring system they are using. Having had this happen to me once nearly a decade ago with a SSL cert I can promise that the 10min of coding to add in a check is much more pleasant than a day of meetings to describe to everyone what went wrong.
Any time I have a domain about to expire, GoDaddy is right on top of things with a phone call; Really good follow-up as well. I guess Network Solutions is more akin the government. I'm sure if one renews for ten years, the IT guy that was there at the beginning has moved on and his email address goes no where, except a honeypot.
Why not just liquidate the company. That's the correct procedure. At least in my country.
I still remember when someone got sued/threatened when MS did the same thing and it killed passport. After calling and unsuccessfully telling them how to fix the problem, he registered the domain, and pointed it back to where it should have been pointed and mailed them saying he would give them back the domain, he just wanted his stuff to work. They unleashed the vicious attack lawyers anyway.
That which is not dead may eternal lie,and in strange aeons even death may die
that's why you don't send it to a single email, you send it to a distribution list.
Anyone on Slashdot who gets smugly superior about this and how "stupid companies are" is just being a hypocrite. We have ALL forgotten things in our lives. We've all forgotten an event we were supposed to be at, a bill we were supposed to pay, something we were supposed to bring with us. It happens.
What's more, everyone has been in a situation where something didn't happen because they, and everyone else, assumed someone else was going to deal with it. You don't go and check on everything that ever happens around you or involving you, you mentally categorize things you are and are not responsible for and ignore the latter.
So ya, companies, which are made up of people, can fuck up too. It's amusing, but perfectly normal.
Not surprising really, but I would partially suspect that it has to do with network solutions being the worst possible domain registrar. Yes, Sony should have renewed the domain, yes they should have updated their info with network solutions, but it would surprise me if that was made very difficult by this being such a terribly outdated and overpriced registrar.
come on guys.. There's lots of reasons to hate on SOE. Hell, I haven't bought an SOE product in 10yrs because of the Foglok fiasco... I was actually banned from their forums for a few months back in the day for suggesting they didnt exist, only later find out I was right. The title of the freek'n thread to announce the disappointment was "CharlieMopps was right, not a troll, there are no frogloks!!!" (paraphrased, the threads been deleted for some time now) If you don't know what thats about you've no reason to hate on SOE. Ok ok, I'm just tryning to point out I have no love for them...
Anyways... Managing a domain is a pain in the ass. I've worked in a few places with large website, I'm sure a few of you have. Maintaining that domain registration is deceptively difficult. Think about it as if you were the one in charge of it.
You tell your staff "Register out domain!"
They go off and come back "well, it appears we can register it for anywhere from 1yr to 5yrs, which you would like?"
You say "5yrs of course!"
They tell you "how would you like it billed? We can pay it one time now... or put it on the company credit card?"
You say "The company card of course! It will renew!"
***5yrs later your site goes down***
How could this happen?!?! An in-depth review shows that the entire team you assigned to take care of that task has either moved on or transfered elsewhere in the company. Doh! Even worse, credit cards only last for 5yrs before they are canceled and reissued, you were doomed from the start. All the phone numbers you gave them were moved, the people gone, and those that answered barely knew what a domain was in the first place. You're biggest fault was apparently setting the renewal so far out. If you'd set it for 1yr at least you could have a repeating process for people to get use to as newhires rolled in and out.
But wait! There's a "contracts" department that should have cought this!
Well "contracts" kind of sorts things in order of importance by cost and that domain registration cost what? $20? So that out it between free Twinkie Friday and the new coffee pot... not really on their radar.
As many times as I've seen this happens it still baffles me to this day why there isn't a service that went something like "$10k per year and you'll never have to worry about any of your domains... ever... pay us, we take care of it"
anyways, whatever... point is, it's not as simple as it appears on the surface.
Could the users have used another server to connect with each other?
Not much of a gamer, I take it? Most, if not all, of the games affected are not peer-to-peer style multiplayer games; they're MMOs. There's no matchmaking servers involved here.
People will pass up steak once a week, for crap every day.
A Phone call wouldnt of hurt? since a non-used email address was to blame!
Did is what they get for outsourcing IT.
Even if they still had some stuff still in house things can get lost in the shuffle
From: Kazuo Hirai
To: John Smedly
Re: SOE Rego
@#$$%^&*()!) (*$%@#$$%^&*()!)(*$%@#$$@ #$$%^&*()!)(*$%% @#$$%^&*()!)(*$%
"No fear. No envy. No meanness." Liam Clancy
With 10 year registrations available, there's no guarantee that former_group_members@example.com is much better than former_employee@example.com, especially in fast moving industries.
Stop thinking in terms of employees, that's the point of this exercise, the email addresses on the distribution list can include functional roles. company_web_site_manager@sony.com, senior_web_admins@sony.com, etc. Basically the slots in the corporate org chart come with an email address based on the function so you don't necessarily have to know who the person in that role is nowadays.
You're suggesting a tactical solution to a process issue. Better to have the responsible group track and update necessary renewals on a regular basis, instead of depending on notifications from external parties being received.
So your calendar server has a list of people rather than your email server, that's not much of a difference.
Another big company that I worked for, curiously also starting with S, had exactly the same problem. With an internal server, so nobody buy the people working there noticed it. Why? I have no idea.
But once you spend a few years working, you notice that the term "professional" only means "doing it for money". Not "doing it professionally". So please, don't think corporations are in any way more efficient than you are, or that they would or could do a better job. And how should they? It's just people doing for money what other people do for you as a favor.
In other words, if you need to get something fixed, ask a friend. Don't hire someone.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
In Germany, you order a domain, and the company you ordered from charges your bank account automatically every month or year, depending on your contract.
So you can "never" forget to pay a domain. Why is this not common outside of Europe?
Fucking billionaires... SMH
They can't bother to keep their (required annually) admin whois contact email up to date, but their phone number is going to be a working one that connects to someone who can resolve the problem?
Is it just my observation, or are there way too many stupid people in the world?
SOE's president, John Smedley, has admitted that the expiration notices were being sent to an "unread email" address.
The same one used for customer service inquiries.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
I bet they fired the person who was responsible for this and no one checked his email.
What's a DNS admin? Do we need him? You, PHP programmer, you now have his job on top of what ever you already do.
Because some of us never forget.
Hello?
I have been doing web work for a decade, and I can tell you this happens all the time. In fact, older employees in marketing have told me horror stories about 800 numbers and mailing addresses that were never set up, misprinted, or never updated.
I always tell clients that they should set up emails that describe the job/function, like marketing@example.com and webmaster@example.com, and make sure that those emails go to a distribution list that goes to at least two people.
You wouldn't believe how often critical accounts and webforms are only accessible with the email addresses of Sally the Secretary or William the Webmaster. When they leave, no one knows there is a problem, until it is a big problem.
Long live the Speaker Bracelet
Rolo D. Monkey
I'm still not able to log on to Star Wars Galaxies.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
There are people left playing Everquest to notice they couldn't log in?
This sort of lapse has happened in every company I've worked in
Hmmm. Have you considered that you might be the problem?
I find nothing funny about not being able to access a big website or to play a game that I subscribe to. I also want to play Vanguard and Wizardry online before Sony discontinues it. I was not able to access the forums to read the latest news regarding the next versions of Everquest.
I do not share the same sense of humor that Dragoness Eclectic shares.
Situation normal, all Sony'd up.
Fascism: An authoritarian and nationalistic right-wing system of government and social organization. See also: NAZI's
and b) my excuse was, "I don't read that email account any more."
Apparently the actual excuse was "Went to my junk filter lol."
[Someone else pointed out that sony.com itself was only renewed 4 days before expiry, and only for two years. What, are you worried about paying too far in advance in case the company decides to stop using the internet and you can't get your $35 back? I mean, fuck.]
Science is all about firing a drunk pig out of a cannon just to see what happens.
EVE Online still holds the record for most epic fallout from not paying a bill...
I had to edit my HOSTS FILE.
But I'm not that guy.
I tried to play Planetside 2 on Tues. The launcher got an error message. Their website down too... Was odd. Reddit to the rescue. Someone posted the domains and IP addresses necessary to add to the hosts file. Worked fine but game pop was way down. Lowest I think I've seen since the recent server merger.
There were other recommendations to switch to Google's DNS servers wich apparently updated much faster after SOE fixed it, but like I really need Google to know every single domain name I connect to.
Ya, John Smedley is a moron.
Jeruvy