Slashdot Mirror
GoDaddy Bobbles DST Changeover?
Several readers alerted us to this piece in PC World reporting on concerns that GoDaddy might not be ready for the DST changeover. Some readers, and others, claimed that GoDaddy's servers are not reachable now and are not serving email or web sites; but others see no evidence of this. The article recounts the rather flip response one GoDaddy customer got from their tech support: "As Daylight Savings [sic] does not apply to our servers, since we are on Arizona Time and our time zone does not change, our servers wouldn't update." When IDG News Service contacted GoDaddy they got an altogether more sensible reply.
For international services like domain registrars, switch to UTC already. Running the server on a local timezone will only lead to confusion.
All my internet servers just use UTC. NTP synchronized, naturally.
Oolite: Elite-like game. For Mac, Linux and Windows
I dont understand, all that godaddy does is manage dns, web servers for parking space and basic MX services. How can someone fuck up with this kind of setup? Even if DST patches are off the only problem that i see is with
1. DNS TTLs being incorrect.
2. Your mail showing incorrect time
3. Web server logs (who analyzes these anyway) showing an incorrect time.
How can any or all of these bring down a site. WTF?
Microsoft: "You've got questions. We've got dancing paperclips."
IDG news service got a more sensible reply... which is...? Or is this a telepathic moment?
we will end no whine before its time
The real question is, is the problem DST related, or is it a coincidence?
;-)
Sure, it happened around the day of the change. Sure, they were pretty flip about responding to peoples' questions about their DST change readyness. But is it fair to jump to the conclusion that it [the outage] is because of the new DST rules? It could be that they are incompetent in other ways.
--Xandu
It said GoDaddy has conducted risk analysis on all its systems in preparation for the DST switch, as well as contacted its vendors and received recommended patches.
I wonder if they have done a risk analysis on bad pr and customer support.
Libertarian Leaning Political Discussion Forum.
With much noise and fanfare, Godaddy announced that they were being paid to run MS on its server. So it's no surprise GoDaddy can no longer handle even minor changes like DST. Why is this article here? So we can laugh, and say "told you so" ?
Remember IntellAdmin, offering a free DST patch for Windows 2000? Well, it doesn't work. I installed it on a Win2K system, and the time didn't change to DST. I contacted Intelladmin, and got "workaround instructions" (open clock, change to another time zone, change back, then reset the clock to the correct time.). It only changes to DST the next time you manually set the clock.
So if you deployed this "patch" on your Win2K machines in a corporate environment, the time is going to be wrong when everybody shows up on Monday.
I can confirm that they were down, but it looks like they might be coming back up. Some of my hosts are responding now.
For a bit, the GoDaddy support site mentions "technical difficulty". Godaddy.com
The Internet Storm Center has notes, too: SANS Internet Storm Center
i was meaning to move away from godaddy anyway but when my nameservers quit working this morning i immediately changed them and started the domain transfer.
GoDaddy's forwarding wasn't working for my site this morning, but simply going to my account and reviewing the settings (I changed nothing) seemed to clear that up. It could be coincidence, but who knows.
I have not been able to access my Yahoo! mail account for 7 hours now. Can someone confirm that Yahoo Canada's mail system is also not ready for the change over?
I resell GoDaddy's services, and I can tell you that today sucks.
I have three accounts with them, and the two most recent ones are working just fine.
However, the earliest account (which incidentally hosts many of my clients) is totally off the map today. No websites, no email - I can't even resolve servers that I know are up. It's making today quite hectic - though thankfully, I have a built in "excuse" for the customers - DST.
However, for Godaddy to have dropped the ball like this is not only out of character, but certainly troublesome. I hear 1-and-1.com is not experiencing these issues...
This is an ancient problem, and the industry is learning it the hard way again. Things like time zones are in flux all the time, this state does, this state does not --. It is why UNIX has zic. You just edit the file and use zic to update the files.
Where the problem comes in is that we design applications to use local times on a GMT/network time system. Internet apps should almost never use local time and rely on unchanging GMT0.
I have started putting all new systems in with GMT0 as the system default. It prevents system apps and this Java (yes too, Java needs a patch) to get messed up. If a user wants it to be CST or MST, I set it in their profile.
Amazing. How can any professional data center be caught by this?
Here is how I updated a Linux machine (Debian Woody) for Eastern
:) /etc/localtime /etc/localtime -> /usr/share/zoneinfo/US/Eastern, /root/dst2007 /root/dst2007 /usr/share /root/dst2007/zoneinfo/ . /usr/share/zoneinfo/America/New_York /etc/localtime /etc/localtime | grep 2007
time, if anyone is interested. Some of the information I found on
thar Intraweb was, well, sloppy, and it took some trial-and-error.
The following was exactly what I typed, and it "took" correctly
this morning, with a nice 1-hour gap in the Apache log at 2am. I
don't know if this is the best way, but it worked.
su -
# root password, of course
ls -l
# (mine said:
# in case we have to reverse the procedure below)
mkdir
cd
wget ftp://elsie.nci.nih.gov/pub/tzdata2007c.tar.gz
tar -xzf tzdata2007c.tar.gz
zic -d zoneinfo northamerica
cd
mv zoneinfo zoneinfo.old
mv
ln -sf
zdump -v
# (should include Mar 11 in listing)
godaddy site is still down
Hi all, Both Yahoo mail and GoDaddy-hosted websites were unreachable as of 11am PST until about 12:30pm. Regarding the GoDaddy website, first it looked like the DNS requests timed out. When they went back up, the hosting servers were not working. Tony
http://www.bruguier.com
www.2cups.com website, dns, and email are all not available right now. These are hosted with godaddy.
The core problem looks like a name resolution issue, however with their apparently lax attitude towards patching, who knows what other problems might be going on.
--Colin
I have several domains registered with them using their DNS and I can tell you for a fact it was down for a few hours this afternoon, as were their account control pages. DNS is back up now, haven't tried to log in to my account.
.technomancer
That's what you get for using Microsoft IIS.
At least they didn't fumble or bobble the Fingle dopple!
Comment of the year
I live in Arizona.
/etc/environment...A field that doesn't exist if you're MST-7 w/o DST.
:)
I'm a Unix admin for one of the largest ISPs in the state. We're an AIX house.
I can't speak for our Windows bretheren, but our AIX boxes required absolutely no patching. Our servers calibrate against a UTC source, and the patch IBM offers only affects the optional right-hand portion of $TZ in
GoDaddy's response is entirely sensible.
The only thing not "sensible" here is that you have a bunch of clowns in Congress making decisions with ramifications far beyond their ability to even wrap their brain around.
By the way, our trains run just fine without DST.
GoDaddy's support page still says that they are still having technical difficulties but my hosted site and E-Mail to all domains seems to be back up. Looks like they're trying to complete with Jet Blue for hamhandedness superiority. Now I'm awaiting my letter of apology from Bob...
My GoDaddy service was interrupted today. The godaddy.com website was working but my domains were not. However, the problems seem to have been fixed now.
-d
"Here Lies Philip J. Fry, named for his uncle, to carry on his spirit"
I had about ten domains down today, from about 1:00 PM to about 2:30 PM. When I pinged the name servers, the response was sporadic. I just hope the same thing doesn't happen when the old DST date roles around.
There's nothing wrong with anything - Phillip J. Fry
They've just removed the "We're experiencing technical difficulties" message from their Support page, and indeed their DNS servers appear to be working reliably again. My website / email (not hosted with them) were down due to the DNS problems, but eveything seems fine now.
People should do a SIMPLE check before spreading this kind of FUD
my sites are back up now
I swore it said GoDaddy Boobies Changeover.
James Tiberius Kirk: "Spock, the women on your planet are logical. No other planet in the galaxy can make that claim."
Comment removed based on user account deletion
My godaddy site and webmail was down until about 30 minutes ago. Glad I only have one site left there and moved all the others to 1-and-1 since those sites have been fine all day.
Run and catch, run and catch, the lamb is caught in the blackberry patch.
There is a good chance that the issue was related to DST, but out of GoDaddy's direct control. It could have been caused by one of their downstream providers. Regardless, GoDaddy should have made sure everyone in their 'network' was ready.
http://www.yawpco.com/
Anyone and everyone who stupidly used their name servers essentially got a DOS on their sites and mail. All of my domains are registered through them. Up until about 2 hours ago only 3 of them used their name servers all of my other domains used my web host's name server. Now, none of them use their name servers.
I'm not going to enjoy the kick to the pocket book, but I'm also going to switch to a different registrar. That was just plain fucking stupid. 1) stupid of me to think their name servers would be good to use and 2) more importantly stupid of them to fuck this up.
Now I'm also going to be switching name servers to some place else.
Why on earth would servers have to know about everyone else's timezones? I don't see everyone here in the UK patching their servers because parts of the US have changed their timezone rules. If Azerbaijan changes their timezone rules, they don't start having problems because no-one in the US knows about it.
I had problems all day with my mail (from Godaddy) not working. It seems to be working now.
I don't now about them Winders servers, but as far as I know, their answer is correct for Linux/Unix.
Except for perhaps their email messages or logs having the wrong date, it shouldn't cause any real problems. Any software that cares about times would be using *internal* times (e.g. time_t in C) which doesn't change with DST anyway. If software uses *external* time, it likely broken with any DST change, no matter when it happens. DST changes only change the external representation of the internal value of the clock, not the actual value of the clock. Geez, changing the clock for DST would be stupid.
I've been trying to get onto yahoo mail all day and it craps every time. Something says they didn't apply the DST fix to their old windows boxen!
What's more, the local TV station's website was two hours off, so that really convinced me that time.gov was correct at the time :-)
Did anyone else notice that time.gov was off this morning?
Really, mod down the root post. That is crap.
I just recently migrated a whole office over to the new DST settings. Basically followed the instructions on the Microsoft website, went with my brother and applied a registry hack to all the machines. In fact, if the domain had been set up properly, we could've had it automatically apply the patch to any Win2K box on the network as soon as it boots.
I mean, never mind that the XP boxes and my own Linux systems have been wholly unaffected -- just grab the latest patches and you're good.
But why is this hard for people? Worst case, you can manually go edit the timezone! Microsoft provides a tool called "tzedit", which allows you to set when DST starts, when it ends, and what the delta is.
Don't thank God, thank a doctor!
My domain DNS hosted through GoDaddy, showed the following symptoms.
park27.secureserver.net and park28.secureserver.net where reachable per ping, but did not respond to DNS requests.
Interestingly the two servers have both the same 4 IP addresses. Looks like a round robin load balancing between four servers (hopefully).
It appears back on line at this time.
K
Busy helping non technical users of OpenOffice.org - http://plan-b-for-openoffice.org/
GP is correct, actually. Suppose a user schedules an event for 9 am local time. The server re-maps this to UTC, then stores the UTC time. Then, government comes along and changes the mapping between local time and UTC by rescheduling the start of DST. Now, when the server maps back from UTC, the event ends up as 8 am local time. This is probably not what the user wanted.
you're an idiot.Thus, you're actually better off storing everything in UTC, because then you know what time everything really took place / will take place, in any timezone you care to know it in.
Whether it's done in UTC or local time zones, having local decisions made based on the local time can be problematic when hardware, firmware, and software manufactures don't provide updates.
The DST has changed. I am now taking inventory of hardware that didn't properly make the change. I don't count things like the digital clock in my car, because it doesn't support DST in any way.
Items that have failed and support DST and still failed include my wall Atomic Clocks, and my Linksys Router with the latest (Feb 2007) firmware updated. The manufactures website on both of these items makes absolutely no mention of the DST change as if nothing happened.
I have work-arounds for both failures. It involves turning off the broken DST and changing the time zone one zone to the East. The Linksys router is a non-issue for most folks, but I use the clock for access restrictions, otherwise the school age kid requires lots of prodding to get offline and go to bed. Having his access shutdown eliminates lots of nagging.
Why can't Linksys even admit the issue and state on the website the latest firmware update did not address the issue? I should not have to check to see if the software is working properly. I think I will submit a bug report and see what happens.
The truth shall set you free!
This morning, March 11, some of our Go Daddy services came under significant and sustained distributed denial of service attacks resulting in intermittent disruptions of various services, including shared hosting and email.
Our Internet Security and Network teams immediately invoked counter-measures to respond to these large scale, sophisticated attacks.
After 4-5 hours of intermittent disruptions of various services this morning, including shared hosting and email, the attack was contained.
Our Internet Security and Network teams will continue to analyze and assess the nature of today's attacks and their characteristics to identify additional defense mechanisms that can be used in the ongoing efforts of Internet Security.
Go Daddy has made and will be continuing to make significant investments in our information security infrastructure to protect from these shifting types of attacks.
This in no way related to the switch to Daylight Savings Time, as some have speculated. With regard to DST, Go Daddy has been engaged in preparation and patching and worked closely with our vendors for some time leading up to the DST change. leading up to the DST change.
Neil Warner
Chief Information Security Officer
The Go Daddy Group, Inc.
This is not the first time GoDaddy has been discussed on Slashdot.
MySpace and GoDaddy Shut Down Security Site
GoDaddy Holds Domains Hostage
Alternative Registrars to GoDaddy?
etc.
I've thought to myself many times before that I should move to a different registrar but I typically procrastinate. However this morning all the domains that I have registered were down due to GoDaddy's boo-boo and I've had it with them. A company that can afford to spend $2.4M for a superbowl commercial but can't properly handle a daylight savings change doesn't need my business.
-- I'm old enough to have lived through six different meanings of the word "hacker."
Maybe you live somewhere that your clock ended up locking into CHU, the atomic clock in Canada...
I've never seen a clock that synced from CHU (3.33 MHz and 7.335 MHz)...or from WWV/WWVH (2.5, 5, 10, 15 and 20 MHz), for that matter.
The clocks and watches that feature "atomic time" use the signals from WWVB on 60 KHz.
--
73 de Maggie K3XS
Editor, Phil-Mont Mobile Radio Club Blurb - http://www.phil-mont.org/
Elecraft K2 #1641 -- AOPA 925383 -- ARRL 39280
-=Maggie Leber=-
Quicky quiz: is your cron up to using time zones? If you run a daily job at 02:30 local time, will it be skipped when a DST changeover advances local time by one hour? Or run it twice when 02:30 comes by a second time the same day?
No peeking!
I don't recall the correct answer offhand (I seem to recall most Free OS's got it right), but when a colleagued asked me this question my initial counterquestion was: why would it matter? Haven't you designed in resilience?
Bert Driehuis -- All I asked was a friggin' rotatin' chair. Throw me a bone here, people.
Glad I use GKG.NET. They didn't go down at all. I even called them on Friday and asked if they were up to date with their DST fixes, and they assured me they were.
The day Microsoft creates a product that doesn't suck, it will be known as the Microsoft Vaccuum Cleaner!
My atomic clocked changed just fine. Also, I was always under the impression that the DST function wasn't built into the clock itself, but into the data stream coming from WWV or WWVH (depending on your location.)
The clock manufactures have the option of either reading the DST bit, or using an internal table. Many manufactures have opted for the internal table, especialy those marketing to non-USA markets such as South America. My SkyScan clock did not update. I even forced a reset to see if it didn't catch the update. It still has no idea it's daylight savings time. I switched DST off so it does not become wrong in April and moved timezones one to the East. My clock uses an internal table and does not use the DST bit. It is not mentioned in any of the clock specifications.
The truth shall set you free!
Suppose a conference call event is scheduled on the event server. The main conference happens at 9:00 in _Paris, France_. But the event server is in California.
Your server created events according to local time California. Now the US government changes the DST.
Your events occur an hour off.
Solutions:
1. store events in local event time of where the event actually occurred.
You need to store: event-local-time and event-local-timezone.
Then the time of event calculations would happen on the day of the event, with the current time calculations for each client.
2. store events in UTC time, with any DST adjustments reverted. Also store event-timezone. The DST offset calculations happen on day of event.
3. store event in UTC time. Also store event-timezone. Also store current event-timezone-offset.
Note that both 1, 2, and 3 work, and are equivalent if implemented correctly.
The adjustments/conversions required on event day are:
(1) requires two conversions in a simple implementation: event-local-time to UTC, UTC to client-local-time.
(2) requires DST adjustment for event-time, and UTC to client-local-time conversion.
(3) requires DST adjustment for any event-timezone-offset change (comparing stored event-timezone-offset and calculated offset from event-timezone), and UTC to client-local-time conversion.
I would prefer (3), but any of these are ok.
Broken implementations are:
-1. Store events in server local time. Breaks if actual events are somewhere else, and DST changes. Can break, or need recalculation if server changes timezone
-2. Store events in UTC without storing information needed recalculate DST adjustment. Breaks if DST changes and is not stored.
Pfft. Clearly, GoDaddy knew that the DST changeover would be a problem, so they started faking network problems early on so that they could pretend that it was nothing to do with DST.
If you'd been wearing your tinfoil hat, your brain would be unclouded by the control waves and you'd have been able to figure that out for yourself!
We run a local news site godaddy handles just our email and dns. Our server is elsewhere. We went black for about 3 hours due to DNS. Has anyone heard a guess on how many sites were affected because of this? Hundreds of thousands?
Okay, this thread desperately needs a knowledge-injection.
o rmation
(This is all based on Windows 2000/XP, I understand NT4 and Vista are similar, no promises for 9X.)
Windows stores time zone information in two places. One is what Microsoft calls the "time zone database". This is the collection of all the time zones that Windows knows about. It's kept here:
HKLM\Software\Microsoft\Windows NT\CurrentVersion\Time Zones\
Now, when you set or change the local time zone, Windows copies the appropriate time zone table from the above into:
HKLM\System\CurrentControlSet\Control\TimeZoneInf
The second location is what gets actively used for local time zone calculations.
So anyway, a lot of people (and it sounds like IntellAdmin might be among them) were updating the first location, but not the second. This typically manifests as clocks being wrong during DST (now), but going in and toggling the local time zone fixes it.
This is all described in excruciating detail in MSKB 914387, "How to configure daylight saving time for the United States in 2007".
There's an additional weirdness I've seen. I was patching some Win 2000 SP4 systems today (post-DST switchover). After installing the registry patches, the clock in the Explorer System Tray was still an hour off. But when I double-clicked the clock to bring up the Date/Time Control Panel, the correct time was displayed. Closing and restarting Explorer seemed to fix the problem.
Additional tip: You can cleanly close Explorer without logging off by clicking Start and then Shutdown to get the "Shutdown" dialog box, and then holding down [CTRL]+[ALT]+[SHIFT] and clicking the "Cancel" button.
Hope this helps,
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Could be a Windows issue? Windows and PCs usually store local time, and map back to GMT through time zone info.
Windows expects the hardware clock to be set to local time (mainly because that's all MS-DOS could understand, and so Windows follows suit). Windows does store time internally in UTC. So upon startup, it reads the hardware clock, converts it to UTC, and runs with that.
I still don't see how a DST issue could cause this much havok for GoDaddy. Like others have said, there isn't much they do that should be affected by being an hour off. Of course, programmers are always finding new and interesting ways to screw things up, so who knows?
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
aptitude install tzdata
Ortholattice, you worked way too hard. Before you do a source install in Debian, ALWAYS check the repositories.
aptitude search tzdata
It made the change without even a hiccup. I used the Karen's Power Tools update software to handle the Windows install in the VM... which also worked just fine.
Tech Public Policy stuff
As you can see above in my response, I updated timezones in Debian by simply installing tzdata via automated installation... if I'd waited, tzdata might have been pushed out via automated update. I'll never know, I suppose.
But this could equally well have been done via any of the Debian GUI installation tools that work with apt-get.
Updating Debian for the new DST was a lot less work than installing the Windows fixes... which I had to find, download, install, and run. tzdata was a single command line command... only because I felt like installing it that way... as in tell it to download/install... and forget it. The GUI would actually have been more work.
Any Windoze user who managed to update in time would have been able to do this in Linux with less hassle.
Tech Public Policy stuff
Yes, that's exactly how they blew it. Mod parent up. Thanks.
crazy cabbie?
I've talked to several people this morning whose web sites were down for a few hours over the weekend -- and their DNS and hosting are managed by companies other than GoDaddy. They're mostly smaller regional ISPs rather than national names, but still. A few national donation systems for nonprofits were unresponsive or very sluggish yesterday morning, as well.
At my organization, we applied all of Microsoft's patches to our servers and we STILL have DST issues... our I.T. guys worked all day yesterday to minimize the damage, but there are still a lot of incorrectly-scheduled meetings in our Exchange calendar. That's ridiculous. ("No," they replied, "it's Microsofticulous.")
$nice = $webHosting + $domainNames + $sslCerts
I called goDaddy on Saturday around noon CST. They said that a service provider is having problems with a router. The tech said "we are leaning on them very hard and it should be back up very soon". Took about another hour and things where back to normal.
Go back to the linksys forum and play mod "you troll"
who do you think buys this router familys with kids
and the fact that they could care less about that
shows you what company they truely are
they are also very decietful
EXAMPLE the wrt54g was a pupular selling router
that worked pretty good and had a linux based os
they then lowered the mem changed the os to some gargbage
name and kept the wrt54g name and then hid the fact by having different
versions of it stamped on the bottom of the routers
then they renamed the old linux os version and called it the wrt54gl
when people started to comlain about that
so if anybody read an old review and there out there all over
people will go looking for the wrt54g thinking its that good
linux based os router SEE WHAT IM SAYING KNOW?
I wouldnt trust linksys OR CISCO anymore for anything
the stock firmware is horriffic
DON