GitHub Service Outage (github.com)

New submitter thebigjeff writes: Beginning at around 7:30pm EST on 1/27/2016, GitHub's core services have been offline. Most repositories and other functionality is inaccessible. The status page is calling it a "significant network disruption." More from The Register: GitHub falls offline, devs worldwide declare today a snow day.

Comments

"7:30 PM"

[Anonymous Coward]

So is that Ulaan Bataar Standard Time, or what?

This is almost as bad as the suits in California thinking that *of course* all its employees in Europe and Asia use PST. Gawdz.

Re:"7:30 PM"

[whipslash]

Updated with correct time zone (EST). Good catch.

Re: "7:30 PM"

[prefec2]

Still they are an international service with customers around the world. Therefore, it would be logical to usw UTC. Especially, as most people know their difference to UTC, but not some freaky timezone in a country far way. Alternatively they could use AOE , but that is only known by scientists as the paper deadline time zone.

Re:"7:30 PM"

[Nadir]

Brussels is in CET (Central European Time)

Re: "7:30 PM"

[loufoque]

also what is this 27th month they speak of?

Re: "7:30 PM"

[loufoque]

Europeans don't put "Standard" in their time zone, because that's meaningless.

Re:"7:30 PM"

[Bert64]

This is a very common problem... I quite often get people online giving me their phone number without a country code, or their address without a country and in 99% of cases they are in the US when they do this. Often the last part of the address is a two letter code for their state (e.g. CA) which could easily be misinterpreted as a country code.

The Internet is a global network, when sending emails or posting data online you should absolutely declare the country if you're giving out a physical address or phone number. Not doing so is akin to giving your website or email address without the TLD.

Re:"7:30 PM"

[Mal-2]

Within North America, we don't use country codes, even when calling internationally. Calls to Canada or the Caribbean are just direct-dialed exactly as if they were in the U.S. Therefore, we don't think about country codes, even if we do transact cross-border business. I couldn't even tell you what the country code for the U.S. is, or if there is just one code for all NANP territories or if there are separate codes per territory. When we do call outside the NANP area, we have to prefix 011 to indicate this, though in writing this is commonly represented simply as + (and people are just expected to know 011 is the replacement).

As for the country of a postal address, you can usually pick that up by the code at the end. In the U.S. we have a ZIP code of five digits or five-plus-four, such as 01234 or 01234-5678. (Yes they can start with zero.) In Canada, they use a six-character code, which I believe always alternates letter and number, like A1A1A1. These come after the state or province.

Re:"7:30 PM"

[Anonymous Coward]

"Within North America, we don't use country codes, even when calling internationally"

Yes you do.

You've reinforced the parent poster's main comment that American's don't know shit about how things work in the rest of the world.

The USA and Canada have the *same* country code. So guess what? You don't need a code to call between them. This also applies to some US protectorates and the Caribbean, but not Mexico, that's +52. See below for a listing.

+1 COUNTRY CODES

        +1 Canada
        +1 United States, including United States territories:
                +1 340 United States Virgin Islands
                +1 670 Northern Mariana Islands
                +1 671 Guam
                +1 684 American Samoa
                +1 787 / 939 Puerto Rico
                +1 808 Hawaii
        +1 Many, but not all, Caribbean nations and some Caribbean Dutch and British Overseas Territories:
                +1 242 Bahamas
                +1 246 Barbados
                +1 264 Anguilla
                +1 268 Antigua and Barbuda
                +1 284 British Virgin Islands
                +1 345 Cayman Islands
                +1 441 Bermuda
                +1 473 Grenada
                +1 649 Turks and Caicos Islands
                +1 664 Montserrat
                +1 721 Sint Maarten
                +1 758 Saint Lucia
                +1 767 Dominica
                +1 784 Saint Vincent and the Grenadines
                +1 809 / 829 / 849 Dominican Republic
                +1 868 Trinidad and Tobago
                +1 869 Saint Kitts and Nevis
                +1 876 Jamaica

Re:"7:30 PM"

[ls671]

Hey Watson! There are several time zones is US.

P.S. Everybody is used to UTC to avoid fuckups in major big iron stuff.

Re:"7:30 PM"

[ls671]

Easter Standard time aka America New York Time which is called EDT (Eastern Daylight Time) in summer UTC-5/UTC-4

Re:"7:30 PM"

[RabidReindeer]

Except that at the moment, they're not on DST (EDT), they're on EST.

UTC is useful, but I'll settle for the name of the locale. It's not like people don't know where New York is.

Re:"7:30 PM"

[RabidReindeer]

People use the US country code all the time and have for decades. It's the "1" in "1-800" numbers.

But since the US does so much intra-quadrasphere calling where an area code is sufficient, they don't realize it's a country code and just assume it's part of the "800" system.

Re:"7:30 PM"

[OzPeter]

Updated with correct time zone (EST). Good catch.

Who are you and what did you do with the /. editors* ????

*And by editors I mean badly written Perl scripts that don't even have a spell check module installed.

Re: "7:30 PM"

[Anonymous Coward]

"Standard" is super meaningful.

EST = Eastern standard time. New York is on this right now, but when daylight savings begins, it won't be.

EDT = Eastern Daylight Time. This is used in America during the summer on the east coast.

Re:"7:30 PM"

[johnw]

Within North America, we don't use country codes, even when calling internationally.

Good trick if you can manage it. What do you use - owls?

though in writing this is commonly represented simply as + (and people are just expected to know 011 is the replacement)

This is the standard way of representing "code to go international", the point being that it varies from country to country.

I can give my telephone number as +44 1491 NNNNNN

and then anywhere in the world, anyone knows to dial their international access code (in the UK it's 00; it used to be 010), followed by the rest of the number.

FYI, the international code for North America is "1".

Re:"7:30 PM"

[johnw]

It's been many, many years since the +1 has been mandatory for a large portion of the US

It's still mandatory if you're calling from outside North America - same as everywhere else on the planet.

Re: "7:30 PM"

[mark-t]

It's not meaningless... it differentiates between daylight savings time and not. EST during daylight savings time is EDT, so the states that use daylight savings time in that time zone switch their time zone to EDT for the summer, while those that do not use it remain on EST.

Re:"7:30 PM"

[TheRaven64]

Which would be equally unhelpful for readers that are not familiar with UTC

Fortunately, for now, most readers of Slashdot are from Earth. Every time zone is expressed as an offset from UTC. Given a time in UTC, it is easy to work out what it is in your own time zone. Given a time in any other time zone, you do the conversion by first looking up the UTC offset of the reported time zone, subtracting it, and then adding your own UTC offset.

Re: "7:30 PM"

[loufoque]

Right, and that's extremely confusing because elsewhere in the world S means Summer.
WET/WEST, GMT/BST, CET/CEST, etc.

There is nothing "standard" about a time zone, so the deduction that S refers to summer time makes more sense.

Re:"7:30 PM"

[KingMotley]

FYI, the international code for North America is "1".

So when people say America is #1, they are just static a well known fact.

Re:"7:30 PM"

[KingMotley]

Yay, autocorrect.. stating a well known fact.

Re:"7:30 PM"

[Mal-2]

+1 and 1 are not the same, just as +52 and 52 are not. We don't write +1-xxx-xxx-xxxx because that would mean actually dialing 011-1-xxx-xxx-xxxx. +52 really means 011-52. 1 was always billed as "the long distance flag" so to speak, until area codes became mandatory for local dialing as well due to overlays.

Re:"7:30 PM"

[Mal-2]

Within North America, we don't use country codes, even when calling internationally.

Good trick if you can manage it. What do you use - owls?

We use area codes, and within NANP territory, these are country-agnostic. This is convenient in some ways, but also disastrous in others because someone can give me a number in the Bahamas where I'll get charged $2.00 a minute for the call, and it won't look any different from a call to another state or even within the same state. There is absolutely no rhyme or reason to area code assignments *except* for the Caribbean, where they were generated based on three-letter country codes, and some very old assignments where they were given low digits and a 1 in the middle to minimize dialing time on rotary phones (like 212 for NYC and 213 for Los Angeles) -- but that doesn't make them look any different. A call to Sint Maarten and a call to Seattle and a call to Toronto are handled identically.

It may be unique to North America, but our numbering plan spans multiple countries as if they were a single entity. Thus it is not ingrained that calling across a border always means the use of international calling procedures.

Re:"7:30 PM"

[Bert64]

So you're familiar with the postal code system in every country in the world?

Sometimes slashdot is slow

[NotInHere]

but it wasn't long ago (1 hour) since I found out about the outage, by seeing an unicorn. Congrats slashdot for being fast.

Must be the OS/2 Source Code

[martiniturbide]

...all the OS/2 related apps source code that I have uploaded.. people are just spamming github to get it. https://github.com/os2world

MMMMMM.. how about that mother fucking cloud?

[Rinikusu]

maybe they should've backed up their cloud in another cloud. Cloud.

Re:MMMMMM.. how about that mother fucking cloud?

[sims 2]

I hear Google Actual Cloud is good.

Re:MMMMMM.. how about that mother fucking cloud?

[Anonymous Coward]

This cloud precipitated a failure.

Re:MMMMMM.. how about that mother fucking cloud?

[BitZtream]

They seemed to have done so for their billing platform which had no problem charging me last night at 7:58pm

Back up now

[the_povinator]

aaand, it's back up

did github backup

[turkeydance]

or just back up?

Re:did github backup

[whipslash]

Looks like its back up. So probably both.

Decentralized source control

[manu0601]

The selling point of git was to be a decentralized source control system.

It is interesting to see people telling about a snow day while they have a tool that do not require a central repository

Re:Decentralized source control

[Anonymous Coward]

I came here to say exactly this. Then I realised that there's more to github, such as the issue trackers etc, than just git itself.

Re:Decentralized source control

[sunderland56]

Even if you are using a source control system that *does* require a central repository to be up: since when did the inability to check in prevent you from writing and debugging code? If interacting with git/svn/clearcase/etc. is more than 0.1% of your work day, maybe you're not doing it right.

If source control being inaccessible means you get the day off.... let's just say that ClearCase users would be extremely happy.

Re:Decentralized source control

[SoftwareArtist]

Github is much much more than version control. It's also bug tracking, feature tracking, discussions, web hosting, wiki, release management, etc. When all that goes down, you can still write code, but you can't communicate with the other devs anymore.

Re:Decentralized source control

[sunderland56]

So: no incoming bugs or feature requests, no merging other people's code, nobody pinging you every 5 minutes? Around here that's called "a day where I can be productive".

Re:Decentralized source control

[XXeR]

If source control being inaccessible means you get the day off.... let's just say that ClearCase users would be extremely happy.

As someone who used to be a ClearCase/MultiSite admin in a former life, I have to say that if that system went fully offline for anything more than a few minutes, then your admin isn't doing it right.

Re:Decentralized source control

[Dutch Gun]

I know it's fun to be snarky about the fallibility of the cloud at times like these, but in fairness, I think one has to measure these unexpected outages against the productivity gains of having a convenient centralized point to synchronize your project online, especially for historically decentralized teams like your typical open source projects.

The notion that "git is decentralized" is obviously tempered against the requirement to synchronize everyone's repositories, right? Still, I agree... the whole "github is down, I can't code today" is an even weaker excuse than something like "it's okay if I'm goofing off - I'm compiling." One of the benefits of git (and Mercurial as well, which is actually my system of choice) is that it's trivial to make a local branch and start working on some new feature. If you're working on a project, then by definition you have an entire copy of the repository locally - it's not like you need to connect to github just to see your code or check in changes locally. Even if you can't see your bug/todo list, that just means it's a great time to make a branch and start some other little project, like doing some refactoring or code cleanup - or even, heaven forbid, some documentation.

Re:Decentralized source control

[Todd Knarr]

Not all development work involves solitary coding. How do you get the latest changes from a co-worker when you can't access the repository you both normally push changes to and his personal machine with his copy of the repository doesn't accept incoming connections (and neither does yours, so you can't have him push the changes to you)? How do you access the branch you didn't know you needed until now which isn't in your local copy? How do you get that refactoring a colleague just committed and pushed before the outage that you need to have because your part of the work's predicated on it? How do you get anything into the build process when the build process pulls from the repository that's offline?

All of those can be worked around, but amusingly what you'd need to do is almost exactly what you'd need to do with a non-distributed version control system.

Re:Decentralized source control

[AuMatar]

As a user of source control in general- if you need an admin for it, you're doing it wrong.

Re: Decentralized source control

[prefec2]

There are also decentralized ticket systems. Even though you should know enough about your tickets and tasks to have something to do for one day ;-) And if all fails, update the documentation.

Re:Decentralized source control

[Richard_at_work]

It is decentralised, I was still able to commit changes to my repo and carry on as normal. What I couldnt do was use GitHub.

GitHub has value adds which make it a nice thing to use - its an off site repo for backup, it has a nice PR and issue handling system, it has nice metrics, it has commit hooks, it acts as a good point for CI service to integrate with automatically (alternatives being you either have to handle CI locally, manually push changes to a CI repo, or expose a git repo somehow so a CI service can grab checkins and build them).

So I couldn't push my changes to GitHub and my CI service didn't run new commits for a few hours. Not to worry, its already caught up with the back log.

Do not confuse GitHub with git - the two are entirely different. GitHub could use CVS and still have all the value adds, it would just use a shit source code management system.

Re:Decentralized source control

[Actually, I do RTFA]

That's why I have an office I can go to. You know, for all the working with coworkers,

Re:Decentralized source control

[Actually, I do RTFA]

Or , you could export the patches, move them to another machine, and import them. Exacly what would happen automatically wiht push/pull. But it still will have the same information as far as changesets/etc. so it'll seemlessly work once its up.

Re:Decentralized source control

[DarkTempes]

Isn't that why the enterprise version of GitHub exists for locally hosting the service? Isn't that how GitHub makes money and subsidizes the free services?
And you're putting up some pretty big barriers by saying neither user's box allows network shares or an sshd...

Now one big issue with github going down isn't it stopping programmers from writing code but preventing some people from deploying code.

Node.js npm and Rust crates package managers sometimes point to github repos for packages...

Re: Decentralized source control

[bill_mcgonigle]

And if all fails, update the documentation.

Github mostly hosts open source projects.

</snark>

Re:Decentralized source control

[turbidostato]

Or you send patches by e-mail, just like Linus did himself before start coding git.

Re:Decentralized source control

[Antique Geekmeister]

No ability to use the automated build system that pulls updates or source code exports from git tags at github. No configuration publication or web content updates with github based branches. Sharing code between repositories locally is still feasible, but loses the insurance that the code submitted to production has been submitted somewhere accessible to other programmers.

Re:Decentralized source control

[Antique Geekmeister]

I've worked with source control since RCS very shortly after RCS was first published in 1982. If you don't have an admin, to at least ensure backups, consistent merging practices, and cleanups when someone stores huge binaries accidentally, your source control is in real danger. I'm afraid those are all typically human usage issues that require at least a slice of someone's time.

Debugging corrupt content on the back end of the service is its own issue. It's happened with every major source control system I've seen and used in more than 30 years. It can, and does, happen with git and mercurial and the other distributed systems, usually but not always due to human error.

Re: Decentralized source control

[ljw1004]

Say you're on a team of 10-20 devs in a mature product. Your day job is to work through the issue backlog, hopefully checking in fixes for three or so issues a day. You need to pick issues from the issue tracker (GitHub), read comments, maybe interact with the filer to get repros, submit a PR for code review by your peers (on GitHub), send it off for regression testing (via guthub messages), code review about 5 other fixes from other team members (GitHub). Also triage incoming issues (github). That's a heck of a lot more GitHub for someone in a larfe OSS software team, like we have in my team in Microsoft, than just 0.1%.

Developing major new features involves a heck of a lot less GitHub of course! But if your OSS software us widely used, you'll find a few orders of magnitude more issues than major features.

Re:Decentralized source control

[Antique Geekmeister]

Linus and the kernel developers had been using Bitkeeper for free when there was a licensing problem. _That_ was when Linus wrote git, to effectively use the style of good quality merges that Bitkeeper previously provided, in a free software way.

Re:Decentralized source control

[BitZtream]

If you have a clue, that's true.

99.999% of "developers" only know that you click commit and push in SourceTree ... They don't actually have any clue what that does.

Re: Decentralized source control

[prefec2]

True. however, good OSS projects document in the code describing the intent of a module or provide text files with the code (for example the Linux kernel). Others ignore documentation and then they wonder why they cannot really attract new programmers. Anyway, the beauty of git is, you can work offline and still be productive or counter productive (depends on your writing skills).

BTW: Commercial software also has often no documentation, documentation which describes what the code does (which should be obvious by reading the code), or documentation which describes the software in an envisioned version of it in the past. Most valuable would be an architecture description and a spec. of interface semantics (if they are important), like with libc.

Apart of making fun of OSS documentation 'strategies' ;-) , private software is often worse.

Re:Decentralized source control

[SQLGuru]

Shhhh......you're ruining another excuse to swordfight in rolly chairs.

http://xkcd.com/303/

Re:Decentralized source control

[luis_a_espinal]

Don't you have pull requests that need your attention?

Maybe there are pull requests that required attention before the outage, and that are in the works. Maybe you cannot commit, but you have more test cases to refine, more documentation to write. Oh yes, the stuff you have on your backlog, you can work on that too while you are at it. And if you really have to have someone get your changes (because shit, they are urgent), you can pull a worst-case scenario and e-mail your changed files to the appropriate recipient (because I'm certain your e-mail is not dependent on git.)

There is always work to be done. Unless you are complete unorganized and are unable to work independently, without peerage and supervision.

I refactored some of your code to use more inclusive variable names and expect you to respond by January 29th, otherwise we will demand you step down from your role as project maintainer.

CAPTCisHCisC: brothers

Because doubling down on the bat shit crazy makes your argument all the more plausible.

Re:Decentralized source control

[luis_a_espinal]

No ability to use the automated build system that pulls updates or source code exports from git tags at github. No configuration publication or web content updates with github based branches. Sharing code between repositories locally is still feasible, but loses the insurance that the code submitted to production has been submitted somewhere accessible to other programmers.

Email the deltas if you have to. And if you are in a real emergency, you can clone and upload your local copy into bitbucket.

We had a situation like that where we lost our infrastructure a couple of months ago. We couldn't code, we couldn't build, we couldn't do integration testing. Total blackout. Rather than waiting for Ops to bring everything back online, we stopped coding and migrated everything we needed on a different system. We lost a lot of history, but we were back on track.

It was either that on pick our noses during a down time twice as long.

Shit happens. If a team cannot find ways to work around it and make progress, however shaky it might be, I question their abilities. The fallacies of distributed systems applies to people, too.

Re:Decentralized source control

[luis_a_espinal]

As a user of source control in general- if you need an admin for it, you're doing it wrong.

No. You are doing it wrong (inexcusable) , or you are not working on a large scale system (understandable.). As your systems and teams grow in size and complexity, you need gatekeepers. And you need people in charge of doing sysadmin work, backups and stuff, including maintaining and backing up your main repositories.

Beyond a certain team size, it is not cost effective to have developers managing those resources. You want them to develop. Yes, you might have a few developers part-timing on those roles (or even better, have a close relationship with IT support, ala DevOps.)

But you need specialization. This specially true when your organization has source control platforms that cater not only to your projects, but other projects within the organization. Then you need centralized administration.

Re:Decentralized source control

[turbidostato]

"Linus and the kernel developers had been using Bitkeeper for free when there was a licensing problem."

There is a difference between "before" and "immediately before" you seem not to grasp.

Re:Decentralized source control

[AuMatar]

It makes sense to have the boxes all under 1 team, sure. But source control is a pretty fucking simple concept. If you need a team (or even 1 full time admin) to keep it running, your system is fucked up (hi ClearCase users). It should be yet another box owned by IT, not a full time source control admin.

Re:Decentralized source control

[AuMatar]

You need an admin for the server, who's also admin for the other boxes you own. You don't need a specialized admin for source control, like ClearCase did (in fact medium sized installations generally had a small team of admins, and the fucker still sucked preformance wise. RCS was less annoying to use, much less anything newer).

Re:Decentralized source control

[Antique Geekmeister]

Oh, I see. You meant "if you need a dedicated admin whose sole task is source control, you're doing it wrong". Yes, I'd agree with that.

Re:GitHub is probably experiencing a DDoS

[Anonymous Coward]

Probably a side effect of the Code of Conduct. Some SJW admin decided to sed -i s/manual/femual/g and it brought the whole thing down.

Timespan

[Hadlock]

It was between 7:22 and 7:24pm EST when it went down judging by my last commit and the next attempted commit

Not everyone

[SuperKendall]

Some of us use BitBucket you know... no interruptions for me today!

Re:Not everyone

[salimma]

I've been bitten by more BitBucket outages than I've seen GitHub disruptions :p

Re:Not everyone

[eWarz]

Not me, I use github for business and bitbucket for semi-personal. In my 5-6 years of using BB i've never seen it go down. Probably timing (used BB full time from 2011-2014 and part time for 2015-2016), but Github going down seems to be a regular occurrence. I don't mean to criticize Github at all (though BB has a much better freemium setup and less downtime), but it seems like they tend to have a lot more issues then competitors at Bitbucket or other locations.Also note that Atlassian, who owns Bitbucket, also owns sourcetree, jira, confluence, and hip chat. I absolutely hate jira (dog slow) and hip chat has gone waaay down hill since going free, but sourcetree, confluence, and bitbucket still seem to be pretty decent.

Re:Not everyone

[BitZtream]

Why? Because it's down SO often you just stopped using RCS all together?

No, but you'll have one tomorrow, 2 on Saturday, ssh won't work most of Sunday ...

Don't try to pretend BitBucket is reliable. It is hands down the most unreliable service I deal with on a daily basis.

Our company is currently split between BitBucket for private Repos and GitHub for public ... But only for a few more weeks as we finish abandoning the crapfest half assed mess known as BitBucket

Re:Not everyone

[luis_a_espinal]

I've been bitten by more BitBucket outages than I've seen GitHub disruptions :p

Which is why you should have code bases in both (pick your primary in either, and keep the other one as a hot backup.) If a system is worth going through the trouble of constant availability and reliability, this is the only way to go.

Re:How is it a snow-day if you're using Git?

[agm]

Sure, unless today is the end of a sprint and you have 20 developers trying to merge their feature branches into the deployed branch.

Remember that Github wikis are also repos

[Idou]

I thought I was in trouble because I needed to get to my project's wiki for some reference info during the outage and then I remembered that I had done a git pull on the wiki not too long ago.

Re:How is it a snow-day if you're using Git?

[Anonymous Coward]

A proper scrum master would never have such problems.

Project Late

[kamaaina]

Somebody's project was late and needed an excuse so they crashed github.

Re: Fuck GitHub!

[Anonymous Coward]

I find your views intriguing and would like to subscribe to your newsletter.

Re:How is it a snow-day if you're using Git?

[TWX]

it just means you make a few more commits before pushing and pulling

Funny, I got the same suggestions in health class in high school...

Dude, just, no.

[luis_a_espinal]

Github is much much more than version control. It's also bug tracking, feature tracking, discussions, web hosting, wiki, release management, etc.

Not enough for a snow day (unless you are doing it wrong.) Hell, if you are doing it right, you can still be productive during a days-long outage.

When all that goes down, you can still write code, but you can't communicate with the other devs anymore.

Email, IM, skype. I mean, Jebuz on a pony, you make it sound like there a civilization collapse, and that we start using smoke signals and runners carrying clay tables filled with cuneiform.

Re:How is it a snow-day if you're using Git?

[luis_a_espinal]

A proper scrum master would never have such problems.

Wait, scrum masters do source control administration? :)

Welcome to the cloud!

[MoarSauce123]

It is 2016 and the cloud is still broken.