GitHub Faces 'Major Service Outage' [Update] (github.com)

Code repository GitHub is facing a major service outage, it said moments ago. Earlier today, the company said it was facing a minor service outage. The downtime comes less than two weeks after it was facing another "minor service outage," which lasted for several hours. The cause for today's disruption remains unknown. The open source company's Twitter feed suggests it has faced several issues over the past few months.

Update: GitHub reports all the services are now operational.

Comments

This is why I use GitLab

[Anonymous Coward]

Seriously, GitLab is a well-run, transparent company. They've had literally one major outage throughout the life of their service, and when it happened, they explained in incredible detail what had gone wrong, how they fixed it, and how they'll prevent in the future. I honestly can't believe they aren't more popular when this is the second big GitHub outage in, what, 8 weeks? Props to GNAA.

Re:This is why I use GitLab

[darkain]

GitLab also doesn't have the growing pains that GitHub faces at scale. When GitLab reaches that scale, either 1) they'll have the same issues, or 2) it'll have already been solved by other projects (like GitHub), and they would have learned from where others implemented fixes and new solutions already.

Re:This is why I use GitLab

[Junta]

For me, GitLab scales great in one respect: I can host a completely isolated instance of gitlab. So if it's an internet wide repository, I pretty much have an obligatory github mirror of it because that's the 'proper' thing to do. However when github goes down as it tends to do, it's not a huge deal because my team can still collaborate on our own instance of gitlab.

Re:This is why I use GitLab

[Gr8Apes]

Yeah, I don't get this. Even if I were to host a public/private github repo, I'd still run and have the authoritative copy in house. It's not like it's a huge effort to run a copy either. (if you're really wanting to do it in house on the cheap, buy a Mac Mini and OSX Server - boom - instant repo in about 10 minutes, otherwise, get a decent Apple/Linux box and setup GitLab or something similar)

Re:This is why I use GitLab

[Actually, I do RTFA]

So, what you're saying is that I get to avoid all the growing pains of GitHub by sticking to GitLab, and free ride on GitHub's growing pains? Plus use a more open solution? What's the downside?

Re:This is why I use GitLab

[Anonymous Coward]

Gitlab have literally had many outages, most of them completely avoidable. Gitlabs most famous outage, which is the one you're probably referring to, was caused by incredible incompetence on pretty much all levels. You'd have to be an idiot to trust them.

They have a shoddy product and a team stacked with incompetence. Their future "preventative measures" completely missed the problems. There are better self hosted git solutions out there, and better hosted solutions, so you really don't need their brand of trash to work outside of Github.

really?

[lastman71]

Maybe is down, because everyone is checking if is really down...

Re:really?

[93 Escort Wagon]

An interesting theory... DDOS a service provider by posting a story on Slashdot saying that service is down. Everyone here starts hammering the service, and - viola!

Re:really?

[grungeman]

No, that was in the old days when websites got "slashdotted" when the were mentioned on Slashdot. Those days are over, Slashdot is not really relevant anymore, which is a shame.

Re:really?

[kelemvor4]

Maybe is down, because everyone is checking if is really down...

Are they checking because it's down or is it down because they are checking. The world may never know.

Re: really?

[Anonymous Coward]

There is a cat in a box that can help with that. Or can it?

So?

[Anonymous Coward]

Nothing is 100% up; why is this even posted?

Re:So?

[Anonymous Coward]

ROTFL!

Probably

[ArchieBunker]

because of systemd issues. Nah in all seriousness its probably the feds installing backdoors.

This is why git sucks.

[Anonymous Coward]

People who do real work trust Microsoft's Team Foundation Server to get their work done instead of stupid Linux git.

Re:This is why git sucks.

[Anonymous Coward]

People who do real work trust Microsoft's Team Foundation Server to get their work done instead of stupid Linux git.

/sarcasm

FTFY

Re:This is why git sucks.

[Anonymous Coward]

I thought this was some kind of funny joke, but I looked it up, and Microsoft really has a product named Team Foundation Server--and it even embraces git! (Of course they extended it with their own version control system that offers additional functionality.) I had to clutch my side to keep from falling off of my chair. This must mean that there are still people out there who believe Microsoft has mended its ways! "C'mon baby, I love you so much, I swear I won't beat you up this time, I know I said that every time for the last three decades, but this time I've really changed--I swear."

Re:This is why git sucks.

[93 Escort Wagon]

Microsoft Team Foundation Server: It never fails under load.

(Because there is no load)

Re:This is why git sucks.

[Anonymous Coward]

Actually, for a long time TFS used it's own version control system, called TFVC. It was much later that MS added git support, and started moving their own stuff into git-based repos.

My company still insists on using TFVC, which is crap next to git (but orders of magnitude better than MS's old system, Visual Source Safe), and I really wish they'd switch. TFVC just has too many little deficiencies.

Re:This is why git sucks.

[Gr8Apes]

VSS? ROTFLMAO. Seriously. Even CVS/Subversion with their known issues with atomic commits seem like warp drive starships next to the VSS travois.

The good and the bad

[Parker Lewis]

The good: Git is decentralized, so everybody can continue working. The bad: several companies are using it as origin.

Re:The good and the bad

[Junta]

Of course, even if that's the case, it's easy enough to change origin or add a new remote if it becomes intolerable. I tend to have an 'internal' remote and a 'public' remote, and 'upstream' so that github going down is no big deal, but when it's up people new to projects can find it and submit pull requests when it can.

Basically opportunistically taking advantage of 'everyone is on github' without the downsides of 'screwed when it goes down'.

Re:The good and the bad

[Zero__Kelvin]

Somebody needs to invent git commands that allow that to be fixed!

Re:The good and the bad

[hord]

Not to mention all the open source projects that rely on GitHub for distribution. I wonder how much Node.js development gets done when GitHub is down.

Other peoples computers

[OzPeter]

That's what the cloud be.

One Basket

[Anonymous Coward]

Yes, let's continue to put all our coding eggs into this one proprietary "cloud" basket. Brilliant!

It's probably DEF CON related

[Anonymous Coward]

Crazy theory, but someone probably hacked something at DEF CON and tried to release it to GitHub.

Excessive Dependency Syndrome

[ErichTheRed]

I've spent a 20+ year career in IT, and have noticed the trend over the last decade has been to rely almost fully on a service you don't control for a key part of operations. Most of my work has been with Azure lately, and Microsoft is shifting from releasing updates in packaged format that they publish and host to just pumping it out onto GitHub. There was a story last year about how tons of web projects were broken when a developer removed a portion of functionality from a JavaScript framework hosted publically -- and it turned out to be a trivial sting-manipulation function.

There is nothing wrong with not reinventing the wheel every time and using other peoples' resources. There is something very wrong with assuming that third parties will keep their systems running 100% of the time and never have bad days. Even Microsoft won't guarantee that Azure regions, of which most are _collections_ of fully redundant data centers, will be fully available all the time and that your applications will never experience downtime. You should never assume a resource will be published in the same place and remain online forever, nor should you rely fully on a third party service to provide your only means of providing the service you're providing.

Re:Excessive Dependency Syndrome

[Anonymous Coward]

Aye, this is why things like having a local nuget cache server are handy. The good thing about git though, is that every dev has a backup of the repo locally

Re:Excessive Dependency Syndrome

[Junta]

I agree with your general sentiment. Software devs cares less and less about good packaging and publishing a manageable low frequency stream of 'releases' for folks to consume, but not uselessly long token release cycles. Part of this is overconfidence in continuous integration, part of it is a misunderstanding of 'Agile', part of it is that on the whole developers are currently a bit spoiled by the way industry worships the skill of software development so they don't have to do things like make lives easier on downstream consumers of their code.

I will say that github is a bit special *as long as you don't abuse it*. Generally, we need *something* where people can reasonably expect to find your code. It needs to be serviceable enough to work, but most critical of all everyone has to agree. It can change as a fad and suddenly we are all back on sourceforge or something tomorrow.

The problem is too many people are using git the *exact same way* as they use centralized VCS. Their 'origin' remote goes down and they are helpless to do any collaborative work. Despite git having excellent capability to create a new remote and moving data to it. Or to even manage it entirely through email. Github is there to facilitate project discovery and providing a standard way for a stray developer to contribute a suggestion or bugfix. Being reliable is a plus, but good usage of git means that you aren't as pissed when it goes down, and your team has recourse to move or augment to another git hosting service, and still be on github and accept contributions there.

Re:Excessive Dependency Syndrome

[Anonymous Coward]

muh cloud services

Re:Excessive Dependency Syndrome

[Gr8Apes]

Depending upon how it's done, they have a backup of the part of the repo they're tracking, nothing more.

Re:Excessive Dependency Syndrome

[Gr8Apes]

Most "Software devs" today are barely aware of anything approaching real large scale code development. Therefore they have never been exposed to the need for proper packaging and maintaining manifests. It's sad state of affairs that the bulk of the profession is barely able to tie their figurative shoelaces.

Re:Excessive Dependency Syndrome

[Junta]

Can in part blame containers and things like ansible/puppet/chef/salt.

They are all useful tools for adequately skilled admins to manage what they put together, but increasingly it's been a crutch of software devs to not bother doing good packaging and release management. So what if a hapless admin is slapped with some inscrutible stack trace or syntax error? So what if you get lazy and you ship a vulnerable system library that got updated by the distro vendor ages ago? The alternative would be spending a few minutes learning how to package and actually keeping in mind the user's perspective as part of your development cycle, and that would be terrible!

As of 14:00 EDT

[Dracos]

The status page reports "Everything operating normally."

How to spot people who shouldn't write code

[Anonymous Coward]

Their project fails to build when GitHub goes down.

That's why you have backups

[maelkum]

You don't host your code on GitHub *only*, right?

You have mirrors on GitLab, Savannah, and a public repository at project's website (say, git.example.org), a local copy on your machine, and - of course - a copy stored safely at an off-site backup location (Tarsnap, S3, hubiC, etc.). I thought so.
The fact that GitHub went down is just a minor inconvenience to you, right?

Build your own GitLab

[tepples]

GitLab scales great in one respect: I can host a completely isolated instance of gitlab.

Case in point: Build your own GitLab appliance with a Raspberry Pi. I wonder if anyone else has built an appliance to run Savane, the SourceForge fork powering the GNU Savannah repository farm.