Slashdot Mirror


GitHub Back Online After Service Outage

The Next Web reports that GitHub — home to many open source projects — suffered (and quickly recovered from) a service outage this morning, starting around 14:00 UTC. Other than that the problem "appears to have been caused by its database server," the cause isn't clear.

55 comments

  1. Sponsoring a High Availability solution? by DF5JT · · Score: 1

    We are seriously considering sponsoring github with a free platinum support and maintenance contract for the Linux cluster stack, i.e. DRBD, Heartbeat, Pacemaker, Corosync.

    Would that help?

    1. Re:Sponsoring a High Availability solution? by Anonymous Coward · · Score: 0

      No. It wouldn't prevent outages (nothing will), and it sounds like they recovered quickly. For all you know the source of the outage was a meteor strike. Having said that, it might allow them to reduce some operational expenses so maybe it would be worthwhile.

    2. Re:Sponsoring a High Availability solution? by Anonymous Coward · · Score: 0

      meteor strike is easy to plan for -- put the data in two data centers very far apart -- East/West coast of the US is good enough.

      fat finger / bad database schema changes are much more difficult to catch every time.

      free platinum support won't really help there.

    3. Re:Sponsoring a High Availability solution? by Cammi · · Score: 2

      Actually, it would help if it was a simple database outage. However, it sounds like someone over at Github did something stupid .. .like, deleted the database itself. In that case, nothing can help.

    4. Re:Sponsoring a High Availability solution? by Cammi · · Score: 1

      HA and DR will ... That's their sole purpose.

    5. Re:Sponsoring a High Availability solution? by slashmydots · · Score: 1

      Oh, that's not a fun kind of problem because then if you check the site with downforeveryoneorjustme.com it will show as up even though the site isn't "working."

    6. Re:Sponsoring a High Availability solution? by monkeyhybrid · · Score: 2

      meteor strike is easy to plan for -- put the data in two data centers very far apart -- East/West coast of the US is good enough

      That's fine for most meteors, but what about asteroids that could destroy everything on Earth? That's why you should backup to the cloud, on another planet.

    7. Re:Sponsoring a High Availability solution? by Anonymous Coward · · Score: 2, Funny

      If the meteors go on strike I think we'd have known about it.

    8. Re:Sponsoring a High Availability solution? by Anonymous Coward · · Score: 3, Informative

      In my opion their system already handles everything quite well. I've noticed few outages (this one I only heard about because I was reading slashdot, it's sunday, I'm not even working on my project right now) and, so far, they all have recovered quickly. Moreover, due to the nature of git, the outages could even be longer and more frequent that it would still not be a big issue. The issue tracking system is a bit more critical, but I hardly think anything of great value will be lost due to the problems I've seen so far. From the perspective of one of their paying users, the only change I'd care about is lower price. The system itself is already available enough that I don't consider that an issue at all.

      The most worrisome thing on github so far was when it got hacked. However, having high availability would not help in this situation. That was a one time thing. That seems to have happened once to every major service out there. If it was a regular thing, then it would be a big issue.

    9. Re:Sponsoring a High Availability solution? by Anonymous Coward · · Score: 0

      What if the meteor strikes in places that make many american backbones out of power or destroyed? What if verizon gets hacked? What if the system itself gets hacked? What if a solar flare causes a blackout in the US?

      Outages are simple out of our control.

    10. Re:Sponsoring a High Availability solution? by Mitchell314 · · Score: 4, Funny

      Do what Starbucks does. Use portals to other dimensions. Why do you think they're all over the place?

      --
      I read TFA and all I got was this lousy cookie
    11. Re:Sponsoring a High Availability solution? by dkf · · Score: 2

      The issue tracking system is a bit more critical

      They should put each project's issues in a git repository, so that you can trivially keep them replicated on your own systems.

      --
      "Little does he know, but there is no 'I' in 'Idiot'!"
    12. Re:Sponsoring a High Availability solution? by aflag · · Score: 1

      Then downforeveryoneorjustme.com should consider the HTTP status response. I'm sure it's not 200 when something bad happens.

    13. Re: Sponsoring a High Availability solution? by Anonymous Coward · · Score: 0

      Get rid of your Opstards.

    14. Re:Sponsoring a High Availability solution? by Shimbo · · Score: 4, Funny

      Do what Starbucks does. Use portals to other dimensions.

      I thought that was just to avoid tax.

    15. Re:Sponsoring a High Availability solution? by hobarrera · · Score: 2

      If everything on earth is destroyed, all remaning github users will have no issues with the new uptimes anyway.

  2. I know we like to sleep late by MLBs · · Score: 5, Funny

    But describing 14:00 as morning is a bit too much.

    1. Re:I know we like to sleep late by uberbrainchild · · Score: 1

      Well it was morning in the US

      --
      Anveto
    2. Re:I know we like to sleep late by MLBs · · Score: 1

      It's always morning somewhere

    3. Re:I know we like to sleep late by Anonymous Coward · · Score: 5, Informative

      GitHub is based in San Francisco. The time given is UTC. 14:00 UTC is 7am in San Francisco. "Morning" is a local time reference.

    4. Re:I know we like to sleep late by loufoque · · Score: 2

      It's morning until you've had lunch.

    5. Re:I know we like to sleep late by serviscope_minor · · Score: 1

      It's morning until you've had lunch.

      It's morning until you have your first beer. Morning ends at about 8:45am around here.

      --
      SJW n. One who posts facts.
    6. Re:I know we like to sleep late by ZDroid · · Score: 0

      But describing 14:00 as morning is a bit too much.

      Here's The World Clock. ;)

  3. Github DB failure following govt announcement by Anonymous Coward · · Score: 1

    Not soon after the US government announced it would distribute info via github, github suffers failures. Coincidence? You decide.

    1. Re:Github DB failure following govt announcement by game+kid · · Score: 2

      Maybe Github shut down to...voluntarily add law enforcement backdoors. Yeah. Completely voluntarily. Totally not due to legislation by BSA bribe or anything.

      --
      You can hold down the "B" button for continuous firing.
  4. This is front page? srsly? by dbc · · Score: 4, Insightful

    So, sure, I use github. But... it goes down for a couple of hours and SlashDot panics? This isn't news.

    1. Re: This is front page? srsly? by Anonymous Coward · · Score: 0

      "frontpage" lol

  5. Re:This is front page? srsly? by Anonymous Coward · · Score: 1

    News for nerds, that used to mean something around here. I think this sits pretty squarely within that, even if it doesn't affect you at at all. Certainly better than much of the news on here that's been not even slightly nerd related.

  6. Re:This is front page? srsly? by Anonymous Coward · · Score: 1

    No, this counts as news. It was definitely an "oh, sh!t" moment for many people, and requires an explanation. Only in hindsight is it known that the outage was temporary.

  7. Re:This is front page? srsly? by Antique+Geekmeister · · Score: 3, Interesting

    It's news. I've corporate partners who rely heavily on gitbub.com for access to their open source tools and even for their corporate git repositories, since they're more reliable than almost any in-house source code repository I've dealt with. This especially includes the hand-built, written by the CIO source control systems, that are surprisingly common in startups before they mature. I know companies whose automated software continuous build environments because of this, so it's certainly news.

  8. Centralization is risky by Bing+Tsher+E · · Score: 1

    For years, going back to the days when SourceForge (forgery of source code??) was closely associated with Slashdot, I have been nervous of centrally-located sites for massive numbers of projects. Yes, locate resources there as a robust distribution front end. But have an independent presence on the net as well.

    Centralization of something that is otherwise as free-wheeling and independence minded as Open Source Software, just seems contradictory.

    1. Re:Centralization is risky by aflag · · Score: 1

      At first sight centralization looks like a great way to reduce costs, however, there are problems that are likely to make the cost actually rise. I'm unaware of any studies showing when it's good to centralize and when it's bad. It would be a very interesting study to read. While you may need fewer machines to run 20 websites together than you'd need to run them separately, but you'll also need more qualified staff. An avarage 16 years old with a bit of time in his hands and an interest for technology can easily configure a machine capable of handling hundreds of requests per second. That's more than most sites need. As you strive to share more and more resources, you'll need more and more qualified staff, capable of figuring out how to manage the resource sharing. Resource sharing is one of the most difficult problems in computer science. With machine prices being so low, is it really worthy to solve those problems for all but really big, google sized, sites?

    2. Re:Centralization is risky by VortexCortex · · Score: 1

      Centralization of something that is otherwise as free-wheeling and independence minded as Open Source Software, just seems contradictory.

      Yep. I never understood Giant Websites. We give the world a distributed information network with fault tolerance and a self healing structure designed to withstand entire cities disappearing (censorship, nuclear war, etc), and what do folks do? Centralize the shit out of everything. Data Silos?! CLIENT SEVER architecture?! That was the whole point. There IS NO CLIENT on the web. EVERYONE is supposed to be a server. It's a shame that greed has firmly entrenched the essentially prototype protocol IPv4 (which was never meant to be used for this long, hence the 32bit IP addr...) and reinforced the divide between content providers and consumers via NAT.

      This is why we can't have nice things. Long Live The Internet, but Fuck the Web.

    3. Re:Centralization is risky by aflag · · Score: 1

      Some centralization is required, though. There must be at least an index, like google, to find the IPs of relevant servers with the information you need. How would you solve information retrieval problem without a server-client platform. I think it's likely to be the cheapest solution for the problem. That been said, there is really no need for everyone sharing the same email server, for instance. I agree with you that there are more client-server designs than needed, but they are needed. They can also be the simplest technical solution, which is good by itself. A social network, like facebook, for instance. Didn't need to be centralized, and it would be a lot better if it wasn't.

    4. Re:Centralization is risky by Anonymous Coward · · Score: 1

      For years, going back to the days when SourceForge (forgery of source code??) was closely associated with Slashdot

      Back to the days? They're both still owned by the same company (currently Dice Holdings).

    5. Re:Centralization is risky by anyanka · · Score: 1

      Well, funnily, the index (DNS) that allows you to find IPs of relevant servers (and anything else) *is* decentralized. At least for many values of "decentralized".

    6. Re:Centralization is risky by Anonymous Coward · · Score: 0

      Good thing that everybody who clones the repository has all the history, so it's trivial to migrate to another hosting platform or set up your own.

    7. Re:Centralization is risky by Anonymous Coward · · Score: 0

      You're looking at it the wrong way. This isn't about centralization of machines, it's about specialization of people.

      I've maintained and hosted my own VCS, SMTP, HTTP, NNTP, and SSH servers for well over a decade now. I'd never rely on third parties for this stuff, because I don't trust them, and I don't have to trust them. Not only can I admin my own systems, I can understand and hack on all the relevant source code. But most other people can't do this, including supposed "geeks".

      Once upon a time the badge of cool was running your own server, now it's just having an account on some website. Which is sad, because people will never truly understand how these systems work, and how they _should_ work, unless they actually spend time creating and maintaining them themselves.

      It's doubly sad because as the number of people like me dwindle, at some point government and big corporations will start blocking access to my mail server and my web server. My freedom will be taken away because you all are lazy.

      Also, FWIW, one nice thing about a distributed VCS is that nothing is centralized. It's entirely possible to use Github and maintain your own master (or multiple master trees) separately. I doubt many people do this, but it's possible. So Github is slightly less evil this way.

    8. Re:Centralization is risky by aflag · · Score: 1

      Well, it's more delegated than it is decentralized. In the end you still have an authority dictating the rules. Anyway, DNS has a very simple query and a very simple and straightforward answer. However, if you want to find some content on the Internet, the query and the answer are much more sofisticated. When you take into account spam, algorithm improvement and so on. It looks very hard to think of a decentralized index that could possible work and provide better results than google.

  9. Re:This is front page? srsly? by bill_mcgonigle · · Score: 1

    It was down on Thursday for a little while too - if the story were about a pattern, perhaps it would be noteworthy if not newsworthy.

    But, hey, I appreciate big sites being down every once in a while. When my systems have better uptime than those that Amazon runs, it's at least an understandable point of reference for PHB's.

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  10. Re:how the fuck by Anonymous Coward · · Score: 0

    Maybe it was just slow news day.

  11. Re:This is front page? srsly? by greg1104 · · Score: 1

    I've corporate partners who rely heavily on gitbub.com

    gitbub is the best there is at what they do, but what they do best isn't very nice.

  12. Isn't this the point behind git? by Anonymous Coward · · Score: 1

    Isn't the benefit of git that you keep the repository on your local machine, not on a central server like svn? Can't a git project absorb a brief outage?

    1. Re:Isn't this the point behind git? by Oceanplexian · · Score: 1

      As someone working at a small, relatively new startup using a 'hip' stack, unfortunately, no.

      Most new shops heavily tie into Github for code deployment, configuration management, and most importantly, Agile development practices that require continuously querying Github to test code (CI) plus frequent pulls and branching.Github being down basically means development grinds to a halt and startups' ability to run their own stack diminishes.

      Like the Amazon outages, hopefully this serves to teach people that infrastructure isn't just something you can outsource to the 'cloud' and expect to work perfectly without doing your homework.

      The reality is that Github is just a hosting provider and no different than any other hosting provider in the last 20+ years.

  13. Re:This is front page? srsly? by Anonymous Coward · · Score: 0

    A client's very large company pays GitHub for commercial grade service. The outages of varying levels have been frequent and getting in the way of team development efforts. The outages have cost us days of work with many dozens of developers being stuck and unable to work.

  14. Re:This is front page? srsly? by Anonymous Coward · · Score: 0

    Github is down constantly. Enough that if you use it full time for your dev work it will impact you several times a year. They need an ops person badly.

  15. Re:This is front page? srsly? by Anonymous Coward · · Score: 0

    Actually, I've seen github down 4 times already this year when I needed it. And when it went down, there's just basic status that really says, "wait". The last time this happened for me, I waited 5min and gave up... but sure put be behind work half a day (there is such thing as work schedules and NON-work schedules).

    With this high availability around us, I'm really considering going back to a local git or hg. I mean the whole reason for github is availability... for critical projects (I have a paid account). I know some will beg to differ, aka it's for social coding, but there's so many other ways to share code nowadays (e.g. googlecode).

  16. Re:This is front page? srsly? by VortexCortex · · Score: 2

    I just use Git. You know, that DECENTRALIZED source control thing.... yeah, might want to think about using it in a decentralized way, since, well, you know? That's what it is? I mean, unless you mean that all of the systems you're writing code with are collectively less reliable. Seems like PEBKAC to me. Hell, you can comment out one line in a .git config file to enable a post-update hook, and you're pretty much done setting up an "in-house source code repository".

    Put it this way: If any one part of your DISTRIBUTED source control goes down for a few hours, and that's a big deal.... Then you're a fucking idiot.

  17. Re:This is front page? srsly? by serviscope_minor · · Score: 1

    Put it this way: If any one part of your DISTRIBUTED source control goes down for a few hours, and that's a big deal.... Then you're a fucking idiot.

    Or, truly ingenious. I have a paid up github account for work purpouses I wouldn't actually have any idea how to make a github outage become a big deal.

    --
    SJW n. One who posts facts.
  18. Re:This is front page? srsly? by Anonymous Coward · · Score: 0

    Lol. Hi idiot. Let me explain what decentralization means within the context of Git. It lets one person who has one cloned the repository upload it to a new host, and be ready to rock. Any of the other contributors can add it as a remote, then push their changes and you're good to go. This means they have an "in-house source code repository" times the amount of people who have ever contributed to the project.

    However what Antique Geekmeister primarily described was the companies depending on Github for retrieving open source tools, which an in-house repository would not help with... Unless you mean to clone *all* of github? :)

  19. Re:This is front page? srsly? by Antique+Geekmeister · · Score: 1

    Actually, I referred both to open source repos and to corporate sponsored, private repositories which are the _reference_ clones for other git users to update or clone from. this is particularly important for automatic build systems, which should only be _pulling_ changes form the common repository, never publishing changes to that reference repository.

    git does not force this approach, you can switch to other repositories, but it is nonetheless extremely common and just how most people wind up using git.

  20. Re:This is front page? srsly? by Antique+Geekmeister · · Score: 1

    > config file to enable a post-update hook, and you're pretty much done setting up an "in-house source code repository".

    This kind of thing is _precisely_ why many developers,and many IT departments, don't get along well. For example, any developer can instlal sendmail: or Apache or a file server. Running a 24x7 critical high availability service with backup, account management, and user support is a larger task, and the IT department really has to think in those terms if they're skilled.

    github has been a critical central repository for thousands of open source projects, hundreds of which I've had to work with in the last few years. Your personal home repository doesn't do me any good, nor will it collect and merge the changs from other developers.

  21. Re:This is front page? srsly? by Anonymous Coward · · Score: 0

    You can still push/pull from a secondary remote if you like. Doesn't make a jot. If you don't trust github's reliability use github and your own solution.