Slashdot Mirror
GitLab Says It Found Lost Data On a Staging Server (theregister.co.uk)
GitLab.com, the wannabe GitHub alternative that went down hard earlier this week and reported data loss, has said that some data is gone but that its services are now operational again. From a report The Register: The incident did not result in Git repos disappearing. Which may be why the company's PR reps characterised the lost data as "peripheral metadata that was written during a 6-hour window". But in a prose account of the incident, GitLab says "issues, merge requests, users, comments, snippets, etc" were lost. The Register imagines many developers may not be entirely happy with those data types being considered peripheral to their efforts. GitLab's PR flaks added that the incident impacted "less than 1% of our user base." But the firm's incident log says 707 users have lost data. The startup, which has raised over $25 million, added that it lost six hours of data and asserted that the lost doesn't include users' code.
Obviously, data loss is embarrassing. I think we all appreciate the importance of not only having multiple backups, but testing to ensure that your backups work, and are sufficient to fully restore operations. GitLab is just the latest in a long tradition of sites and services that have found themselves facing the consequences of not regularly testing their recovery plans.
But I do respect their response. They quickly recognized what had happened, and they diagnosed what went wrong with their backups. They did not try to use PR-speak to conceal their mistake -- they publicly copped to it, in plain industry-standard language that their users would understand, and even offered a livestream of their team resolving the issue. I think this has been a masterclass in how to recover from a blunder. I bet you that this is not a mistake GitLab will be repeating anytime soon.
Also, I think it's very fortunate that they're in the git repo business, and presumably users who had data that was affected by the loss still have a copy in their own local repos. Thank god for distributed SCM.
GitLab does a bunch of stuff which GitHub doesn't. The most significant for me is the integrated CI, and that you can host your own runners and workspaces on your own infrastructure (or some cloud provider). Compared with Travis or some other CI hook on GitHub, this is vastly more flexible and powerful. I also find the ability to assign people for review, milestones and such on issues and merge requests to be very nice features which GitHub lacks. It is a GitHub clone, but they seem to have taken the lead in implementing more advanced functionality. At work, we're currently looking into a trial of GitLab plus our own multi-platform CI runners as an alternative to GitHub+Travis and internal Jenkins with several hundred jobs. It stands to greatly simplify the amount of failures, admin time and developer time keeping that lot going.