Slashdot Mirror
GitLab Says It Found Lost Data On a Staging Server (theregister.co.uk)
GitLab.com, the wannabe GitHub alternative that went down hard earlier this week and reported data loss, has said that some data is gone but that its services are now operational again. From a report The Register: The incident did not result in Git repos disappearing. Which may be why the company's PR reps characterised the lost data as "peripheral metadata that was written during a 6-hour window". But in a prose account of the incident, GitLab says "issues, merge requests, users, comments, snippets, etc" were lost. The Register imagines many developers may not be entirely happy with those data types being considered peripheral to their efforts. GitLab's PR flaks added that the incident impacted "less than 1% of our user base." But the firm's incident log says 707 users have lost data. The startup, which has raised over $25 million, added that it lost six hours of data and asserted that the lost doesn't include users' code.
Have gnu, will travel.
The hard part is having a backup plan for your "cloud." Some places make it easy, but some make it VERY hard. Never used gitlab so I can not comment... But if YOU do not have a backup, there are no backups. As Codespaces users found out, and now Gitlab, kinda...
"GitLab.com, the wannabe GitHub alternative" ... Uhm, is that really accurate?
It is pitch black. You are likely to be eaten by a grue.
GitLab is actually quite good at it, really.
1. You can get all the wiki and code repo data by git cloning into a backup repository.
2. You can set up a remote mirror that gets automatically updated for the code. I don't think you can do that for the wiki, though.
3. Project admins can download a metadata dump to import in some other gitlab instance (e.g. a local instance of gitlab CE (floss) or EE (paid):
The following items will be exported:
Project and wiki repositories
Project uploads
Project configuration including web hooks and services
Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, and other project entities
4. The data which is not exported (LFS objects, build traces and artifacts, container registry images) can be downloaded in some other way. E.g. LFS is usually cloned along with the git code repos.
Note that (3) **includes** the webhooks data that was not fully recovered.
So, yeah, anyone who lost truly important data in this gitlab.com event was actually just as guilty of not following the "Tao of Backup" properly as gitlab.com's sysadmins.
Obviously, data loss is embarrassing. I think we all appreciate the importance of not only having multiple backups, but testing to ensure that your backups work, and are sufficient to fully restore operations. GitLab is just the latest in a long tradition of sites and services that have found themselves facing the consequences of not regularly testing their recovery plans.
But I do respect their response. They quickly recognized what had happened, and they diagnosed what went wrong with their backups. They did not try to use PR-speak to conceal their mistake -- they publicly copped to it, in plain industry-standard language that their users would understand, and even offered a livestream of their team resolving the issue. I think this has been a masterclass in how to recover from a blunder. I bet you that this is not a mistake GitLab will be repeating anytime soon.
Also, I think it's very fortunate that they're in the git repo business, and presumably users who had data that was affected by the loss still have a copy in their own local repos. Thank god for distributed SCM.
Why the hell would you "self-host" a cloud service?
Almost any server can be "cloud service". There are several interesting solutions to the problem "I need to access a Git repository over the net" in "the cloud" or otherwise. For example, I self host because my code is so amazing, I can't risk having anyone see it lest they die from heart attack due to the overwhelming splendor.
Being it a Git repository, you don't have to worry too much about your "centralized" hosting provider – Each developer that has cloned a (non-shallow) repository will locally have everything needed to rebuild history were both providers to disappear. Git is a great backup strategy by itself :-)
"pr flacks"
number doubting '"less than 1% of our user base." But the firm's incident log says 707 users have lost data"
Why the negative tone? I am not a coder. I do not use GitLab or GitHub except for an occasional download. However, generally competition is good. Sure this company lost data.. so do many. The real questions are is this indicative of a systemic issue or just a one time occurrence. I just don't see why this level of negativity is being pushed against this company.
Silence is a state of mime.