Slashdot Mirror
GitLab Says It Found Lost Data On a Staging Server (theregister.co.uk)
GitLab.com, the wannabe GitHub alternative that went down hard earlier this week and reported data loss, has said that some data is gone but that its services are now operational again. From a report The Register: The incident did not result in Git repos disappearing. Which may be why the company's PR reps characterised the lost data as "peripheral metadata that was written during a 6-hour window". But in a prose account of the incident, GitLab says "issues, merge requests, users, comments, snippets, etc" were lost. The Register imagines many developers may not be entirely happy with those data types being considered peripheral to their efforts. GitLab's PR flaks added that the incident impacted "less than 1% of our user base." But the firm's incident log says 707 users have lost data. The startup, which has raised over $25 million, added that it lost six hours of data and asserted that the lost doesn't include users' code.
Have gnu, will travel.
The hard part is having a backup plan for your "cloud." Some places make it easy, but some make it VERY hard. Never used gitlab so I can not comment... But if YOU do not have a backup, there are no backups. As Codespaces users found out, and now Gitlab, kinda...
On the other hand, having now made this mistake they are probably not going to make it again. Could be more reliably than companies which by chance have not needed to restore from backup yet.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
To lose more of your data.
Sounds like they restored from a backup. Backups are generally taken once per 24-hours, although PITR on databases is ... interesting, and complex as hell to pull off in the real world (I don't know why; it should be a simple operation, but no database seems to make it as easy as "look here for alternate binary logs and play forward until $TIME").
Data loss of 6 hours of issues, MRs, comments, and the like is ... data loss of 6 hours. It's a lot in aggregate for something with over 70,000 users and 238,000 projects, but not much for one project unless project members spend all day writing hundreds of issues and comments instead of code.
Support my political activism on Patreon.
"GitLab.com, the wannabe GitHub alternative" ... Uhm, is that really accurate?
It is pitch black. You are likely to be eaten by a grue.
GitLab is actually quite good at it, really.
1. You can get all the wiki and code repo data by git cloning into a backup repository.
2. You can set up a remote mirror that gets automatically updated for the code. I don't think you can do that for the wiki, though.
3. Project admins can download a metadata dump to import in some other gitlab instance (e.g. a local instance of gitlab CE (floss) or EE (paid):
The following items will be exported:
Project and wiki repositories
Project uploads
Project configuration including web hooks and services
Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, and other project entities
4. The data which is not exported (LFS objects, build traces and artifacts, container registry images) can be downloaded in some other way. E.g. LFS is usually cloned along with the git code repos.
Note that (3) **includes** the webhooks data that was not fully recovered.
So, yeah, anyone who lost truly important data in this gitlab.com event was actually just as guilty of not following the "Tao of Backup" properly as gitlab.com's sysadmins.
Obviously, data loss is embarrassing. I think we all appreciate the importance of not only having multiple backups, but testing to ensure that your backups work, and are sufficient to fully restore operations. GitLab is just the latest in a long tradition of sites and services that have found themselves facing the consequences of not regularly testing their recovery plans.
But I do respect their response. They quickly recognized what had happened, and they diagnosed what went wrong with their backups. They did not try to use PR-speak to conceal their mistake -- they publicly copped to it, in plain industry-standard language that their users would understand, and even offered a livestream of their team resolving the issue. I think this has been a masterclass in how to recover from a blunder. I bet you that this is not a mistake GitLab will be repeating anytime soon.
Also, I think it's very fortunate that they're in the git repo business, and presumably users who had data that was affected by the loss still have a copy in their own local repos. Thank god for distributed SCM.
Why the hell would you "self-host" a cloud service?
Almost any server can be "cloud service". There are several interesting solutions to the problem "I need to access a Git repository over the net" in "the cloud" or otherwise. For example, I self host because my code is so amazing, I can't risk having anyone see it lest they die from heart attack due to the overwhelming splendor.
Restored from a manual backup and admin happened to take. All the automated systems failed... That said, nothing promotes fire safety like a good fire.
Being it a Git repository, you don't have to worry too much about your "centralized" hosting provider – Each developer that has cloned a (non-shallow) repository will locally have everything needed to rebuild history were both providers to disappear. Git is a great backup strategy by itself :-)
Of course, forgot to add — this will *not* include comments, issues, the whole social ecosystem built around your code — but anyway, you don't get to backup it if you replicate your project over several different Git-hosting providers.
"pr flacks"
number doubting '"less than 1% of our user base." But the firm's incident log says 707 users have lost data"
Why the negative tone? I am not a coder. I do not use GitLab or GitHub except for an occasional download. However, generally competition is good. Sure this company lost data.. so do many. The real questions are is this indicative of a systemic issue or just a one time occurrence. I just don't see why this level of negativity is being pushed against this company.
Silence is a state of mime.
That matches my experience. My company offers an offsite, bootable backup solution so if anything bad happens to your server, you just boot the appropriate clone in our cloud and you're back in business. A LOT of our customers get our service when they find out the hard way why *proper* offsite backups are important. Many weren't too concerned about backup and business continuity until something bad happened to them.
AFTER they have a major loss they get serious about making sure it won't happen again.
Yep. 3 2 1. https://www.backblaze.com/blog...
It's a server (or set of servers) where you stage a new release of your site/software before an actual production release - it provides an environment as similar to prod as possible, and the idea is to help test test your release before unleashing it to the world.