Slashdot Mirror
GitLab Says It Found Lost Data On a Staging Server (theregister.co.uk)
GitLab.com, the wannabe GitHub alternative that went down hard earlier this week and reported data loss, has said that some data is gone but that its services are now operational again. From a report The Register: The incident did not result in Git repos disappearing. Which may be why the company's PR reps characterised the lost data as "peripheral metadata that was written during a 6-hour window". But in a prose account of the incident, GitLab says "issues, merge requests, users, comments, snippets, etc" were lost. The Register imagines many developers may not be entirely happy with those data types being considered peripheral to their efforts. GitLab's PR flaks added that the incident impacted "less than 1% of our user base." But the firm's incident log says 707 users have lost data. The startup, which has raised over $25 million, added that it lost six hours of data and asserted that the lost doesn't include users' code.
Reputation is ruined forever. Everyone involved will never work in tech again, should kill themselves right now.
Have gnu, will travel.
Or GitLab, even.
The hard part is having a backup plan for your "cloud." Some places make it easy, but some make it VERY hard. Never used gitlab so I can not comment... But if YOU do not have a backup, there are no backups. As Codespaces users found out, and now Gitlab, kinda...
Of course it doesnt include users code - it's GIT for god sake. Developers have the whole repo on their own machine...
You can self host your own gitlab server and handle backup yourself if you want.
To lose more of your data.
"GitLab.com, the wannabe GitHub alternative" ... Uhm, is that really accurate?
It is pitch black. You are likely to be eaten by a grue.
GitLab is actually quite good at it, really.
1. You can get all the wiki and code repo data by git cloning into a backup repository.
2. You can set up a remote mirror that gets automatically updated for the code. I don't think you can do that for the wiki, though.
3. Project admins can download a metadata dump to import in some other gitlab instance (e.g. a local instance of gitlab CE (floss) or EE (paid):
The following items will be exported:
Project and wiki repositories
Project uploads
Project configuration including web hooks and services
Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, and other project entities
4. The data which is not exported (LFS objects, build traces and artifacts, container registry images) can be downloaded in some other way. E.g. LFS is usually cloned along with the git code repos.
Note that (3) **includes** the webhooks data that was not fully recovered.
So, yeah, anyone who lost truly important data in this gitlab.com event was actually just as guilty of not following the "Tao of Backup" properly as gitlab.com's sysadmins.
So they have found the data randomly on a server somewhere.
I'm a good cook. I'm a fantastic eater. - Steven Brust
Obviously, data loss is embarrassing. I think we all appreciate the importance of not only having multiple backups, but testing to ensure that your backups work, and are sufficient to fully restore operations. GitLab is just the latest in a long tradition of sites and services that have found themselves facing the consequences of not regularly testing their recovery plans.
But I do respect their response. They quickly recognized what had happened, and they diagnosed what went wrong with their backups. They did not try to use PR-speak to conceal their mistake -- they publicly copped to it, in plain industry-standard language that their users would understand, and even offered a livestream of their team resolving the issue. I think this has been a masterclass in how to recover from a blunder. I bet you that this is not a mistake GitLab will be repeating anytime soon.
Also, I think it's very fortunate that they're in the git repo business, and presumably users who had data that was affected by the loss still have a copy in their own local repos. Thank god for distributed SCM.
Why the hell would you "self-host" a cloud service?
I switched from using DropBox in the cloud to a FreeNAS file server at home since I rarely access those files over the Internet. Now I don't have to worry about losing my data via the Internet.
Why the hell would you "self-host" a cloud service?
Almost any server can be "cloud service". There are several interesting solutions to the problem "I need to access a Git repository over the net" in "the cloud" or otherwise. For example, I self host because my code is so amazing, I can't risk having anyone see it lest they die from heart attack due to the overwhelming splendor.
True. You've moved it to where you have to worry about losing data in a fire. Of course if you do offsite backups you are fine. The point is that moving from local to cloud or cloud to local is just shifting the potential failure point around. The solution of course is redundancy and multiple locations.
Nice thing about having all these release stages is that they are tested before promoting to the next stage.
Process for updating any stage:
1. copy data from next stage.
2. deploy new code
3. TEST TEST TEST
I would mod you +1 Informative for that. Yes, automated backup of the cloud is key. You can not verify what you do not see.
While we don't self host, we have two independent git repositories - Bitbucket and Github. the probability of both disappearing over night is pretty slim.
Yes, there are questions about how this happened, how an admin was seemingly under a bit of pressure that that happened, the question about non-existent backups and whether they have people with enough Postgres skills, but I was impressed about the way they admitted it. They didn't butt cover, they admitted upfront and point-blank "Yer, we've deleted the production Postgres data directory, our backups don't work, we're seeing what we can salvage elsewhere."
Yes, if you have copies of your production data in staging as a last resort when all else is not good, they can be used as backups. I would imagine they wouldn't want that stomach dropping feeling again..................
The worst offender is Apple's iCloud, IMHzo. Backup your photos onto your own drive: I can offers you hoops and dead-ends. I really feel cloud services should provide easy options.
Jumpstart the tartan drive.
How do people still lose data in a time when so many options available to limit or even prevent it... synchronous or asynchronous replication to off site storage, snapshots, raid 6... we have the technology available to make data loss nearly unheard of... it's relatively easy to plan and implement, and it works... and yet morons everywhere STILL manage to lose data...
Every X months or years someone can find some of the missing gitlabs data on a server somewhere. Just when you thought that was all they would recover, someone finds a few kilobytes of missing gitlabs data on an SD card floating in a sewer.
Being it a Git repository, you don't have to worry too much about your "centralized" hosting provider – Each developer that has cloned a (non-shallow) repository will locally have everything needed to rebuild history were both providers to disappear. Git is a great backup strategy by itself :-)
Of course, forgot to add — this will *not* include comments, issues, the whole social ecosystem built around your code — but anyway, you don't get to backup it if you replicate your project over several different Git-hosting providers.
"pr flacks"
number doubting '"less than 1% of our user base." But the firm's incident log says 707 users have lost data"
Why the negative tone? I am not a coder. I do not use GitLab or GitHub except for an occasional download. However, generally competition is good. Sure this company lost data.. so do many. The real questions are is this indicative of a systemic issue or just a one time occurrence. I just don't see why this level of negativity is being pushed against this company.
Silence is a state of mime.
I read that in the voice of Snagglepuss.
"Private cloud" means you lease a VPS, such as an AWS EC2 instance, and install an application there. It's useful for keeping personal information within your own country.
reduce the urgency in a disaster by making the manner in which you would recover part of your daily routine - to whatever extant possible.
"It's a doughnut stuffed with M&M's. That way when you finish the doughnut, you don't have to eat any M&M's."
Isn't the entire point of the "cloud" being that they take care of that crap for you?
It is a selling point, certainly. I wouldn't say it is the entire point.
Other selling points are:
1. Access to the data from anywhere
2. Collaboration with internal and external users
3. Cross platform availability (device agnostic)
4. Simplified billing / accounting
5. Broader spectrum of tools (example: you could buy just Word for, say, $100 and own that one program or you can get an O365 sub and rent SharePoint, Word, Excel, Outlook, Publisher, Access, Skype, Exchange, PowerBI, OneDrive and a raft of other software for $15/mo)
6. Automatic updates and/or upgrades to new versions
7. Access to enterprise level infrastructure without having to buy it yourself
My eyes reflect the stars and a smile lights up my face.
Am I the only one that read the title thinking the data was recovered?
There is no cloud...it's just someone else's computer. If you're not comfortable with your stuff on someone else's computer, that would be good justification for self-hosting. I have a FreeNAS box at home providing ownCloud, Plex, and some other services, as well as some Git repositories (currently without a web interface). Some of my Git repos (especially my Portage overlay) are at GitLab for public access (used to be at GitHub, but I yanked everything off of there after they became SJW-converged).
Right now, the Git repos live within their own jail and are accessed over SSH. I tried bringing up GitLab on my server when it was running Gentoo, but didn't get very far...ISTR their packaging and install docs being somewhat Ubuntu- or Debian-specific.
20 January 2017: the End of an Error.
That matches my experience. My company offers an offsite, bootable backup solution so if anything bad happens to your server, you just boot the appropriate clone in our cloud and you're back in business. A LOT of our customers get our service when they find out the hard way why *proper* offsite backups are important. Many weren't too concerned about backup and business continuity until something bad happened to them.
AFTER they have a major loss they get serious about making sure it won't happen again.
It could have been far worse, and I imagine GitLab will make damned sure backups and suchlike work properly in future.
John_Chalisque
A company with 25 million VC bucks and customers like IBM, Redhat, and NASA doesn't have a working backup system? Let me guess, everybody at Gitlab is a developer, and the whole thing runs on node.js in Docker containers.
Yep. 3 2 1. https://www.backblaze.com/blog...
Why the hell would you "self-host" a cloud service?
Because in today's modern world, it pays to be fully buzzword-compliant.
I have my personal git repo hosted locally on my LAN, and use Dropbox as a backup source, with a nighly cronjob packing it up and gpg-encrypting it before shipping it off to Dropbox. It's been working great for 5 years now.
Well, I was involved verifying that we were in compliance. Over 100k products and some percentage were software, probably under 5%. A few projects were archived in the company archives. Funny coincidence I was there when the initial procedures were established. I didn't establish them, but I used them to archive a few software projects I was involved with. Bounce back to Y2K and I am requesting source code from several projects, now defunct but quite possibly will existing users. Simple, we will read the source, establish if there was or was not any time related silliness and go back to the main projects and 10's of thousands remaining. Of course you have already guessed it. No files were available. All the storage media provided was blank. I suspects that the dd-equivilent was done backwards. Take the blank storage and dd it to the incoming data. Fun. I doubt if it has ever been corrected as my comments were ignored back then. BTW, I am blissfully retired.
Have you ever had to do a bare metal restore?
If not I suggest you do it. That's part of what got GitLab, they never verified their backups were restorable. If they had they'd have found there was no data there.
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
I self host because my code is so amazing, I can't risk having anyone see it lest they die from heart attack due to the overwhelming splendor.
Best reason ever.
"First they came for the slanderers and i said nothing."
What kind of IT organisation has $25 Million at their disposal, has a core business of looking after developers data and yet doesn't have snapshots and backups on that data? Seriously? In this day and age? Most modern filesystems have snapshot abilities and the ability to export those snapshots, wouldn't you then do an rsync or tape backup as a belt and braces thing? Also, check your backups, have backup monitoring in place, copy the data somewhere else as a DR plan, undertake test restores, copy important data to two locations (i.e. metadata)?
It's a server (or set of servers) where you stage a new release of your site/software before an actual production release - it provides an environment as similar to prod as possible, and the idea is to help test test your release before unleashing it to the world.
From what i gathered from the obscurely worded article, it seems that they tried to restore data from their staging server after their five backup systems failed. Staging servers require production-like data so it is common to keep them somehow synchronized with prod data (a database copy, for example), but it is kinda sad that's the only thing they had left by then.
Do you work or possibly, formerly worked, at Gitlab?
There is a concept called various things, but most often "vendor lock in" ; it may limit your potential market to the idiots in your industry, but if you can get those idiots to accept it, you're on a road to permanent customers, plus they'll send their first born daughter (or son - your choice) round to service you when you want to empty your balls.
Did you never see that big cheese-eating grin on Billy Gates face? Nerd paradise through vendor lock-in.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"