Slashdot Mirror
GitLab Says It Found Lost Data On a Staging Server (theregister.co.uk)
GitLab.com, the wannabe GitHub alternative that went down hard earlier this week and reported data loss, has said that some data is gone but that its services are now operational again. From a report The Register: The incident did not result in Git repos disappearing. Which may be why the company's PR reps characterised the lost data as "peripheral metadata that was written during a 6-hour window". But in a prose account of the incident, GitLab says "issues, merge requests, users, comments, snippets, etc" were lost. The Register imagines many developers may not be entirely happy with those data types being considered peripheral to their efforts. GitLab's PR flaks added that the incident impacted "less than 1% of our user base." But the firm's incident log says 707 users have lost data. The startup, which has raised over $25 million, added that it lost six hours of data and asserted that the lost doesn't include users' code.
It is.
die by the cloud. I don't have too much sympathy for anyone that was relying solely on GitHub to keep their code safe and secure.
Reputation is ruined forever. Everyone involved will never work in tech again, should kill themselves right now.
Have gnu, will travel.
Of course it doesnt include users code - it's GIT for god sake. Developers have the whole repo on their own machine...
Staging Server = backup strategy.
nice, i'll have to remember that one.
To lose more of your data.
"GitLab.com, the wannabe GitHub alternative" ... Uhm, is that really accurate?
It is pitch black. You are likely to be eaten by a grue.
So they have found the data randomly on a server somewhere.
I'm a good cook. I'm a fantastic eater. - Steven Brust
Obviously, data loss is embarrassing. I think we all appreciate the importance of not only having multiple backups, but testing to ensure that your backups work, and are sufficient to fully restore operations. GitLab is just the latest in a long tradition of sites and services that have found themselves facing the consequences of not regularly testing their recovery plans.
But I do respect their response. They quickly recognized what had happened, and they diagnosed what went wrong with their backups. They did not try to use PR-speak to conceal their mistake -- they publicly copped to it, in plain industry-standard language that their users would understand, and even offered a livestream of their team resolving the issue. I think this has been a masterclass in how to recover from a blunder. I bet you that this is not a mistake GitLab will be repeating anytime soon.
Also, I think it's very fortunate that they're in the git repo business, and presumably users who had data that was affected by the loss still have a copy in their own local repos. Thank god for distributed SCM.
No worries mates,
A part-time intern dev found a week old backup on an SSD he stole and lost in the sofa.
Toodles til next time,
GitLab
> developers may not be entirely happy with those data types being considered peripheral [...]
I use untyped lambda calculus, you insensitive clod!
[Meta: am I the only who cringes when a technical term becomes buzzspeak? Am I the only one to feel like punching some idiot?]
Nice thing about having all these release stages is that they are tested before promoting to the next stage.
Process for updating any stage:
1. copy data from next stage.
2. deploy new code
3. TEST TEST TEST
Yes, there are questions about how this happened, how an admin was seemingly under a bit of pressure that that happened, the question about non-existent backups and whether they have people with enough Postgres skills, but I was impressed about the way they admitted it. They didn't butt cover, they admitted upfront and point-blank "Yer, we've deleted the production Postgres data directory, our backups don't work, we're seeing what we can salvage elsewhere."
Yes, if you have copies of your production data in staging as a last resort when all else is not good, they can be used as backups. I would imagine they wouldn't want that stomach dropping feeling again..................
How do people still lose data in a time when so many options available to limit or even prevent it... synchronous or asynchronous replication to off site storage, snapshots, raid 6... we have the technology available to make data loss nearly unheard of... it's relatively easy to plan and implement, and it works... and yet morons everywhere STILL manage to lose data...
Every X months or years someone can find some of the missing gitlabs data on a server somewhere. Just when you thought that was all they would recover, someone finds a few kilobytes of missing gitlabs data on an SD card floating in a sewer.
"pr flacks"
number doubting '"less than 1% of our user base." But the firm's incident log says 707 users have lost data"
Why the negative tone? I am not a coder. I do not use GitLab or GitHub except for an occasional download. However, generally competition is good. Sure this company lost data.. so do many. The real questions are is this indicative of a systemic issue or just a one time occurrence. I just don't see why this level of negativity is being pushed against this company.
Silence is a state of mime.
Shouldn't everyone that pushes to Git still have their local? In the case of group projects, a ton of locals.
reduce the urgency in a disaster by making the manner in which you would recover part of your daily routine - to whatever extant possible.
"It's a doughnut stuffed with M&M's. That way when you finish the doughnut, you don't have to eat any M&M's."
Am I the only one that read the title thinking the data was recovered?
That matches my experience. My company offers an offsite, bootable backup solution so if anything bad happens to your server, you just boot the appropriate clone in our cloud and you're back in business. A LOT of our customers get our service when they find out the hard way why *proper* offsite backups are important. Many weren't too concerned about backup and business continuity until something bad happened to them.
AFTER they have a major loss they get serious about making sure it won't happen again.
It could have been far worse, and I imagine GitLab will make damned sure backups and suchlike work properly in future.
John_Chalisque
Git repositories are just a public facing version of the local version you should already have a backup for. Even if they lost everything so F'ing what. Just re-sync your repo to your local and it's all back online again. If each developer isn't making local backups with redundent HDD or raid then that's completely your fault. Git is just a repository, anything lost worth a damn you can always reupload to their server again with no problem. If you can't do it this way and rely solely on Git to be the one and only storage of your code then you're an idiot... idiots and programmers generally are not synonymous so I really don't see any issue here at all.
A company with 25 million VC bucks and customers like IBM, Redhat, and NASA doesn't have a working backup system? Let me guess, everybody at Gitlab is a developer, and the whole thing runs on node.js in Docker containers.
Emphemeral data tends to have little value long term, so lost "issues, merge requests, users, comments, snippets, etc" is not so big of an issue.
That any data at all is lost indicates there is a broken process where they are unable to protect even trivial bits of data. Finding a copy is just dumb luck, nobody plans to find copies of missing data.
Well, I was involved verifying that we were in compliance. Over 100k products and some percentage were software, probably under 5%. A few projects were archived in the company archives. Funny coincidence I was there when the initial procedures were established. I didn't establish them, but I used them to archive a few software projects I was involved with. Bounce back to Y2K and I am requesting source code from several projects, now defunct but quite possibly will existing users. Simple, we will read the source, establish if there was or was not any time related silliness and go back to the main projects and 10's of thousands remaining. Of course you have already guessed it. No files were available. All the storage media provided was blank. I suspects that the dd-equivilent was done backwards. Take the blank storage and dd it to the incoming data. Fun. I doubt if it has ever been corrected as my comments were ignored back then. BTW, I am blissfully retired.
What kind of IT organisation has $25 Million at their disposal, has a core business of looking after developers data and yet doesn't have snapshots and backups on that data? Seriously? In this day and age? Most modern filesystems have snapshot abilities and the ability to export those snapshots, wouldn't you then do an rsync or tape backup as a belt and braces thing? Also, check your backups, have backup monitoring in place, copy the data somewhere else as a DR plan, undertake test restores, copy important data to two locations (i.e. metadata)?