Slashdot Mirror
GitLab.com Melts Down After Wrong Directory Deleted, Backups Fail (theregister.co.uk)
An anonymous reader quotes a report from The Register: Source-code hub Gitlab.com is in meltdown after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups. On Tuesday evening, Pacific Time, the startup issued the sobering series of tweets, starting with "We are performing emergency database maintenance, GitLab.com will be taken offline" and ending with "We accidentally deleted production data and might have to restore from backup. Google Doc with live notes [link]." Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated. Just 4.5GB remained by the time he canceled the rm -rf command. The last potentially viable backup was taken six hours beforehand. That Google Doc mentioned in the last tweet notes: "This incident affected the database (including issues and merge requests) but not the git repos (repositories and wikis)." So some solace there for users because not all is lost. But the document concludes with the following: "So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place." At the time of writing, GitLab says it has no estimated restore time but is working to restore from a staging server that may be "without webhooks" but is "the only available snapshot." That source is six hours old, so there will be some data loss.
Don't tell the customer anything until the dust settles! Geez... What's with these amateurs?
Don't tell the customer anything!! Geez... What's with these semi-pros?
Backups were ineffective. 30% of our users lost their home directories permanently. He never lived it down. Check your backups!
Actually: Check your privilege! (Especially if rm -rf is involved).
but NOT on your production hardware running live services.
me thinks gitlab should have browsed their hosted repos for some backup software.
Just imagine if git had some other magical copy of the repo somewhere, maybe even on the local machine you develop on, now that would save your data in a case like this
paki chimps in jungle
Someone failed geography class...
"I don't know, therefore Aliens" Wafflebox1
Sing with me, kids:
One backup in my bunk
One backup in my trunk
One backup at the town's other end
One backup on another continent
All of them tested and verified sane
now go to bed, you can sleep once again
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Good choice. But, I always use this prompt:
PS1='C:$(echo ${PWD//\//\\\} | tr "[:lower:]" "[:upper:]" | sed -e"s/\\([^\\]\\{6\\}\\)[^\\]\\{2,\\}/\\1~1/g" ) >'
SJW n. One who posts facts.
Q: You can never have too much money, too much sex, or ___ ____ ______. (Fill in the blanks.)
(A: "Too many backups".)
--Actual question from the final exam for the Networking 100 class I took in 1998.
Il n'y a pas de Planet B.
Look at his user ID. Give him time, he'll come around.