Slashdot Mirror
Man Deletes His Entire Company With One Line of Bad Code (independent.co.uk)
Reader JustAnotherOldGuy writes: Marco Marsala appears to have deleted his entire company with one mistaken piece of code. By accidentally telling his computer to delete everything in his servers, the hosting provider has seemingly removed all trace of his company and the websites that he looks after for his customers. Marsala wrote on a Centos help forum, "I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line. All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)." The terse "rm -rf" is so famously destructive that it has become a joke within some computing circles, but not to this guy. Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?"Rm -rf" would mark the block as empty, and if the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.
Offsite, offline BACKUPS
Does he use --no-preserve-root by default? I think that it is there for many years. Of course, if his servers are running on something from 2004, then his rm might be without this safeguard...
I saw the post on ServerFault, and while the original scenario could have happened, the OP's follow-up blunder to reverse the input and output parameters of dd when trying to preserve the disk seemed just a wee bit too unlikely. I looked at the article to see if there was any additional data to suggest this was real, but it seems entirely based on the SF thread. Until corroborated, I'm going to call bs.
This is borderline bait at this point.
Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?
There are plenty of examples already and keeping a set of backups physically disconnected from running infrastructure is pretty well established practice, with random software bugs and screw ups being just one of many reasons. That said people will continue to have all their backups fully accessible (and destroyable) or just not back things up at all and things like this will continue to happen.
Guy can possibly recover the data, but the company is probably still screwed reputation wise.
I have that cold feeling in my stomach just reading this summary. ick.
I did something similar (though not quite so destructive) nearly 20 years ago when I was first learning Linux.
I my case I was trying to get rid of all the hidden files in root's (/root) home dir using 'rm -rf .*'
Guess what that did?
Yeah, that wasn't a highlight of my career...
My eyes reflect the stars and a smile lights up my face.
While this guy was most likely using traditional HDDs where block level recovery is a possibility, for those of you using SSDs that have TRIM properly enabled, don't expect to be able to recover deleted files from the same drive unless you are really really fast.
TRIM automatically zeros the blocks of deleted files and they are GONE aside from vague sci-fi and probably nonexistent NSA-type forensics.
AntiFA: An abbreviation for Anti First Amendment.
This has such a smell of BS around it. given the fact that backups are indeed offsite and that a company has more the 1 server etc.etc. Even my own simple setup consisting of a pc, laptop, tablet, qnap and some external HDD and sticks is impossible to delete with 1 script. total bollocks.
Wonder if he found incriminating material or has gambling debts, far more plausible
Manishs, you seem to actually critically read articles before posting them, and you actually provide insight after the summary. What is up with that?
This was a blatant troll on a forum and now because some idiot millennial wrote an op-ed piece, some idiot (manishs) put it on the /. frontpage?
Are the admins now supporting the things the moderation system fights on their own site?
This story is more of an embarrassment than the political vomit I've had to endure because _this_ story doesn't even qualify as news. e.g. What Company did he destroy exactly? You would think the incredibly obvious lack of facts would be a tipoff to someone.
Often wrong but never in doubt.
I am Jack9.
Everyone knows me.
You missed one.
-- Some anonymous coward complains about people calling out trolling, trolling.
The funny part is the person you responded to did not claim "trolling", they expressed a healthy skepticism. That last part is something more people should have. There are plenty of liars out there. Quite often they work for main stream media outlets and hold public offices.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
I collect these stories for people who I mentor. Even if they're trolls, they work as cautionary tales, because lots of people have had similar smaller scale disasters (as evidenced by posts in this thread) and it's healthy for mentees to get a taste of what can happen when you (for example) forget to error check your script parameters.
In a big way it doesn't matter if it's true or not, it could be true which makes it a teachable moment. I'm sure everyone who reads the story will run a mental checklist to see if they have a script somewhere that could EVER do it. Do they have their backups mounted when they should be rsyncing, etc.
Min
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
Man ALLOWS his entire company to be wiped out in one command.
Man DESIGNS his entire company to be wiped out in one command.
Man SETS UP his entire company to be wiped out in one command.
Hint: I work in schools. Once I had a teacher delete their entire planning folder. Then (and DO NOT ask me why, because I don't understand it either), they emptied that folder from Recycle Bin. They rang up in the more embarrassed panic.
And then it was explained that we still had copies of that folder in:
a) Shadow Copies of the profile on the client.
b) Network Copies of the profile that they were logged in as (and which fortunately hadn't logged off once they realised what they did).
c) Shadow Copies of the profile folder on the server.
d) Copies of the profile folder on all the other servers.
e) Copies of all the servers on replica servers.
f) Copies of the server VM's and storage in a primary backup location.
g) Copies of the server VM's and storage in a secondary backup location.
h) Copies of the server VM's and storage in a tertiary backup location.
i) Several off-line and off-site copies of the server VM's and storage .
k) Random, casual backups all over the place.
And that's just for the crap that teachers think is important (i.e. a lesson plan they have to write every two weeks and which they can't re-use anyway).
Fuck knows what this guy was thinking, but there's no one one command ANYWHERE should be able to do that many actions, let alone dangerous actions that you haven't evaluated properly. Honestly, some of those machines don't even TURN ON until the backup window, and even the backup devices have rollback and shadow-copy-like functionality on top of whatever the backup software gives (incrementals, etc.). And several are DELIBERATELY offline for almost their entire lives and have entirely disparate credentials so no one command could ever affect them.
Not being funny, but we're talking a small school of 400 5-14 year olds here. He actually has more customers than I have users. And you just can't fuck about like that, so if he thinks he can, I honestly have zero sympathy and can only laugh.
I just got put on a project at work as "the systems guy" for a project being built in Azure. This is in support of a reasonably critical system, and the development staff are salivating over the chance to self-deploy code and infrastructure. It sounds like this problem was caused by the first thing I noticed as a risk -- if you don't limit what Azure users can do, it's just like giving them the keys to the data center. And this isn't in an "evil BOFH control freak" sense, this is just the fact that everything in Azure is virtual and easily changed either manually or through automation. So, someone who's having a bad day could easily make a mistake and get rid of things they have permissions on -- it's possible in AWS too.
It's a really different mindset than even a hosted IaaS service. There, if you do something stupid, at least the physical infrastructure doesn't get rolled up and carried off. Now hopefully you have backups if that happens and can just restore the VMs and storage as needed, but if developers are running the show I would highly doubt it. (In Marco's case, I would imagine this was caused by the classic "run as root, because I'm the boss" issue.
So, in summary, all the (good) sysadmins worrying about the cloud taking their jobs need not worry. The rules of designing a safe computing environment have changed, but they haven't gone away entirely! I'd be a little worried if I were a savant-level EMC or Cisco guru right about now, but generalists with good heads on their shoulders are still in demand.
Put backup copies in truck.
Drive them to the backup site.
Repeat regularly.
I'm an American. I love this country and the freedoms that we used to have.
He admitted it publicly?
-- I am. Therefore, I think!
Rsnap is a very popular backup system which uses network mounted drive as it's default/most common configuration. I constantly remind people on the rsnap mailing list about the existence of cryptolocker type malware.
A much safer way to do it is to have the backup system PULL backups using a read-only account. That way no command on the live system can touch the backups, and the backup system can't change anything on the live system - either accidentally or maliciously.
One solid backup / hot spare system that does it safely by default is Clonebox.
For most users the cloud is more reliable and more accessible than anything they'll ever be able to do in a SO/HO environment.
The problem is that medium sized (1000+ employees) seem to think this too.
And then have no backups, version control or anything else because some PHB said 'put it in the cloud, and stop arguing with me.' to the IT engineers who wanted local & remote backups, version controls, redundancy of hardware, network and power, etc, etc, etc.
There are two types of people in the world: Those who crave closure
Recently the USPA was talking about stuff that kills skydivers. It's almost never just one mistake. It's a chain of mistakes where one single good decision anywhere in that chain would break the chain and prevent entirely preventable deaths. In the case of this story, if it had actually happened, which it didn't, the decisions made to violate best practices all along the chain (IE, running your bash scripts as root or as any user ID that has authority to delete anything on the file system, not pushing just pushing your backup data to isolated storage, not having numbered sequential backups, etc) would be so egregious that the story would simply be an example of Darwin at work. The conversation would go "Oh hey, did you hear about that guy who designed his system so badly that he was able to delete the whole fucking thing with one mistyped command? Yeah, the council of sysadmins voted to kill him. Said it was for the good of the species."
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
"To err is human. To really fuck things up, you need a computer."
I prefer that any bulk or query-based "delete" command ask for confirmation along with basic feedback. Example pseudo-code:
> delete *:*.*
You are about to delete 832 folders and 28,435 files.
Your choices are:
1 - Proceed with deletion
2 - List path details about the above folders and files
3 - Cancel deletion
Your Choice: __
(end of example)
It may be slower and/or more resource intensive, but that's better than mass boo-boo's.
An optional command parameter could switch off verification, but verification should be the default. This is something Unix/Linux gets backward in my opinion: the default should be confirmation mode, not the other way around. In other words, a command switch should be required to switch off confirmation rather than requiring a command switch to turn confirmation on.
Typical SQL doesn't have a confirmation mode, so I usually do a verification query on the WHERE clause before running the actual:
-- check
SELECT count(*) FROM myTable
WHERE x > 7 AND foo='BAR'
-- actual, keeping same where-clause
DELETE FROM myTable
WHERE x > 7 AND foo='BAR'
I also often inspect at least some of the actual rows, not just the count. Thus, as a rule of thumb, do random spot-checks of actual data, and a total count before final command execution.
Table-ized A.I.
The best of all worlds is pull based backup software. However, the enterprise based programs are extremely pricy, well out of the range for a home user. The cheapest around would probably be Windows Fundamentals which is a descendant of Windows Home Server.
What I've wound up doing on a small scale (this won't scale up past a few machines) is having a hardware NAS appliance. It had a samba share and account for every machine. The Windows boxes use Veeam to dump their data onto the individual shares. Every 15 minutes, the NAS pops a snapshot of each share, where several are kept for each hour/day/week/month/year, and the rest get tossed after a while. Every eight hours, the NAS backs itself up to an external HDD. This protects against ransomware in several ways. If ransomware just zaps the share, restoring the snapshot and bare-metal loading the machine isn't too bad. If ransomware takes its time and zeros files over an interval, because I have weekly, monthly, and backups over a duration, there is a good chance that I will still have the file around, either in a snapshot, or on the backup drive. Because each machine dumps to a separate share via a separate account, ransomware on one box can't destroy or access another machine's data.
The ideal would be having the NAS maker writing an agent that sits on Windows and uses SSH or another time-tested protocol to pull backups. This would not just guarantee that backups are done, but are protected against ransomware.
It was a pretty obvious troll if you read the whole thread:
1. The guy claims to have made the most insanely improbable mistake to kill his entire set of servers. Possible, but unlikely. Most took the bait on this one.
2. He had no explanation as to why "--no-preserve-root" didn't save him - basically looked like he didn't know about it, and he was lying.
3. later on he responded to someone's suggestion to use dd to backup saying he reversed if and of - which is probably the second most joked about UNIX sysadmin error after "rm -rf".
So, either you are pretty clueless about any of this, or, like another poster suggested, you are also a troll. Based on the specifics of your moronic post, probably the latter.