Man Deletes His Entire Company With One Line of Bad Code (independent.co.uk)

Reader JustAnotherOldGuy writes: Marco Marsala appears to have deleted his entire company with one mistaken piece of code. By accidentally telling his computer to delete everything in his servers, the hosting provider has seemingly removed all trace of his company and the websites that he looks after for his customers. Marsala wrote on a Centos help forum, "I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line. All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)." The terse "rm -rf" is so famously destructive that it has become a joke within some computing circles, but not to this guy. Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?"Rm -rf" would mark the block as empty, and if the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.

Comments

--no-preserve-root

[zopper]

Does he use --no-preserve-root by default? I think that it is there for many years. Of course, if his servers are running on something from 2004, then his rm might be without this safeguard...

Fun thing about TRIM

[CajunArson]

While this guy was most likely using traditional HDDs where block level recovery is a possibility, for those of you using SSDs that have TRIM properly enabled, don't expect to be able to recover deleted files from the same drive unless you are really really fast.

TRIM automatically zeros the blocks of deleted files and they are GONE aside from vague sci-fi and probably nonexistent NSA-type forensics.

Re:Fun thing about TRIM

[Rockoon]

When the OS sends a trim command, with it is information about what the logical sector should look like if an attempt is made to read it again. IIRC the options are zeros, ones, and random.

Without trim the ssd has to preserve the entire logical block device its emulating, ie if you have a 64GB drive then even if it only has 4KB of "files" on it, the device still has to preserve all 64GB because it doesnt even know what a file is, let alone that you deleted one.

With trim the ssd only has to preserve what the OS told it was important to preserve. So instead of preserving 64GB if data it only has to preserve your 4KB of data. Trim marks logical sectors as dont-preserve.

What the SSD will not do is overwrite trimmed physical sectors just because they were trimmed. In fact, that data could linger there for years even with a high amount of read/write activity because SSD's only erases entire physical blocks, not just the subsectors within blocks that were trimmed.

So recovering is not sci-fi. Recovery is a fact. What can't be done is recovering the data via commands that target the logical rather than physical device.

Re: Three words

[GameboyRMH]

Addendum - just checked a CentOS server, and rm --help says that --preserve-root is enabled by default, and has to be overridden with --no-preserve-root.

Re:Three words

[Megane]

Because he is a retard.

All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script

Clearly a case of a fool thinking that a sync (copying data to another place regularly) is a backup. It's not a backup if you can easily copy corrupted data to your only copy. Or, in this case, if you can easily delete the data from your "backup" copy.

Re:manishs

[Verdatum]

I mean that I really do appreciate it. Keep up the good work!

Chain of Mistakes

[Greyfox]

Recently the USPA was talking about stuff that kills skydivers. It's almost never just one mistake. It's a chain of mistakes where one single good decision anywhere in that chain would break the chain and prevent entirely preventable deaths. In the case of this story, if it had actually happened, which it didn't, the decisions made to violate best practices all along the chain (IE, running your bash scripts as root or as any user ID that has authority to delete anything on the file system, not pushing just pushing your backup data to isolated storage, not having numbered sequential backups, etc) would be so egregious that the story would simply be an example of Darwin at work. The conversation would go "Oh hey, did you hear about that guy who designed his system so badly that he was able to delete the whole fucking thing with one mistyped command? Yeah, the council of sysadmins voted to kill him. Said it was for the good of the species."