Slashdot Mirror
Man Deletes His Entire Company With One Line of Bad Code (independent.co.uk)
Reader JustAnotherOldGuy writes: Marco Marsala appears to have deleted his entire company with one mistaken piece of code. By accidentally telling his computer to delete everything in his servers, the hosting provider has seemingly removed all trace of his company and the websites that he looks after for his customers. Marsala wrote on a Centos help forum, "I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line. All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)." The terse "rm -rf" is so famously destructive that it has become a joke within some computing circles, but not to this guy. Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?"Rm -rf" would mark the block as empty, and if the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.
Offsite, offline BACKUPS
..."In the cloud" does not mean "Is backed up."
Does he use --no-preserve-root by default? I think that it is there for many years. Of course, if his servers are running on something from 2004, then his rm might be without this safeguard...
two words: offline backup
It's amazing how such simple things seem to be so hard to sink in for way too many "professionals".
TFA and even TFS says he deleted all the backups too. In a hurry to try and get first post?
Which command does one type to do that? (Note: /root, /usr/sbin, /etc, etc. are gone...)
I saw the post on ServerFault, and while the original scenario could have happened, the OP's follow-up blunder to reverse the input and output parameters of dd when trying to preserve the disk seemed just a wee bit too unlikely. I looked at the article to see if there was any additional data to suggest this was real, but it seems entirely based on the SF thread. Until corroborated, I'm going to call bs.
This is borderline bait at this point.
Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?
There are plenty of examples already and keeping a set of backups physically disconnected from running infrastructure is pretty well established practice, with random software bugs and screw ups being just one of many reasons. That said people will continue to have all their backups fully accessible (and destroyable) or just not back things up at all and things like this will continue to happen.
Guy can possibly recover the data, but the company is probably still screwed reputation wise.
... about unix filesystems. Or about all the other customers of this webhoster guy's hosting company.
You'd have to completely freeze everything then spend a fsckton of time with fs debuggers. That, or you restore your offsite backups. Er, right. So those "offsite backups" weren't very offsite now then, hm?
I have that cold feeling in my stomach just reading this summary. ick.
I did something similar (though not quite so destructive) nearly 20 years ago when I was first learning Linux.
I my case I was trying to get rid of all the hidden files in root's (/root) home dir using 'rm -rf .*'
Guess what that did?
Yeah, that wasn't a highlight of my career...
My eyes reflect the stars and a smile lights up my face.
While this guy was most likely using traditional HDDs where block level recovery is a possibility, for those of you using SSDs that have TRIM properly enabled, don't expect to be able to recover deleted files from the same drive unless you are really really fast.
TRIM automatically zeros the blocks of deleted files and they are GONE aside from vague sci-fi and probably nonexistent NSA-type forensics.
AntiFA: An abbreviation for Anti First Amendment.
"line of code". "programmer". Ok, media.
At least the Google crawler didn't delete it.
a couple cheap Kimsufi servers from OVH for remote backup in EU and In Canada?
by TheSpoom (715771) Uncaring Linux user here. I have nothing to add to this but please continue. *munches popcorn*
Indeed, it's something that happens to most eventually. I managed to take out an entire file server by accidentally running the script we use to populate diskless nodes on it. People were woken up, little sleep was had, backups were hastily restored, and my name will likely be prefixed with "pulled a" and used to describe pretty much any minor technical screw up for the remainder of my career (but hey, least I kept my job).. shit happens.
The failure here was entirely in how backups were managed.
This has such a smell of BS around it. given the fact that backups are indeed offsite and that a company has more the 1 server etc.etc. Even my own simple setup consisting of a pc, laptop, tablet, qnap and some external HDD and sticks is impossible to delete with 1 script. total bollocks.
Wonder if he found incriminating material or has gambling debts, far more plausible
I'll bet there's a coupon for that.
Guards around the command asserting the variables?
Manishs, you seem to actually critically read articles before posting them, and you actually provide insight after the summary. What is up with that?
Why in hell is is running scripts out of ansible? Why are those scripts not running on a QA system thats a block for block clone of production? Finally what idiot thinks that some mounted drives he copies stuff for is a backup system?
Tape disk I do not care just treat disk as tape, plenty of backup system are more than happy to do just that. Rsync is not nor will it ever be a backup, snapshots are not a backup, some script some guy wrote that works ok is not a backup. Now they can all help to meet your RTO but at the end of the day it's not backed up till the device is offline sitting offsite.
No sir I dont like it.
always use sudo, disable root account. Well let's hope its a lesson learned /rofl...
This was a blatant troll on a forum and now because some idiot millennial wrote an op-ed piece, some idiot (manishs) put it on the /. frontpage?
Are the admins now supporting the things the moderation system fights on their own site?
This story is more of an embarrassment than the political vomit I've had to endure because _this_ story doesn't even qualify as news. e.g. What Company did he destroy exactly? You would think the incredibly obvious lack of facts would be a tipoff to someone.
Often wrong but never in doubt.
I am Jack9.
Everyone knows me.
Unless you also have your nerves re-routed, I doubt you'd taste anything. Also, you'd need to also transplant the Vas Deferens, prostate and testicles to be able to ejaculate.
Hobbling the default rm command slightly would make a sense, possibly having a second command (oblit or something) for the really nasty stuff, would make sense. Many commands can be unnecessarily destructive, and those destructive commands are too easy to invoke by accident. Possibly requiring a --really and a --reallyreally switch on rm to enable things like rm -rf crossing filesystems, would make sense. I did once make a quick hack so that rm -rf would require an environment variable to be set in order to work, but then various scripts broke.
John_Chalisque
I thought that was simply implied. Duh!
When an friend an I got started with Linux he wanted to remove his Slackware install from a dual boot PC. For fun he ran rm -rf / on that install. We had a good laugh when the message scrolled by of the OS trying and failing to remove files from the CDROM. That was until he realized that he had mounted his Windows partition too. It didn't fail to remove files there :-)
This guy has learned the hard way about losing data because he wrote bad code.
He had a backup, but his code bypassed common sense and mounted the backup server's remote drives in the filesystem.
Which he then wrote a script that ran as root that in this case deleted everything because it didn't sanitise inputs before running the 'rm' command.
Ideally he would have built the path to be deleted, so he could check that it was sane, i.e., starts with "/users/" or similar, before passing it to 'rm'.
Worse, is if he was using an off-the-shelf-hosting-package that did this. But when you do, assume it is broken, keep a wall between the systems it can touch, and your last hope (your backups).
Backup systems should ideally be push only, with any delete action only occurring after validating the replacement. And let's be honest, in this day and age, you can keep multiple generations of backups, push them into Amazon Glacier to keep things cheap.
I truly hope he was a low end hoster that gave ftp access for website uploads only, so the customers can re-upload their sites, once he reprovisions his servers. Although that's the best option, any databases (for example) would have been wholly reliant on his backup strategy.
Sheesh, who makes their backups part of the system that is being backed up! It's meant to be isolated because of, frankly, the situation that occurred.
Spell it SystemD not systemd.
That way it looks like an ASCII penis.
Because of all the fakesand liars out there.
Heck, once I told somebody that calling me a troll was lame, they tried to explain the meaning of the term, as if they were informing me about some usage of which I was not aware, and that I thought that the reference was to a mythological troll. It was quite the bit of cognitive dissonance.
Never allow push to a backup server... Stuff like this can happen, or intentional sabotage if someone pwns your production server. Making the backup server non accessible keeps things much safer. Also, always quadruple check any recursive deletion, and never do it with variables. If doing it by hand, type it with an echo in front first and then remove the echo once you like the command. This way, if you accidentally hit enter as you just typed the first /, "rm -rf /" just gets echoed instead of run.
You missed one.
-- Some anonymous coward complains about people calling out trolling, trolling.
The funny part is the person you responded to did not claim "trolling", they expressed a healthy skepticism. That last part is something more people should have. There are plenty of liars out there. Quite often they work for main stream media outlets and hold public offices.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
So, he writes a bash script for an important backup job and doesn't "set -u" at the beginning to throw an error if you attempt to use an uninitialized variable? Really?
I was working at a small development shop about 15 years ago and I came in one morning to find the main development server not working. Turned out that the previous night a developer on the other project ran "rm -rf" from the root directory on the Sun box and then tried to fix things before giving up and going home. No note, no call to the boss, nothing to indicate what had happened so I had to figure that out when I arrived around 8 AM. Oh, and no backups of their project. I at least had the latest version of the source code on mine on my laptop the but the history and development environments would have been toast.
Since he toasted the box and it was the only Sun box around (small company) I had to install the OS again. But the only thing that saved us was the fact that rm deletes files and folders alphabetically (or at least at that time on the Sun boxes it did). So as soon as it started removing everything in /dev the machine was toast and it couldn't delete anything else. I first found this out by booting off the CD so I was able to mount the drives. We had our development environments and source code on separate drives (thankfully) so I copied anything that was useful off of the drive used for / onto one of them, installed the OS onto that drive while leaving the other drives alone, and then manually putting things back to where they were. Took less than a day and the owner was very thankful. Never heard a word from the guy that caused the mess though.
Ever since then whenever I've created mount points for new drives on Linux/UNIX servers I've always made sure that they come after dev alphabetically.
Joke or not he's voluntarily entered himself into the timeless database known as Google, viewable with the not-so-secret incantation "google Marco Marsala"
On the other hand, if you read the server fault question, you'd know that the questioner both recovered all his data AND zero'd out his hard drives with dd. Obviously not trolling!
I collect these stories for people who I mentor. Even if they're trolls, they work as cautionary tales, because lots of people have had similar smaller scale disasters (as evidenced by posts in this thread) and it's healthy for mentees to get a taste of what can happen when you (for example) forget to error check your script parameters.
In a big way it doesn't matter if it's true or not, it could be true which makes it a teachable moment. I'm sure everyone who reads the story will run a mental checklist to see if they have a script somewhere that could EVER do it. Do they have their backups mounted when they should be rsyncing, etc.
Min
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
There's where snapshots and boot environments usually help but sensible admins will not save on a good backup solution. .. so there's no excuse.
tarsnap is both cheap and secure
sounds NOT like a sysadmin..
like poor planning
poor arch.
poor documentation
poor implementation
poor accountability
Poor company
I keep a file in my homedir:
---------- 1 root staff 0 Oct 21 2005 0000000%%%%%DANGER%%%%%%
That's so simple and effective, work on local network as well as on remote networks thank's to ssh.
Poor guy, he could have done it a lot faster with SaltStack.
Man ALLOWS his entire company to be wiped out in one command.
Man DESIGNS his entire company to be wiped out in one command.
Man SETS UP his entire company to be wiped out in one command.
Hint: I work in schools. Once I had a teacher delete their entire planning folder. Then (and DO NOT ask me why, because I don't understand it either), they emptied that folder from Recycle Bin. They rang up in the more embarrassed panic.
And then it was explained that we still had copies of that folder in:
a) Shadow Copies of the profile on the client.
b) Network Copies of the profile that they were logged in as (and which fortunately hadn't logged off once they realised what they did).
c) Shadow Copies of the profile folder on the server.
d) Copies of the profile folder on all the other servers.
e) Copies of all the servers on replica servers.
f) Copies of the server VM's and storage in a primary backup location.
g) Copies of the server VM's and storage in a secondary backup location.
h) Copies of the server VM's and storage in a tertiary backup location.
i) Several off-line and off-site copies of the server VM's and storage .
k) Random, casual backups all over the place.
And that's just for the crap that teachers think is important (i.e. a lesson plan they have to write every two weeks and which they can't re-use anyway).
Fuck knows what this guy was thinking, but there's no one one command ANYWHERE should be able to do that many actions, let alone dangerous actions that you haven't evaluated properly. Honestly, some of those machines don't even TURN ON until the backup window, and even the backup devices have rollback and shadow-copy-like functionality on top of whatever the backup software gives (incrementals, etc.). And several are DELIBERATELY offline for almost their entire lives and have entirely disparate credentials so no one command could ever affect them.
Not being funny, but we're talking a small school of 400 5-14 year olds here. He actually has more customers than I have users. And you just can't fuck about like that, so if he thinks he can, I honestly have zero sympathy and can only laugh.
I guess you have a phone call to Thailand to make before you fly there, huh?
I just got put on a project at work as "the systems guy" for a project being built in Azure. This is in support of a reasonably critical system, and the development staff are salivating over the chance to self-deploy code and infrastructure. It sounds like this problem was caused by the first thing I noticed as a risk -- if you don't limit what Azure users can do, it's just like giving them the keys to the data center. And this isn't in an "evil BOFH control freak" sense, this is just the fact that everything in Azure is virtual and easily changed either manually or through automation. So, someone who's having a bad day could easily make a mistake and get rid of things they have permissions on -- it's possible in AWS too.
It's a really different mindset than even a hosted IaaS service. There, if you do something stupid, at least the physical infrastructure doesn't get rolled up and carried off. Now hopefully you have backups if that happens and can just restore the VMs and storage as needed, but if developers are running the show I would highly doubt it. (In Marco's case, I would imagine this was caused by the classic "run as root, because I'm the boss" issue.
So, in summary, all the (good) sysadmins worrying about the cloud taking their jobs need not worry. The rules of designing a safe computing environment have changed, but they haven't gone away entirely! I'd be a little worried if I were a savant-level EMC or Cisco guru right about now, but generalists with good heads on their shoulders are still in demand.
Just great.
Now, I bet you probably want to use the Ladies' room too.
Put backup copies in truck.
Drive them to the backup site.
Repeat regularly.
I'm an American. I love this country and the freedoms that we used to have.
Wrong, the original commenter did claim "trolling".
The original comment's title is the rhetorical question, "Wasn't he trolling?" (emphasis added)!
See the word "trolling" in there? See how it's a rhetorical question? If we get rid of the word games, the title could be rewritten to simply, "He's trolling!"
Don't run in root all the time, USE sudo
Nope, not buying any part of this story, nope. No one is dumb enough to run that without a test. And how were the offsite backups even accessible? doesn't matter, because everything would be recoverable from the systems he "wiped". No, this is another bullshit story spread around by IT departments. this did not happen.
Just a bad brain!
Can you spell "test"?
Sent as ripples into the electromagnetic field. No single photon has been harmed in the process.
...Somebody accidentally transposes two letters when using the dd command...
Did you intentionally transpose your "dd"?
I would have a sig but I am too busy updating programs and restarting my computer
If they weren't on alt.religion.kibology back around '92 or so, they don't know shit about trolling.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
"trolling" are a very used adjective to claim someone is inferior, today. It's an Ad hominem heavy used tool, nowadays! The hell is, really, other people! :P
Every time I post something here, when I come back, the message is gone. And now I'm gone /. I don't now who or why you delete people's messages.
He admitted it publicly?
-- I am. Therefore, I think!
Thanks! I don't always read the subject and missed it. Can I call it an out of "body" experience? :^X The skepticism part is still correct.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
and watch out for the sequels:
powerplant IT worker cuts power to homes in Winter for 6 months. Thousands die. with one line of code
Female IT worker causes multicar pileup - in car AI - one liner.
Female Asian/black IT worker inadvertently fires last remaining male white IT professionals with one liner.
Game Over.
Rsnap is a very popular backup system which uses network mounted drive as it's default/most common configuration. I constantly remind people on the rsnap mailing list about the existence of cryptolocker type malware.
A much safer way to do it is to have the backup system PULL backups using a read-only account. That way no command on the live system can touch the backups, and the backup system can't change anything on the live system - either accidentally or maliciously.
One solid backup / hot spare system that does it safely by default is Clonebox.
I mean, he made /.'s front page. That's pretty darn good!
Declare a holiday tomorrow for all techies.
I already do. But only to "surprise-fuck" the "ladies".
foo bar is right!
I don't like windows, but LOVE PowerCLI for managing VMware environments. And HATE using pyvmoni.
I get to troll everyone.
There are two types of people in the world: Those who crave closure
Seriously, You should never ever concatenate two variables with a literal slash. It is much harder for rm -rf ${PATH}${ext} to resolve to rm -rf / . While its almost promised that rm -rf ${PATH}/${EXT} will one day resolve to rm -rf /. I think we have all ben trolled. Another really good rule is never urn a remove command as root in a script.
The likely hood of this happening is slim, but I sometimes wonder if a minor change is really not that bad. /, until -t/--top is added. Then it is allowed.
In this case, change rm to NOT allow
With this minor change, it could save noobs from themselves and would like not be used that often in the first place.
I prefer the "u" in honour as it seems to be missing these days.
Recently the USPA was talking about stuff that kills skydivers. It's almost never just one mistake. It's a chain of mistakes where one single good decision anywhere in that chain would break the chain and prevent entirely preventable deaths. In the case of this story, if it had actually happened, which it didn't, the decisions made to violate best practices all along the chain (IE, running your bash scripts as root or as any user ID that has authority to delete anything on the file system, not pushing just pushing your backup data to isolated storage, not having numbered sequential backups, etc) would be so egregious that the story would simply be an example of Darwin at work. The conversation would go "Oh hey, did you hear about that guy who designed his system so badly that he was able to delete the whole fucking thing with one mistyped command? Yeah, the council of sysadmins voted to kill him. Said it was for the good of the species."
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Talk about a bad day at the office. As an IT professional, I feel for the guy.
On a Linux system, root is God(*)
God is omniscient, omnipresent, and infallible.
Therefore, when root deletes files, it's never a mistake, and the files should be immediately destroyed forever without question.
(*) Unlike those heathen Windows systems, where there can be multiple gods, some of whom are more equal than others... and not necessarily in ways that are obvious to casual observers... ;-)
"To err is human. To really fuck things up, you need a computer."
I prefer that any bulk or query-based "delete" command ask for confirmation along with basic feedback. Example pseudo-code:
> delete *:*.*
You are about to delete 832 folders and 28,435 files.
Your choices are:
1 - Proceed with deletion
2 - List path details about the above folders and files
3 - Cancel deletion
Your Choice: __
(end of example)
It may be slower and/or more resource intensive, but that's better than mass boo-boo's.
An optional command parameter could switch off verification, but verification should be the default. This is something Unix/Linux gets backward in my opinion: the default should be confirmation mode, not the other way around. In other words, a command switch should be required to switch off confirmation rather than requiring a command switch to turn confirmation on.
Typical SQL doesn't have a confirmation mode, so I usually do a verification query on the WHERE clause before running the actual:
-- check
SELECT count(*) FROM myTable
WHERE x > 7 AND foo='BAR'
-- actual, keeping same where-clause
DELETE FROM myTable
WHERE x > 7 AND foo='BAR'
I also often inspect at least some of the actual rows, not just the count. Thus, as a rule of thumb, do random spot-checks of actual data, and a total count before final command execution.
Table-ized A.I.
Or alt.folklore.urban. Some world class trolling occurred there, back in the day.
There is no God, and Dirac is his prophet.
Somebody accidentally transposes two letters when using the dd command, perhaps after many hours of panic and perhaps after days without sleep? It must be "trolling"!
You are the one trolling here. No matter how many letters you transpose, 'dd' is still 'dd'.
.
Offsite data should be offsite. Period.
It should never be accessible by the day-to-day operational processes, and that includes the routine backups.
He needs another backup step between the local and the offsite backups.
I back up local data to a onsite staging area. Then, in a totally separate process, I back up the staging area to offsite facilities.
I empathize with the guy....
"Rm -rf" would mark the block as empty, and unless the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.
He actually rm -rf'd on a production system. Probably hasn't the foggiest how to pull data off the drives.
Where it would've done this without any user input at all.
Is there any address to send flowers?
When will the funeral be?
I did something similar to my server in 2005. It wouldn't have been so bad, except that (!) I hadn't made a backup in months and (2) the server hosted a disaster recovery project whose purpose was to guard against catastrophic data loss. A cobbler's bairns are aye the worst shod...
This is back on a System V Motorola 8810 Box in the early 90's.
Wrote me a clean up shell script to delete all files in a relative folder older than 2 days, tested it multiple times then dropped it into cron.
Two days later no-one can log in and the box is dead. Now, I had backups and I restored the previous night - all's good but I was a loss to understand what happened (yeah, I mention I was an idiot?).
Two days later, same thing. Finally the penny dropped. My cleanup script, when run via cron, had a relative path of root. Two days later it deleted all untouched objects older than 2 days FROM ROOT. AGAIN.
Boy, did I feel like a complete muppet.
Learned the value of good backups though.
The best of all worlds is pull based backup software. However, the enterprise based programs are extremely pricy, well out of the range for a home user. The cheapest around would probably be Windows Fundamentals which is a descendant of Windows Home Server.
What I've wound up doing on a small scale (this won't scale up past a few machines) is having a hardware NAS appliance. It had a samba share and account for every machine. The Windows boxes use Veeam to dump their data onto the individual shares. Every 15 minutes, the NAS pops a snapshot of each share, where several are kept for each hour/day/week/month/year, and the rest get tossed after a while. Every eight hours, the NAS backs itself up to an external HDD. This protects against ransomware in several ways. If ransomware just zaps the share, restoring the snapshot and bare-metal loading the machine isn't too bad. If ransomware takes its time and zeros files over an interval, because I have weekly, monthly, and backups over a duration, there is a good chance that I will still have the file around, either in a snapshot, or on the backup drive. Because each machine dumps to a separate share via a separate account, ransomware on one box can't destroy or access another machine's data.
The ideal would be having the NAS maker writing an agent that sits on Windows and uses SSH or another time-tested protocol to pull backups. This would not just guarantee that backups are done, but are protected against ransomware.
So an admin took 1500 companies off line permanently and we only hear about it from some random forum? Not very believable.
Sounds like a dago or a wop, what did you expect?
LVM and rsync are your friend.
The easiest way to backup physical is with rsync, LVM and snapshots. No rm required. None of my backup scripts have /bin/rm in them?
If it is a VM just take a snapshot of the VM?
Why would any backup script use "/bin/rm"?
This sounds really fishy.
Or really incompetent.
Or both.
The usual way to do it using the shell file-name expansion is:
.[!.]* .[^.]*
bash/dash/sh: ls
bash/csh/zsh: ls
which will pick all the dot-files but not double-dot (link to parent directory). Only time this will be problematic is if there are files/directories named with the style "...three-dot-file".
You must be on a salt restricted diet. Nobody likes unsalted earth, or earth that only has the salt you added while cooking.
Myself, I like to scorch it, salt it, scrape it into a squarish pile with four corners, dye it red, vibrate it until I'm sure the earth moved, and then chant "Tara! Tara! Tara!" I do all this wearing Birkenstocks. It's a thing.
Tastes much better that way. Really.
I've fallen off your lawn, and I can't get up.
Not transposing letters, but I have managed to swap 'skip=' and 'seek=' when trying to work around a bad block error. The results were, to say the least, not good. Argh...
You had made a block copy backup via direct connection first, right? (I'm just assuming you can still do that, since it's been several decades since I had someone do it for me.)
Backups are the best answer, but if you don't have backups, the first step should be to make a sector-by-sector backup. And I, personally, have had more success having a hardware guy do it for me in hardware. OTOH, I'm not sure I could even edit a disk anymore. The last time I needed to to that I was using MSWind95.
I think we've pushed this "anyone can grow up to be president" thing too far.
Then again, rsnap is a seven kilolines perl script around rsync...
CLI paste? paste.pr0.tips!
rsnapshot is 7klines of Perl script around rsync which does _pull_ backups.
I remember you.
Fuck backups, fuck trolls, fuck that idiot, fuck slashdot, and FUCK YOU!!!!!!!!!!!
So, first, people figured out that rm -rf / is bad, so they added an option that would disallow it by default, but you can turn it off, and rm -rf /* may still work.
But, the command never makes sense. What does it mean to remove /proc? Or /dev? People quote the ideal that you could be able to do really dumb things to be able to do really great things, but why allow things that don't make any sense at all? "rm -rf /dev/null" for example. What does that mean? It's seems reasonable to say "rm -r /" or "rm -r /*" has no meaning, as it asks to perform an operation on filesystems where it has no defined semantics, and therefore, you can always forbid it. After all, you can still list all the directories and files you want manually to get the same effect.
Fuck backups, fuck that idiot, fuck slashdot, and FUCK YOU!!!!!!!!!!!!!!
He just wanted to collect fat-finger insurance. This scam has been around as humanity has had keyboards.
Snapshots.
I accidentally my entire company.
what should I do...is this dangerous ?
First, "Rm -rf gets you an error because the comman "Rm" is unknown. And second, Unix-filesystems are not moron compatible (different from Windows) and recovering files is generally very hard or infeasible.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Wayback machine is his way back
Some good points made by various posters, amongst all the piling on.
Here's one suggestion which, curiously, I don't see having been made by anyone yet. This is a good illustration of how putting "set -u" at the top of the script can help you. It would have prevented this catastrophe, actually. "Set -u" causes any use of an undefined variable to abort before executing the command, and the script will be terminated on the spot with a diagnostic identifying the particular variable as an "unbound variable".
It certainly won't prevent ALL rm boo-boos. but it would have prevented this one. It works in POSIX/BSD sh as well as bash.
Rsnap is a very popular backup system which uses network mounted drive as it's default/most common configuration. I constantly remind people on the rsnap mailing list about the existence of cryptolocker type malware.
A much safer way to do it is to have the backup system PULL backups using a read-only account. That way no command on the live system can touch the backups, and the backup system can't change anything on the live system - either accidentally or maliciously.
One solid backup / hot spare system that does it safely by default is Clonebox.
Both BackupPC and rsnapshot can run from a stand-alone system, which is not accessible by any account on production machine/s, and goes in via an SSH key and takes a copy of whatever you want. You can do full and incremental backups, and have a configurable amount of copies.
'rm' is not recognized as an internal or external command, operable program or batch file.
Nullius in verba
Quite easy to do with btrfs snapshots... Pull via ssh or push via ssh and a limited backup-recipient user... And you only get changed blocks.. full backups every day and incremental every 2 hours works perfectly..
A Linux server running 1535 websites, let me guess another wordpress script kiddy trying to pose as a legitimate businessman. Surprised he's even aware of stackexchange.
Based upon the article, the business has self-selected a Darwin Award for gene pool elimination.
Of course, the bug in the script wasn't the main problem in this case, but all the bigger problems have been done to death in this thread.
What's missing is that nobody seems to mind that in Bash, which is ostensibly targeted at doing bulk operations, system administration, file management, i.e. potentially very dangerous stuff, undefined variables silently expand to nothingness.
This is a stupid question, but does that command actually overwrite the hard drives? I thought that usually you could read all deleted data that wasn't overwritten yet.
this must have been!
When I was a syadmin, a greybeard told me a trick about adding a file to your root directory called "-i". Then if you ever try to execute "rm -rf /" as soon as it sees the -i file, it will be interpreted as a command flag and switch to interactive mode, forcing you to confirm each delete and realize your mistake. Does this not work anymore?
I've seen this happen many times in my career (and not by my hand): people using rm -fr $foo/$bar or rm -fr /$foo and not doing proper validity checks on their variables prior. I also sigh when folks don't double-quote their paths, so spaces in variables (hence pathnames) end up causing "unexpected" grief. This is all shell scripting 101 (maybe 102) type of stuff. Does nobody know about set -u (nounset)? Even find -x (i.e. -xdev or -mount on GNU) would have prevented this. I just don't get it.
Also, for those who missed the paragraph after the rm -fr {blah}/{whatever} explanation: they did offsite backups, but the problem was the admin NFS mounted (I assume NFS) the backup repositories under paths in /, so they got wiped as well.
What really chaps my ass about all of this: why is this fellow doing all of this in the first place? It sounds like he was shell scripting his own backup system. Was rsnapshot (which wouldn't have shot him in the foot this way, in any way/shape/form) not thought of? Even basic rsync with --delete? There are several possibilities/alternate solutions, all which are certainly more ideal.
Instead, all I see described in the article is the sign of an admin who deserves exactly what he got: bad design engineering choices, combined with badly-written scripts, resulted in something awful. My entire life I've stuck to an anti-BOFH mantra, despite being a UNIX SA, but the BOFH in me comes out because my reaction was, and still is: "he chose poorly". I hope the guy at least has learned from his experience -- it took me two fat-fingerings of "zpool destroy" and "dd if=/dev/zero" arguments (two separate occasions/unrelated incidents) to teach me to do backups properly/sanely. And I sure as hell aren't using shell scripts that involve rm -fr.
Init 1 instead of init q, that was always my favorite line that someone used to run on the main server, far too often as it turns out. On a Sun box that requires serial terminal to type "go", to recover from.
There's that brief moment of confusion, followed by the heart clench and a burst of fear in realizing what has just happened. Then the mind quickly scrambles for any and every solution that could possibly put things back to normal. Then you're hit by the icy cold wave of fear like being submerged in frigid water as you realize the ramifications of what has happened and predictions of what your increasingly bleak future may hold. After you realize all hope is lost you hit "fuck it" mode and just try to escape the situation in the outside world, but it's too late... that icy water you're submerged in goes with you.
"something about this story feels weird"
honestly, I got that feeling as soon as I started reading the blurb. but i'm just naturally suspicious of massively destructive computer screw-ups.
Not in linux. Files can be restored, but no paths. His business is fuckd
Funny how is number of users is 2^10 + 2^9 - 1. 'Lot of code smell in this article.
Computer Science is no more about computers than astronomy is about telescopes. --E. W. Dijkstra
Thankfully rm -rf doesn't work on ZFS snapshots...
If your livelyhood really depends on the data, you can invest time to recover it after rm -rf /. Come up with patterns to identify your database records on disk blocks. A few will be split between blocks which are non continuous, but most can be found and dumped from raw device. Then exploit relationship between different records to identify outdated data. Finally, offer suppliers and customers a healthy discount to log in and correct remaining errors. After all, the value of a business is in human relationships more than any single hard drive.
I fixed the article heading for you...
I work with ansible a lot and I'm pretty sure a shell command won't execute if you use any undefined variables in it, instead ansible will report an error.
It was a pretty obvious troll if you read the whole thread:
1. The guy claims to have made the most insanely improbable mistake to kill his entire set of servers. Possible, but unlikely. Most took the bait on this one.
2. He had no explanation as to why "--no-preserve-root" didn't save him - basically looked like he didn't know about it, and he was lying.
3. later on he responded to someone's suggestion to use dd to backup saying he reversed if and of - which is probably the second most joked about UNIX sysadmin error after "rm -rf".
So, either you are pretty clueless about any of this, or, like another poster suggested, you are also a troll. Based on the specifics of your moronic post, probably the latter.
-rf {foo}/{bar} = fubar foobär/ adjective adjective: fubar; adjective: foobar out of working order; seriously, perhaps irreparably, damaged. "the clock in the hall is fubar"
As part of "cost savings" back in 1991, managers decided that the secretary would be able to rotate the backup tapes, instead of IT staff.
After 6 months of hacking and developing, we had a crash of 2 disks simultaneously.
It turned out, all backups, offsite, offline, had all been made on cleaning tapes. Error messages had been ignored.
There is nothing new here. People and managers keep taking shortcuts and hope for the best.
is they fundraising here ? https://igg.me/at/3Lep-i7-ed0
rm - rf / does not work due to in the in-built - - preserve-root. However, rm - rf /* will do the job nicely, so either that second variable interpolates to * or he's running Ubuntu 7.10 or earlier. Or it's total BS.
you can recover many of those files if not overwritten with some restore utility. I have done that before.
I totally second that: http://docs.ansible.com/ansibl... Moderate parent to top level. rm is a tool, ansible too, badly using them is asking for trouble.
[Pruneau
http://meta.serverfault.com/questions/8696/what-to-do-with-the-rm-rf-hoax-question
Hopefully the company able to do backup atleast every month.
Everything in their right place, in their right time.
I saw this right away.... Lol
foo bar ??? Yup. It sure is.
Tweet, tweet, all id10t's out of the gene pool, open swim is over.
I once wanted to remove all files starting with some name. Instead of "rm name_* " I did "rm name_ *". That extra space deleted all my uncommitted code in that folder.
ServerFault Meta admins have confirmed that it was a deliberate "Guerilla Marketing" promotional hoax. Seems it was highly effective. The original thread has been deleted, though admins are still deciding what to do, if/how to punish user account, etc.
> 2. He had no explanation as to why "--no-preserve-root" didn't save him
Maybe because it's a BSD or other system without gnu tools?
Yes. This. PULL PULL PULL.
The system should not have write rights on the backup system. The backup system should have read rights on the system.
The dangers of excessive individualism are nothing compared to the oppressiveness of excessive collectivism
They had just introduced sensu for monitoring. The puppet driven installer wanted to know he host name.
Boom.
With ZFS you can make a snapshot, i.e. a freeze of the entire disk in time. This means you can rollback to any of these time points when you want. These snapshots does not use additional space. This also means that if some virus messes up your system disk, no worry, because you can just rollback to an earlier snapshot in time. I have made much mistakes with Solaris, but each time I am going to do something weird, I make a snapshot (takes one second) and if I mess up, I just reboot in GRUB to an earlier state where the system disk worked fine. ZFS is a godsend.
Guy needs to be a life member in the Error Checking is for Chumps club. /$path/*. Well $path wasn't defined. The guy called it through sudo. I had a record. No more sudo for him. Could still login as root, took an hour or so to fix.
I've seen very close, a chown/chmod -R
I constantly remind people on the rsnap mailing list about the existence of cryptolocker type malware.
Along those lines, could one write or modify a filesystem driver to add mount options which only allowed creation of files and directories (on, say, a snapshot-style-style backup volume), to combat those kinds of attacks?
Mount options may not do much good, because an attacker (or malware) could remount. Also for efficiency you often don't want to write everything again, you want to clone the old through some efficient mechanism such as hard links or volume tricks, then update it by updated files which have been updated, deleting deleted files, and creating any new files. A pull with read-only access to the live system does this well, and read-only is very well supported by existing file systems.
Use ZFS for drives...
See http://www.trymodern.com/artic...
Maybe because it's a BSD or other system without gnu tools?
Maybe you could RTFA. It wasn't.
--HOWTO safe rm on Linux (at least):
- Interactive:
o Use Midnight Commander. Works from a text terminal, no GUI needed.
Insert to mark files/dirs, F8 to delete. Never had a misfire, even as root.
- Script (for the paranoid, and want a log): /tmp/blah/1 " .. && rm -rv 1/* |tee /tmp/rm.txt
IF you have a known dir " destdir=/tmp/blah " with a subdir "
$ cd $destdir && cd 1 && cd
- REALLY safe rm, with find: /bin/rm -v {} \;
# find multiple (known) names of files > 40 days old in a given directory and delete them
bkpath="/mnt/bkpdrive"
pathh="$bkpath/work/bkpsys-laptop-p2400-thinkpad-xubuntu-14-04-LTS--64--sda7"
cd $pathh && \
find $pathh/* \( -name "bkp*gz" -o -name "bkp*bz2" -name "bkp*lzop" -o -name "flist*" \) -type f -mtime +40 -exec
--If you need to delete multiple levels of subdirectories, or dotfiles -- do it interactively, and use MC.
.
== WolfriderV6 == I'm willing to admit that *I just might* be wrong... Are you??
Maybe it was a hoax. Maybe it WAS.
Yes, it was a hoax. It WAS. :)
http://www.snopes.com/man-dele...
I know this article is "old" by now, but, it's already been 100% proven hoax, confirmed by the original perpetrator:
http://www.snopes.com/man-dele... /. lack of any technical editors or researchers strikes again!