Man Deletes His Entire Company With One Line of Bad Code (independent.co.uk)

Reader JustAnotherOldGuy writes: Marco Marsala appears to have deleted his entire company with one mistaken piece of code. By accidentally telling his computer to delete everything in his servers, the hosting provider has seemingly removed all trace of his company and the websites that he looks after for his customers. Marsala wrote on a Centos help forum, "I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line. All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)." The terse "rm -rf" is so famously destructive that it has become a joke within some computing circles, but not to this guy. Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?"Rm -rf" would mark the block as empty, and if the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.

Comments

Three words

[MPAB]

Offsite, offline BACKUPS

Re:Three words

[Nutria]

Multiple off-site backups. Multiple, rotating off-site backups. Weekseven if something happens to the on-site tapes you've still got backups.

Honestly, WTF is it about the PC/Internet mentality that makes sysadmins soooo stupid? Enterprises figured this out FIFTY YEARS AGO.

Re: Three words

[SumDog]

He capitalized the wrong word. BACKUPS instead of OFFLINE

Re:Three words

[Aighearach]

That's all great, but even a less complete, sloppy backup system would be an improvement here.

Another thing people don't understand about cloud hosting... you should still have your own self-managed, non-cloud server that holds your images and ideally runs your service during the low-traffic hours. Whatever your daily lowest traffic 6 hours is, in most cases, should be traditionally hosted. Cloud is super-duper-awesome-webscale for the peak traffic, no way around that if you have peak traffic hours.

Personally, I can re-deploy (including the latest database backup) from my dev workstation using a simple rake task.

Another problem is; relying on your hosting company for backups. Never do that. The same fire/earthquate/bash script/volcano that makes the backup necessary, would destroy it! Expect the hosting company to have insurance, don't expect them to care if your data gets lost. Especially if it "user error."

This has nothing to do with "PC/internet mentality" and everything to do with the latest anti-waterfall, anti-planning, 80% is all that matters mindset. Traditionally, this was easily solved because there was an engineering mindset.

Re: Three words

[thegarbz]

I tried offline backups once, but I couldn't figure out how to get the data off my computer and on the backup without first putting the backup online.

Re:Three words

[lgw]

I have to disagree here a bit. Not with the idea of doing backups -- everyone should -- but that's looking at the half problem the wrong way. It's the right solution for customer data, but not for all the code and other materials that make your web site happen.

I've seen this problem a lot: all the work product that makes a web presence happen gets done on the hosted server. That's beyond stupid - that's failing to even understand your job.

All the work that goes into your hosted web site -- your store, your code that aggregates or helps the customer in whatever way makes you valuable, all that stuff -- needs to live in a version control system you control locally. Ideally github, so backups are free, but not everyone can do that. Your entire web presence other than customer data should be pushed from where the real work is done, and of course there should bee a way to revert as well.

When you look at it that way, it's obvious that a key place to replicate your customer data to is close to the machines you do your build/push work from (not the same machines, unless you have strong read-only protection, but close). That way, if your hosting provider takes your site down on a whim, a couple of scripts you already have give you the same web site with the same data at a new hosting provider. That also makes you safe against physical server failure, rm -rf, and anything else that happens in the cloud.

This isn't rocket science, it the minimum standard that separates amateur from professional.

Re: Three words

[GameboyRMH]

It's true, if you have only one offline backup there are short periods of total risk involved. If you have more than one offline backup though, you just do one at a time and there's never any window of time when all your backups could be taken out by a lightning strike/malware/hilarious accidental deletion.

With newer versions of rm you can also use --preserve-root to prevent this sort of thing from happening.

Re:Three words

[TheRaven64]

In this context, the guy is the cloud provider. His customers, if they're sensible, will have their own backups and so will be able to recover, but they also won't trust his business much if that's their recovery strategy from his incompetence.

Even with online backups, there's no way that this should happen. The backup system should be taking read-only snapshots at periodic intervals, so even if you rm -rf you'll only delete the live data and be able to revert to the snapshot from an hour ago.

Re: Three words

[GameboyRMH]

Addendum - just checked a CentOS server, and rm --help says that --preserve-root is enabled by default, and has to be overridden with --no-preserve-root.

Re:Three words

[tnk1]

Offsite, tape backups aren't even really all that necessary. You just need any backup that you can't use one command in the system to delete all of your data.

You could use AWS S3, and just use something like Glacier to back up your data. Since it takes like 4 hours for it to be rotated back into being online, you have about the same effect.

Also, while offsite backups are useful, for a host with 1,535 customers, who are all making changes, even if you have a daily offsite tape backup, you could find yourself missing a lot of changes. I suppose it is better than losing all of the sites, but I'd think that some simple precautions to ensure your online backups were not immediately delete-able with one easy command would pay much higher dividends.

Re:Three words

[flopsquad]

Offsite, offline BACKUPS

Would not have helped in this situation. His typo resulted in this command:

"rm -rf --no-preserve-root --write-zeroes --shred-mbr --exec-all-ssh-hosts --douse-hydrofluoric --high-velocity-eject-removable-media --carpet-bomb-offsite-backup --salt-earth"

Which, I mean, who hasn't accidentally done that? The keys are like right next to each other.

Re:Three words

[ShanghaiBill]

Traditionally, this was easily solved because there was an engineering mindset.

You seem to be implying that data loss was less common in the "Good ole' days", when all sys admins were highly trained engineers. That is almost certainly untrue, and based on false nostalgia. Backups are much easier today, with reliable high-capacity storage, journaling file systems, ubiquitous connectivity, and plenty of off-the-shelf software solutions.

Re:Three words

[Megane]

Because he is a retard.

All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script

Clearly a case of a fool thinking that a sync (copying data to another place regularly) is a backup. It's not a backup if you can easily copy corrupted data to your only copy. Or, in this case, if you can easily delete the data from your "backup" copy.

Re:Three words

[The MAZZTer]

Three words (Score:1, Redundant)
Offsite, offline BACKUPS

Make them Redundant backups too? Good idea.

Re:Three words

[Aighearach]

Some projects I worked on in the 90s still have tape archives of that data.

You can easily have a situation where the backup tools have improved, and there is less overall data loss now, but that the mindset now is sloppy and leads to a lot of errors of types that were less common in the past.

In the past when you did it sloppy, you'd get called out on it; and sometimes it still sucked, because PHB. But when that was the case, it was at least known and accepted that it was technically inferior to not have correct engineering. These days, the average shop believes that 80% is enough, and that 95% completion is too much and a waste of money. In the old days, there was technical consensus that 100% of the desired functionality... was desired.

Re:Three words

[Aighearach]

In my experience, most of the customers of small hosting companies are paying for fully managed servers, which includes the backups. Most of the customers won't have any backup other than the code they started with. And they wouldn't know how to make a backup any more than they would know how to shoot a fireball spell out of a chopstick.

This is compounded by human nature applying "trust" based on the quality of the personal relationship you have. If you have a nice conversation, by the end they really really want to give you their root password, have you move all their stuff over for them, and just tell them when it is finished. And then their DNS hasn't propagated yet, and they get really upset and become unsure if they should "trust" you, and get indigestion, and start calling every hour.

The "mounted backup" part is just a bridge too far. Later in the comments he says he swapped of/if on a dd command, so now how does he prep the disk for recovery, which seems to verify the troll.

Re:Three words

[hey!]

Four words: filesystem with automatic snapshots.

I've never admined a major customer linux installation myself, but as a developer I've been called into rescue customers who messed up their databases, and let me tell you being able to root through the transaction log and undo mistakes like "delete * from foo where conditionThatIsAlwaysTrue" is a lifesaver. Oracle, which is a company I despise for a number of reasons, does a really good job of that.

The rule for production systems should be "never work without a safety net".

Re:Three words

[Salgak1]

Three MORE words:

Teh Burning Stoopid . . .

Re:Three words

[cyber-vandal]

Where did you work where that mentality didn't exist? I worked for quite a few very large organisations back in the day and "put it live, we'll fix the bugs as we go" was the order of the day, usually after 2 years of shambolic waterfall development and ever-changing requirements.

Re:Three words

[billyoc903]

I have this aliased to 'sl'. Keeps me on my toes.

Re:Three words

[geekmux]

Offsite, offline BACKUPS

Would not have helped in this situation. His typo resulted in this command: "rm -rf --no-preserve-root --write-zeroes --shred-mbr --exec-all-ssh-hosts --douse-hydrofluoric --high-velocity-eject-removable-media --carpet-bomb-offsite-backup --salt-earth" Which, I mean, who hasn't accidentally done that? The keys are like right next to each other.

Man, I haven't laughed out loud like that in a long time. Thank you for that.

Re:Three words

[Aighearach]

If you had changing requirements, you weren't doing waterfall. Sorry.

Re:Three words

[Anonymous Coward]

Well, there are two ways I can think of to accidentally trigger that command.
One is when you are trying to remember how to exit vi and are trying different shortcuts.
The other one is if you slip and press that hotkey in emacs.

Re:Three words

[ShanghaiBill]

In the past when you did it sloppy, you'd get called out on it

I have been in tech for 30+ years, and I have seen no evidence whatsoever that sys admins were less sloppy in the past, nor do I believe that management was better at "calling them out" when they made mistakes. Backups and reliability in particular are way better today.

Every generation tends to believe that young'ins are dumber and lazier than they were. They are usually wrong.

Re:Three words

[Triklyn]

... are you suggesting that there's someone out there that knows how to shoot a fireball out of a chopstick?

please elaborate on that

Re:Three words

[Anonymous Coward]

If you properly cooked the earth, you wouldn't need to salt it.

--sf

Re:Three words

[MachineShedFred]

Or, how about "test before running EVERYWHERE"

Spin up a VM, run your script. If the VM eats itself, you're out of a few minutes of time instead of your entire company and legal jeopardy from your customers.

Re: Three words

[MachineShedFred]

The script was deployed via Ansible. He ran it on everything at once.

And everyone knows that blind deletes via automated scripts are a fantastic idea.

Re:Three words

[Cramer]

Come on, who hasn't done that at least once in their long carrier? Hell, I've done it on purpose a time or two. (it's actually rather rare for it to actually finish, or it used to be... shortly after rm and libc are deleted, things start Going Wrong(tm).)

Re: Three words

[kurkosdr]

Or, an OS which offers decent automation tools, so you won't have to hand cruft bash scripts as if it's 1989 again. There is a reason Windows Server still makes Microsoft money despite linux supposedly being the OS of choice for servers.

Re:Three words

[Eloking]

Which, I mean, who hasn't accidentally done that? The keys are like right next to each other.

I'm quite sure this is from an old bash.org post.

Re:Three words

[Darinbob]

Three words. Blame the intern.

Re:Three words

[mwvdlee]

Why would you ever need or even want to mount your backup storage?

Re:Three words

[RabidReindeer]

Probably waterfall as in "Git 'er Dun!", shovel in changes to System X, ram them into production, sorry, no time to clean up or put in reliability, security or backup/recovery processes, got to get the next waterfall "Dun!" Then after a while, someone comes and complains that System X doesn't do what it needs to do, drag it back out and send it over the waterfall again.

Re:Three words

[Archangel Michael]

Minimums:

3 Copies
2 Locations
2 Formats
2 Mediums

Copies, two local, one remote
Locations, geographically distinct
Formats Natural, Raw, compress etc
Mediums, SATA, USB, Tape, SAN manufacturer etc.

By Minimum I mean bare minimum. the reality is, there should be cascading copies being made, and Long Term Arching able to restore to a set point in time. For Copies you'll need at least three, more likely more version (date specific). You should separate your copies geographically so that when California gets the big one, or Hurricane Global Warming washes eastern seaboard clean or Tsunami wipes out the Pacific Rim, you can resume business relatively quickly somewhere else. Different formats so that you can get the data you need in a way that makes it easy. You'll want the important parts of the SQL database in a non-database (XLS) format. And you'll want to isolate yourself from medium failures, a date bug in SAN, or Trying to find a floppy drive (old school) to put the floppy in.

The problem with this guy, was that he was too cocky and didn't have proper backups. IMHO if he did "Live > Local Backup > Offsite Backup he would have been fine. You backup your live data locally, and then make a copy of that backup to remote / offsite. Three copies, two locations, two media, two formats. Done

Re:Three words

[RabidReindeer]

Past, present. You get called out on sloppy when it all goes to hell. Then it's your fault, you were sloppy.

The rest of the time, if you try and be meticulous, you're being "unproductive". Nobody got time for unproductive. Costs too much money, too.

Re:Three words

[RabidReindeer]

A version control system is nothing more than a specialized form of differential backups.

And something had better be backing up the VCS itself!

Re:Three words

[Tuidjy]

I have not done that one, but I did something close.

When I came to the US, in the early 90s, I brought all the code I had ever written with me. It included some pretty interesting stuff (I did a lot of assembly programming in Bulgaria, in the late 80s) so it was compressed, encrypted, and contained a second, hidden archive that contained the really 'interesting' stuff.

I forgot the password... and I forgot it in the dumbest possible way - it was a longish English phrase, and my English at the time was bad enough that I did not know how to spell one of the words. By the time I needed some of the stuff, I had learned the correct spelling, and could not get to my code.

Much, much later, around 2012, I found the hard drive, and miraculously, it worked. I brute forced it, which I could have done much earlier, but it was never important enough, and it turned out that I had misspelled 'gnarly'. I had some good times going through my 30 year old stuff, as well as seeing what I considered indispensable software at the time. It's hard to believe that we used to write programs that did useful stuff and were less than 256 bytes long.

Re:Three words

[riis138]

Yup, its IT 101. Something about this seems fishy.

Re:Three words

[bruce_the_loon]

Whenever I think of what waterfall development can do to a project, the opening sequence of The Mission comes to mind. https://www.youtube.com/watch?v=xceFQWV3lMM

Re:Three words

[mlts]

I'd say backups and reliability are not easier or harder... but different. Back in the 1990s, there were drive failures, but there were the people who manage to get root and wipe the box for the hell of it. Because of this, people would back up to tape, physically write protect the cartridges so they couldn't be erased, and put them in a safe or have them go offsite.

When things changed about ten years ago where active hacks were not as common, the focus went to dealing with drive, path, and other hardware failures. For this case, having RAID and a secondary SAN offsite that did async replications.

Now, with ransomware, people are realizing that RAID isn't a backup, even if it allows for deduplication. That secondary array just will happily take the zeroing commands and apply them.

Maybe there might be a renaissance for tape. Hard disks are not really a backup medium because malware can zero it out almost instantaneously. Same with shares. Optical is cheap, but in general, the capacity is too small for all for most things. Which brings tape. If someone could bring a tape drive with LTO-6 or 7 capacity to the market at a consumer level price ($1000-1500), has built in encryption, decent backup software, LTFS, compression, and can work on a USB 3 port without shoe-shining itself to oblivion, it just might be a popular seller.

Re:Three words

[crunchygranola]

If you had changing requirements, you weren't doing waterfall. Sorry.

The corollary to this is that nobody ever did waterfall, really - and that is pretty much the truth.

They just pretended they were doing it. Agile methods admit the truth.

Re:Three words

[fyngyrz]

Why would you ever need or even want to mount your backup storage?

I bought my storage from realdoll.com, you insensitive clod!

Re:Three words

[myowntrueself]

Offsite, offline BACKUPS

You'd have to add 'multiple redundant' to that.

This guy did have offline backups which were online in order to actually do a backup. If he had two independent sets of backups he'd have been ok. Or, you know, check that the variables weren't empty in the shell script.

Re:Three words

[myowntrueself]

Also, while offsite backups are useful, for a host with 1,535 customers, who are all making changes, even if you have a daily offsite tape backup, you could find yourself missing a lot of changes. I suppose it is better than losing all of the sites, but I'd think that some simple precautions to ensure your online backups were not immediately delete-able with one easy command would pay much higher dividends.

To be fair, the customers should be making changes, keeping those in a revision control system, backing that up, and uploading to the hosting provider. Never rely on the hosting provider for backups.

Re:Three words

[cerberusti]

I did a chmod 700 / home/me on a production server a couple of years ago as root after setting another user up (note the space typo).

Total downtime was about two minutes, as I figured it out pretty quickly when it did not complete immediately and I hit up to look at my last command.

A chmod 777 which I let get about as far made it work temporarily, and fortunately RPM actually has a setting to restore permissions... but it was very inconvenient as there are rather a lot of permissions not set by that. I ended up setting up another server and not daring to restart that one.

rm I am careful enough with that I have never had a problem, but I also stare at that command for a minute before I hit enter if I am root.

Re:Three words

[E-Rock]

This. A replica is not a backup.

Re:Three words

[budgenator]

You'll only get the chopstick wand fireball spell when you achieve level 5 Sys-Admin.

Re:Three words

[AmiMoJo]

Something like Google Nearline or Amazon S3 would be ideal for this. Easy to set up, scales and it's cheap.

Re:Three words

[budgenator]

Format C: press any key to continue

Re:Three words

[swb]

As I remember it, we went to backups a hell of a lot more in the 1990s than we do now. I think some of it may have just been a software reliability thing where PC based networks and software were just a whole lot less reliable -- direct-access (often shared) databases that shit on files from time to time, applications that wrote gibberish and other types of problems.

I don't think I ever worked in IT where even PC servers weren't backed with RAID, even if it was tiny 5x1 GB arrays in 1995. They were mostly reliable at surviving disk failures, but I can remember a specific HP model that required the system offline to rebuild a failed drive -- there was no auto-rebuild mode, you had to hit a function key or boot with a floppy to get the thing to rebuild.

I'm amazed to this day at the people who insist on automated replication as a solution for anything but site failure. It's like nobody thinks that the most *likely* problems aren't going to be catastrophic storage system failures, but some other failure mode that results in all those problems happily replicating everywhere. I do like asynchronous replication of virtual machines (with adjustable retention history, so I can get the previous version with no issues), but that's usually a software product not a storage feature -- storage doesn't do it will unless you have the patience for a zillion VM-size volumes and replicate them separately.

I agree that an LTO-6 capacity drive that would do USB3 would be great. Flash density is getting to the point where it would probably make it a lot more user friendly to just have 128 GB flash on the tape with a detailed index to allow for faster access and some level of random access to the data, or at least write buffering.

Re:Three words

[LinuxIsGarbage]

How about a special interface for harddrives. Basically like this
Flip a switch and the drive is writable to complete the backup. Flip the switch and the drive can only be mounted read-only.

Re:Three words

[Rei]

When I was just learning Linux, a friend told me the trick for playing wav files by running "dd if=/path/file.wav of=/dev/dsp". Neat, I thought! But I didn't try it out right away, and by the time I did I didn't remember the name of the sound device. "But that's easy," I thought, "I just want the first sound device, so sda, right?" "dd if=/path/file.wav of=/dev/sda"...

Yeah, that wasn't a good day...

For most people back then it wouldn't have been that bad because that was pre-SATA and most people used IDE drives (/dev/hd*). But lucky me, I had a Seagate Cheetah which I was very proud of (10k RPM, wow!), and so had all my most important files on it... :

Re:Three words

[Anonymous Coward]

Sounds like old-school Solaris to me. Nowadays Linux does one better by adding --nuke-from-orbit but it's not POSIX compliant. Also, it requires systemd for some reason.

Re:Three words

[david_thornley]

At home, I've got a lot of stuff in a local Mercurial repository. It's version control. It isn't a backup, because if anything hoses the disk I'm using it hoses the repository as well. It needs to (and in my case does) have backup.

Re:Three words

[david_thornley]

If the customers were IT-savvy enough to keep their own transaction backups, they'd probably be IT-savvy enough not to trust any service where the idiot running it can run one command and wipe out all the data, including the backups.

It's simply not practical to expect every small organization to have an IT person. Such organizations have to trust somebody.

Re:Three words

[magarity]

Format C: press any key to continue

That one's foolproof; I have yet to ever format C: because my keyboard doesn't have an any key.

Re:Three words

[lgw]

Sure. It's nice if you can use github, since that's your offsite backup, but lot of people can't. The main thing is, your local site is the "backup" of the production server, at least if you can launch a new prod server quickly with a script.

Re:Three words

[Locke2005]

I have never done the "rm -rf" cliche, but I have done something stupid: Rebooted a Sun server while logged in remotely from a workstation, not anticipating that I would have to go to the server room anyway and log in locally to bring it back up because I would no longer be able to log in remotely.

Re:Three words

[Locke2005]

I didn't write 256 byte long code, but I did write EPROM code that ran a hand-held terminal with only 256 bytes of ram built into the microcontroller.

Re:Three words

[spire3661]

There is no logical way around "Trust, but verify". If you are a business that uses computers, you NEED an IT person like you need an accountant. They dont have to be in-house, but you better have one at the ready.

Re:Three words

[spire3661]

Can anyone explain why the verbose arguments require 2 dashes?

Re: Three words

[spire3661]

Get the fuck out of here.

Re:Three words

[flopsquad]

Which, I mean, who hasn't accidentally done that? The keys are like right next to each other.

I'm quite sure this is from an old bash.org post.

Right you are.

<tatclass> YOU ALL SUCK DICK
<tatclass> er.
<tatclass> hi.
<andy\code> A common typo.
<tatclass> the keys are like right next to each other.

Re:Three words

[Mr0bvious]

What's that? Have your backups mounted on the same machine?

Hmm backups are supposed to save your ass in situations like this. Hence the reason to have it offline (and more than one copy).

The error/incompetence was not the 'rm -rf', that shit happens all the time. It was the lack of any *real* backup.

Having the one backup mounted on the same machine (IMO) is a misnomer.. That's what I call a "second copy", that's just as vulnerable to loss as the original copy (for some loss vectors).

So, yeah I expect there's been a number of us do such a silly thing before.. But it's still incompetence.

Even I have 2 offsite backups of my own personal data on a week rotation system. It's really not difficult to do.

This guy was looking after the data for over 1500 people... That's incompetence.

Re:Three words

[flopsquad]

There is an exhaustive discussion here, but the crux of the biscuit is the parser (and conventions about how it should do its thing).

Re:Three words

[mlts]

This. Or, if one wanted to be more sophisticated, have a drive controller that would not allow writes to blocks once they were written to. Combine this with the UDF filesystem, and you now have quite usable WORM media.

Ironic thing, it used to be that all SCSI drives had a jumper to flip them read-only. I used this for anonymous FTP servers back in the 1990s to ensure that even if the server was breached, the files wouldn't be able to be tampered with.

Re:Three words

[Eloking]

Which, I mean, who hasn't accidentally done that? The keys are like right next to each other.

I'm quite sure this is from an old bash.org post.

Right you are.

<tatclass> YOU ALL SUCK DICK

<tatclass> er.

<tatclass> hi.

<andy\code> A common typo.

<tatclass> the keys are like right next to each other.

I FUCKING knew it!!!

Brb I'll go check some old memory, been too long.

Re:Three words

[mlts]

The thing about automated replication... salespeople sell it as "cool", and "not your father's tape drive", and people who are not familar with IT, but have the purse strings buy it. People are proud that they toss the tape silos and have new forms of SAN storage like Tintris. It has its uses -- for example, when dealing with virtual machines and upgrades.

Backup options just seem to have gotten worse over the years, especially for home users.

Now for blue-sky stuff:

With SSDs coming down in price, I've wondered about having a LTO tape drive which would have SSD space about twice the size of the tape volume (and a good amount more to help with wear levelling, sector relocation, etc.) When a write is done, the drive can be configured to return that the write was successful either when the write finished with the SSD, or when the write to the tape is complete. Upon eject, the entire SSD is TRIMmed with something like "blkdiscard -s", which generates a new volume encryption key, making all existing data inaccessible. This way, if a tape has sensitive stuff on it, as soon as the drive is power cycled or the tape ejected, the data on the SSD would be gone after all I/O is complete.

For reading a tape, the SSD could act as a cache, so if one restores a file multiple times or does random read I/O, the drive can just read the entire tape onto the SSD, then hand the data from that.

As an added bonus, this makes LTFS into a very usable filesystem.

Of course, there would be some options needed, so that one could turn off the read/write SSD cache if the data written or read is hyper-sensitive and shouldn't hit anywhere but the tape. However, by adding random access and letting the tape drive do the rest, this would not just make the drive faster, but could allow someone to buy it, hook it up to a laptop via USB 2.0 and reliably back it up, with the cache ensuring that the tape drive runs at full tilt 100% without any slowdowns or shoe-shining.

Re: Three words

[valdezjuan]

I did something similar back in the day. I was still getting used to Solaris and ran killall, thinking I was on a Linux server (which just returns the usage). That was a pretty awkward day as I made a mad dash to the colo.

Re:Three words

[well_in_theory]

Suicide Linux; where any typo (as in resulting in command not found) instigates a full 'sudo rm -rf /'. Available as a debian package. https://qntm.org/suicide

Re:Three words

[YukariHirai]

Four words: filesystem with automatic snapshots.

That's very good to have, but should not by any means be all you're relying on. Probably wouldn't have helped the guy in this story, given all the other boneheaded things about his setup.

Re:Three words

[Cytotoxic]

Yeah, I did that setting up my first ever hosts security in Linux back in the old slackware days. I was on a remote terminal session and being careful - making one change at a time.

So the first thing I did was put DENY ALL in an empty hosts file. A millisecond before I hit save I had a thought.... ooops. And the walk of shame to the console ensued. Didn't make that mistake again.

Re:Three words

[DrJimbo]

Can anyone explain why the verbose arguments require 2 dashes?

It is to prevent name-space collisions when you "stack" short options as in "-rf" (which means the same thing as "-r -f"). The convention often used is that multiple letters after a single dash are stacked single character options while multiple letters after two dashes is a single verbose option. This convention makes things easier for the user and the designer because neither one has to worry about being able to spell out a verbose option with single character options.

Re:Three words

[mwvdlee]

You do know constant plugging and unplugging of storage will wear out the connectors, right?

Re: Three words

[invictusvoyd]

It will still "rm" a lot of other data which if not backed up could be unrecoverable.

Re: Three words

[stealth_finger]

You get a pen and write down the 1/0s

Re:Three words

[jabuzz]

Given in the UK that it is a legal requirement to retain the last seven years of financial records in case HMRC decides to do a tax inspection, and I imagine that most countries are the same.

Also in the higher education sector (where I work) all research data has to be keep for at least 10 years. Again I imagine most sane countries are also the same.

In addition law firms have requirements for retention of documents that go well beyond two years. There is good reason why documents going back decades could be appropriated from Mossack Fonseca.

Re: Three words

[Billly Gates]

What do you mean leave tape 10 years ago! Shoot I have to manually come into work every day for LTO3 tapes on our Windows Server 2000 and 2003 backup systems. Data worth millions. Nothing important or anything

Re:Three words

[swb]

I'd like a few things...

A tape cartridge with enough solid state storage such that the drawbacks of LTFS like slow seek times can be avoided for frequently read files and the ability to re-coalesce data on the tape to avoid holes created by deleting files from a linear mechanism.

A filesystem that does hierarchical storage management, where stale data is dumped off to tape drives but where links remain in the original file system allowing access to the data as if it was on disk. This allows you to have a much larger data pool in a unified filesystem. A changer would help here, but I could live with a prompt telling me to insert tape X.

Re:Three words

[Rei]

Haha, reminds me of when I was coding for a LP MUD. I ended up getting into a sort of "wizard battle" with other coders... one person would make a "dest" (kick you off the server) tool, the other would write a tool to counter it, the other would write a tool to counter that, and so forth. It gets complicated fast. Anyway, I was trying out a new tool and due to a bug it thought I was attacking myself, and dested me in response and then fell on the floor. It then proceeded to dest anyone else in the room on sight. Said room just happened to be the login room, so nobody could log back in - and the person who had access to the server for a reboot was rarely available.

Not good! However, from my FTP access I was able to determine that there was still one coder logged on, in a different room and blissfully unaware of what had transpired, still actively uploading and changing files. But I had no way to contact him. However, I noticed that I had write access to the directory where he was modifying files, so I created a file with a name like "AMMON - DO NOT LOG OFF OR GO TO THE LOGIN ROOM - PLEASE READ ME.txt", with a summary of the problem and the solution to fix it. About 20 minutes later he noticed the file, destroyed my buggy object, and everyone could log back on.

All I can say is that I'm glad it happened with that version of the object, because the next version I had been planning to make was going to try to counter attempts to hide from the player search function by means of rapidly teleporting from room to room and object to object, looking for characteristics that matched the target - a sort of hunter-killer drone. Needless to say, that version got cancelled ;)

Re: Three words

[Eunuchswear]

In other words the story is bullshit -- which is perhaps why it is no longer on server fault.

http://serverfault.com/questions/769357/recovering-from-a-rm-rf

Re:Three words

[david_thornley]

And we're now in the problem of a non-IT person trying to judge an IT person. The idiot's clients doubtless thought he was a decent one.

What would really help here is certification. If I wanted an accountant, I'd go to a CPA, so I'd be sure of getting at least a mediocre one. When I need a lawyer, I ask the local bar association for a referral, and get one who may be mediocre but is at least adequate for what I need legal help for. If there were a certification process that would say "this IT guy is not an idiot", it would help the non-IT people.

I know there's a lot of arguments about it, but this is one case where it would be very useful.

Re:Three words

[Ol Olsoc]

In this context, the guy is the cloud provider. His customers, if they're sensible, will have their own backups and so will be able to recover, but they also won't trust his business much if that's their recovery strategy from his incompetence.

So you back up in the cloud, then back up just like you used to before the cloud?

This cloud is an amazing thing that allows us to add another level of work.

Do you back up your local backups in the cloud, and back up your cloud backup locally? We need to get the cloud in the cloud, now that would be the ultimate.

Re:Three words

[Fetko]

SysAdmin hardcore mode.

Re:Three words

[tigersha]

You forgot --nuke-the-site-from-orbit

Re:Three words

[spire3661]

Just no. CPAs are certified because of how OLD the profession is and the mountain of actual malfeasance they have caused. Its a very different thing. CPAs have incredibly detailed rules about how to do their work. They have a standard model they must follow. That would kill good IT. You would hamstring us all so idiots dont have to think.

Re:Three words

[doccus]

This cloud is an amazing thing that allows us to add another level of work....

The "paperless society" .. Hyuk! Remember that? Added another entirely redundant level of work.

Re: Three words

[thegarbz]

No it won't. It will not delete a single file. It will throw a warning in the console and remind you that it's a dangerous operation and ask you to retype the command with --no-preserve-root.

Re: Three words

[Cabriel]

Sounds like something Dana would have done. :) Oh Kesac. The man, the Myth, the Legend.

If this doesn't make sense, you probably aren't who I was expecting.

Re:Three words

[Ol Olsoc]

This cloud is an amazing thing that allows us to add another level of work....

The "paperless society" .. Hyuk! Remember that? Added another entirely redundant level of work.

I have many containers of the paperless society. And at work, as one piece of paper was eliminated, two took it's place.

Re: Three words

[dgallard]

The more apt analogy is of course to forget the WHERE clause in: DELETE FROM ;

Re:Three words

[Anpatt7]

Aw, I wanted to do that...

Re:Three words

[TheRaven64]

A good backup strategy has both on-site and off-site backups. If either is broken then the other can be used to recover.

Re:Three words

[Ol Olsoc]

A good backup strategy has both on-site and off-site backups. If either is broken then the other can be used to recover.

So now instead of the cloud as your everything solution, you have to do exactly what you did before, and add the cloud as just extra work.

Quite the bill of goods.

Re:Three words

[TheRaven64]

Where were you doing off-site backups before?

Re:Three words

[MoarSauce123]

Yep! And before rfing anything make yet another backup locally, even if it on low tier consumer grade storage. Plus, no matter what it is, never run a command on all servers. Easy to say now, but I bet we all got burned by our own sloppy work. And those who have not, now is the time to craft check lists and start using them.

Re:Three words

[Ol Olsoc]

Where were you doing off-site backups before?

local multiple backups under my control stored in another building.

Accountability, not just another customer. It works pretty well

Re:Three words

[david_thornley]

so idiots dont have to think.

No, so people not in IT don't have to learn a lot about IT. A bad IT person can do a lot of harm, and a non-IT person is going to have a lot of trouble telling a good one from a bad one. This is a real problem, and it would be nice to have some approach towards solving it.

If you don't like the idea of certification, do you have another idea to allow an entrepeneur to recognize a competent IT person?

Certifications don't have to come with smothering rules. As I said, I got a lawyer referral from the local bar association, and this means I got somebody reasonably competent (probably not brilliant). My lawyer needs to know what's going on, and how to get me through the court system without serious mistakes. He does not follow rigid rules; his job is to get me not convicted for a misdemeanor I didn't commit.

--no-preserve-root

[zopper]

Does he use --no-preserve-root by default? I think that it is there for many years. Of course, if his servers are running on something from 2004, then his rm might be without this safeguard...

Re:--no-preserve-root

[Anonymous Coward]

That option doesn't work if you are trolling...

Re:--no-preserve-root

[mysidia]

This prevents the root itself from being deleted, but you can still do rm -rf /* even with no --no-preserve-root There are iterations which still accidentally cause a full system deletion, even with this safeguard in place.

Re:--no-preserve-root

[piojo]

You can't prevent the computer from doing what the user tells it to in all cases.

For what it's worth, removing "*" recursively is a weird use case (99% of the time, you would just delete the directory), so "rm -r $foo/*" -> "rm -r /*" is unlikely to occur. (For a vaccine, 99% is a pretty damn good rate of protection.)

If "rm *" or "rm foo/*" were disallowed by all shells (if "rm" were removed as a command and only implemented as a shell builtin), this problem could be eradicated. A more difficult workaround would be to use a programming language with stronger typing, so arguments would not be computed by string interpolation. For example, if the guy's code had run "dir = Path.Combine(foo, bar)" and then deleted the contents of that dir, an exception would have stopped execution before ever getting to the line that does the deletion.

Re:--no-preserve-root

[mysidia]

For what it's worth, removing "*" recursively is a weird use case (99% of the time, you would just delete the directory),

Is that 99% supposed to be based on actual research? I have seen developers commit many accidents involving the rm command and shell globbing. Also, many newbies using Shell globbing when it is not needed, and command options when they are not needed such as "-r", Or running as a root user when they should not be.

One example is the user types 'rm -rf * .o' when they intended to key 'rm -rf *.o'

If their Current Working Directory happens to be /, then it's equivalent to rm -rf /*.

Re:--no-preserve-root

[allo]

A shell is not built to prevent you from shooting yourself into the foot. It's built to optimize the aiming, but let's you decide where you aim at.

But nothing prevents you from using an alias or function in your shell or replace rm by a script, which calls rm.real. You may find yourself at getting the habit to type rm.real all the time. Ask yourself, why people use "-f" all the time. They get used to it, because distros alias "rm" to "rm -i", but "-f" overrides this.

Re:--no-preserve-root

[piojo]

I think we mostly agree. You'll note I didn't suggest making it harder to recursively delete "*/". I suggested disallowing it completely. Directories could be removed recursively, but not emptied recursively. "rm -r $foo" does not present the same problem as "rm -r $foo/*" when foo is undefined. "rm -r /$foo" would still present a problem, but --preserve-root mitigates that, plus who would store an absolute path as a relative path like that? (Okay, I'm sure it happens every day, but it reduces the likelihood of this problem.)

Re:--no-preserve-root

[piojo]

Yes, that's what I'm saying. The cases where you legitimately want to recursively delete a directory's contents, but not the directory itself or its dotfiles, are much rarer than the cases where you just need "rm -r dir" without globbing.

I just made up the 99% figure, but since you're not contradicting me or providing a contradicting use case, is that agreement?

Re:--no-preserve-root

[mysidia]

I just made up the 99% figure, but since you're not contradicting me or providing a contradicting use case, is that agreement?

Nope. I figure it's closer to 50% of the time rm -rf $foo/ and 40% of the time rm -rf $foo/* and 10% of the time rm -rf $foo/*${something}*

Re:--no-preserve-root

[piojo]

rm -rf $foo/*${something}*

Oh, I didn't think of that one. However, it's still distinct from $foo/*, as the shell can tell the difference even when the variable is empty or unset (as long as "rm" is a shell builtin). However, that made me think of a much bigger problem: "rm $foo/*/" to get rid of directories. This is a common use case (for me), so I cannot think of a non-restrictive check will prevent this from wiping out the root folders when $foo is unset.

What do you use "rm $foo/*" for? I find that if I want to empty a directory, I generally want it deleted as well.

Re:--no-preserve-root

[mysidia]

What do you use "rm $foo/*" for? I find that if I want to empty a directory, I generally want it deleted as well.

Maybe don't want to delete the directory if it contains temporary or spool files, for example
rm -rf $tmpdir/*

Of course it would be possible to design new safeguards, But I think they're unlikely to be implemented...
1. Shell modification: When a variable is unset Or set to Blank, the shell should return an error by default, instead of executing the command with a blank; require a declarative structure such as ${tmpdir:-""}
OR Shell modification with /bin/rm modification:
2. The shell is modified to export an environment variable containing the *ACTUAL* command line After variable expansion, and after compression of globbing E.g. /** to /*, but before "list of files" expansion.
3. Rm is modified to check if a subdirectory of / is referenced; If so, the variable is inspected for shell globbing. Rm is modified to require a command line option to proceed with the delete if files in / or an immediate subdirectory were expanded by Shell globbing
OR /bin/rm modification:
4. Instead of just checking for "/" check for the 2nd level as well. Any directory directly under / has a --no-preserve-first that also has to be used, instead of just a --no-preserve root.

5. This gets expanded with an /etc file listing critical system directories. Any recursive deletion of the root of a critical system directory requires a --no-preserve-system

Re:--no-preserve-root

[mysidia]

A shell is not built to prevent you from shooting yourself into the foot. It's built to optimize the aiming

It is a UI design defect however, that it makes it too easy to accidentally aim it in the wrong place and shoot. Even firearms generally have a trigger safety of some kind.

Re:--no-preserve-root

[allo]

you cannot escape bad wildcards by just enumberating a few. Put a space on the wrong side of a * and you're fucked.
But try "set -u" in bash and it will generate an error for uninitalized variables.
Ansible should do so as well. (and uses double braces like jinja2, not single ones)

Re:--no-preserve-root

[allo]

But you're thinking about a gun. That's what a filemanager is, if with gui or something like midnight commander. Using raw shell commands is more like working with the explosives than working with the gun. If you choose to use rather low level tools, because they are effective, you agree that you know how to use them.

And even then you can alias rm to "rm -i". Just DO NOT use "-f" ever, then.

Wasn't he trolling?

[anlag]

I saw the post on ServerFault, and while the original scenario could have happened, the OP's follow-up blunder to reverse the input and output parameters of dd when trying to preserve the disk seemed just a wee bit too unlikely. I looked at the article to see if there was any additional data to suggest this was real, but it seems entirely based on the SF thread. Until corroborated, I'm going to call bs.

Re:Wasn't he trolling?

[crunchygranola]

My operating theory is that the guy is constructing an alibi. Perhaps he has gotten wind of an investigation and wants to look like a hapless idiot and not someone engaged in destroying evidence.

Meh

[Anrego]

This is borderline bait at this point.

Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?

There are plenty of examples already and keeping a set of backups physically disconnected from running infrastructure is pretty well established practice, with random software bugs and screw ups being just one of many reasons. That said people will continue to have all their backups fully accessible (and destroyable) or just not back things up at all and things like this will continue to happen.

Guy can possibly recover the data, but the company is probably still screwed reputation wise.

Re:Meh

[thegarbz]

There are plenty of examples already and keeping a set of backups physically disconnected from running infrastructure is pretty well established practice

This seems to be more of a case for multiple backups instead of online vs offline backups. The way I read the summary it looks like the bug occurred after mounting the backup which could happen in any poorly coded scenario regardless of how secure you leave your offline backups.

Re:Meh

[Anrego]

Right, at minimum there should be two sets, and both should never be connected at the same time for exactly this kinda reason.

Re:Meh

[im_thatoneguy]

There are plenty of examples already and keeping a set of backups physically disconnected from running infrastructure is pretty well established practice

Pixar circa ToyStory 2 springs to mind.

https://www.techdirt.com/artic...

Empathy

[The-Ixian]

I have that cold feeling in my stomach just reading this summary. ick.

I did something similar (though not quite so destructive) nearly 20 years ago when I was first learning Linux.

I my case I was trying to get rid of all the hidden files in root's (/root) home dir using 'rm -rf .*'

Guess what that did?

Yeah, that wasn't a highlight of my career...

Re:Empathy

[c]

I did something similar (though not quite so destructive) nearly 20 years ago when I was first learning Linux.

Same here. Thought I was in /tmp, was actually in /, and did an "rm -rf *".

Fortunately, things were a bit slower back then and glob ordering being what it is I was able to ctrl-C it before it got further than /bin. With rcp being in /usr/bin/, I was able to (carefully) recover from another system.

Re:Empathy

[Scarred Intellect]

My turn.

I was extracted a tarball into my home directory. I was done with it and the contents in my home directory and wanted to remove them. Knowing a tarball of foobar.tar.gz typically extracts to ./foobar, I typed:

rm foo[TAB]* -rf

I expected bash to fill in to the . in foobar.tar.gz, instead, somehow I hit a space between [TAB] and *, executing the command: rm foobar * /rf on my entire home directory (meant to execute rm foobar* -rf). And this was before I knew how to do data recovery.

Similar misuse of rm -rf by a friend deleted a network drive containing my movie library, but I WAS able to recover that.

I'm much more careful of the use of tab and * in the command line now.

Re:Empathy

[cruff]

Fortunately, things were a bit slower back then and glob ordering being what it is I was able to ctrl-C it before it got further than /bin. With rcp being in /usr/bin/, I was able to (carefully) recover from another system.

I also did that years ago on a Sun 1 system, only got part way through /bin. Recovered the contents of /bin from a release tape. Learned to be a bit careful after that.

Re:Empathy

[amicusNYCL]

Well, I have good news and bad news. The good news is that I've removed all of the hidden files.

Re: Empathy

[TheReaperD]

They were available but, there was no commands or binaries to manipulate them supplied. They still don't have those commands; you have to use 3rd party tools.

Re: Empathy

[Anonymous Coward]

No, it's NOT regex.

Re: Empathy

[alexhs]

Try `ls .*`. You'll find that "." matches, but more importantly, ".." matches too. So, if you have the rights on the parent directory, `rm.rf .*` == `rm -rf ..`
`rm -f .*` works just fine for non-directories, but you need to take care of directories separately.
`rm -rf .[A-Za-z_0-9]*` will likely do the job, but if you've funny characters in second position, delete them specifically afterwards.

Re: Empathy

[The-Ixian]

Well. whatever. The result was that .* matched ..* as well as everything in the directory.

Sorry.

Re:Empathy

[The-Ixian]

A good manager will hire someone who has made a few catastrophic mistakes (and can admit to it) because a person like that, presumably, has A) learned a hard lesson and is not as likely to make other mistakes and B) will be more likely to admit to making a mistake instead of trying to cover it up.

Re:Empathy

[Darinbob]

I did the "rm dir/ *" by mistake once. Ie, a misplaced space there. I hit ctrl-C immediately but it was too late. Student account so no backups but at least it wasn't course work.

Re: Empathy

[fhage]

. is a regex term that means "any single character"

Standard Unix shells like bash use the ? character as a match for any single character.

I remove special files using 'rm -rf .??*'

For over 25 years I've always aliased rm to 'rm -i' in interactive shells. I run my rm command and verify it covers the intended targets by hitting Y a couple of times, then I cntrl C.

Up arrow, add a backslash to the front and hit return to complete the rm without prompts. When I'm drunk with confidence, I preface my 'rm' command with a backslash.

Before command line editing I used; 'yes | !!' as the safe rm override.

Re:Empathy

[squiggleslash]

My mistake was, on seeing lots of copies of /etc/passwd entitled "passwd~", "passwd.BAK", "passwd.bak", etc, thinking that "rm /etc/passwd*" would remove those files, because that means "remove anything beginning with passwd but with something on the end, right?"

I'd say rookie mistake, but I'd been using *ix for something like seven years in one form or another, and other operating systems that used * (or in AmigaOS's case #?) for even longer, so there was really no excuse.

Re:Empathy

[jordan314]

I did something similar with a symlink to an SSH mounted server. rm -rf server/ instead of rm -rf server Then I learned 'unlink' is a better option http://stackoverflow.com/quest...

Re:Empathy

[microTodd]

Hell, we've all done that. Check out this lovely gem:

#!/bin/perl
system("rm $foo/$bar");

Not my finest hour, either.

Re:Empathy

[shawn2772]

Along with all the other old stories, I'll add mine, which I think is even funnier.

I got a NeXTstation back in 1991, my first Unix box, though I'd used a few *nix variants in the lab at school -- but without root. The NeXTstation came with a 110 MB hard drive, which wasn't a lot of space even then, and what with the cool pre-installed apps, complete works of Shakespeare and some other stuff, arrived something like 80% full (in fairness to NeXT, the primary purpose of NeXTstations with such small drives was to run with NFS-mounted home directories, so it's not quite as ridiculous as it sounds). Well, my drive got full pretty quickly and I went looking for stuff to delete. I was having a hard time until I found this directory called "/usr/lib" that contained scads of big files with the extension "so".

I knew I didn't use any "so" files, whatever they were, so I deleted them.

Of course, the system kept running just fine for a while, at least as long as I didn't start any new apps. But it gradually become completely non-functional, and when I tried to reboot it failed to come up at all.

Getting it fixed required a 90-minute drive to the University of Utah, to find someone who could netboot the thing from their cube and reinstall the OS. While explaining to me -- in excruciating detail -- just what shared libraries are and why they're important.

I dropped a few hundred bucks on a whopping new 340 MB drive the next time I ran out of space.

Re:Empathy

[wbr1]

My worst gaffe was scripting a 20k home folder move to a new disk. This went fine, but I did not replace the path in the passwd file correctly. This resulted in about 5k calls to our call center as shell accounts and email broke. Fortunately I backed up the passed file first. Ahhhh 1998.

Re:Empathy

[bloodhawk]

I remember getting given my first computer from Tandy back in the 80's I was inquisitive and was testing what everything did. sadly one of the first commands I found on my DOS 3 1/4 inch floppy was format, tested and failed said it required a drive parameter so I typed Format a:

Re:Empathy

[sysrammer]

Well, I have good news and bad news. The good news is that I've removed all of the hidden files.

...and the bad news is *all* of the files are now hidden.

Fun thing about TRIM

[CajunArson]

While this guy was most likely using traditional HDDs where block level recovery is a possibility, for those of you using SSDs that have TRIM properly enabled, don't expect to be able to recover deleted files from the same drive unless you are really really fast.

TRIM automatically zeros the blocks of deleted files and they are GONE aside from vague sci-fi and probably nonexistent NSA-type forensics.

Re:Fun thing about TRIM

[sjames]

That would be a real problem. SSDs don't typically provide low level access or documentation.

Further, oddly enough, rm -rf would result in a lot of writes as metadata is updated.

Re:Fun thing about TRIM

[EmagGeek]

TRIM automatically ONES the blocks of deleted files. The erased state of flash is all ones, not zeroes.

Re:Fun thing about TRIM

[Rockoon]

When the OS sends a trim command, with it is information about what the logical sector should look like if an attempt is made to read it again. IIRC the options are zeros, ones, and random.

Without trim the ssd has to preserve the entire logical block device its emulating, ie if you have a 64GB drive then even if it only has 4KB of "files" on it, the device still has to preserve all 64GB because it doesnt even know what a file is, let alone that you deleted one.

With trim the ssd only has to preserve what the OS told it was important to preserve. So instead of preserving 64GB if data it only has to preserve your 4KB of data. Trim marks logical sectors as dont-preserve.

What the SSD will not do is overwrite trimmed physical sectors just because they were trimmed. In fact, that data could linger there for years even with a high amount of read/write activity because SSD's only erases entire physical blocks, not just the subsectors within blocks that were trimmed.

So recovering is not sci-fi. Recovery is a fact. What can't be done is recovering the data via commands that target the logical rather than physical device.

Re:Fun thing about TRIM

[ChumpusRex2003]

The standard is that a TRIMmed LBA should read as all zeros. This is so as to permit the use of TRIM in RAID arrays while preserving parity consistency.

If the OS needs to TRIM an entire RAID stripe, it fires off a TRIM command to the data drives, and calculates parity for null data, and writes that to the parity drive.

Re:Fun thing about TRIM

[DutchUncle]

That would be a real problem. SSDs don't typically provide low level access or documentation

Maybe not to you, but to the NSA or people with a court order - and the SSD removed from the system and cracked open - many things are possible.

Re:Fun thing about TRIM

[sjames]

I'm pretty sure that wouldn't be relevant to some dude that accidentally rm -rf /.

Re:Fun thing about TRIM

[allo]

nope.
That's what the controller tells you.

TRIM says the ssd "you do not need to delete anything, but zero out the block logically". Then the SSD may or may not reuse the block when it needs a new block to write to. Then it will be overwritten. When you try to read it, the ssd will tell you, it's empty, as it has the entry "block x is empty" in its metadata.

An expert may even restore more from a SSD, because the next block written at this location will be some of the (currently) spare blocks TRIMed some time before, because of wear-leveling. But you need to unplug the SSD and read it without the controller part.

1500+ customers and he can't afford

[future assassin]

a couple cheap Kimsufi servers from OVH for remote backup in EU and In Canada?

Re:1500+ customers and he can't afford

[SumDog]

Read the article. He claimed to have off-site backups in other countries, but they were mounted.

But also read the note under the summary. This whole story is probably bullshit.

Re:1500+ customers and he can't afford

[tnk1]

It probably is bullshit. Who fucking mounts servers in another country to do the backups to directly?

You archive and compress that shit locally and then move it to the remote server. That prevents your daily backup from taking 48 hours to complete and helps considerably on those data transfer charges.

Having a simple archive and transfer via FTP or something alone could have prevented him from deleting the remotes with one command. I have trouble believing he set up a NFS or other remote volume to another country. Holy iowait, Batman.

insurance fraud at best

[Anonymous Coward]

This has such a smell of BS around it. given the fact that backups are indeed offsite and that a company has more the 1 server etc.etc. Even my own simple setup consisting of a pc, laptop, tablet, qnap and some external HDD and sticks is impossible to delete with 1 script. total bollocks.

Wonder if he found incriminating material or has gambling debts, far more plausible

Re:insurance fraud at best

[michelcolman]

That actually does sound plausible. Got wind of an investigation, deleted everything, made it look like an accident. "Oh, I'm such a dufus, ran rm -rf * on all my servers and backups, then used dd the wrong way around, and then the cat jumped onto the keyboard and typed cat /dev/urandom > *, and then I pulled all the drives out to try to save them but accidentally dropped them from the tenth story, right into a garbage compactor truck, and you won't believe what happened next..."

Re:insurance fraud at best

[trogdor8667]

Where are my mod points when I need them? +1 hilarious

Re:Which command?

[danbob999]

He can boot on a CD or USB and recover most files.

guards?

[wyHunter]

Guards around the command asserting the variables?

manishs

[Verdatum]

Manishs, you seem to actually critically read articles before posting them, and you actually provide insight after the summary. What is up with that?

Re:manishs

[msmash]

I hope you're not being sarcastic.

Re:manishs

[Nidi62]

Hey manishs, this is off topic, but I've noticed issues on 2 different computers and 2 different browsers (Chrome and Firefox) that the disable advertising box does not stay checked when moving between pages (or even refreshing the current page). If they aren't already aware, could you pass that on to whoever could use that info? Thanks.

Re:manishs

[Verdatum]

I mean that I really do appreciate it. Keep up the good work!

Re:manishs

[msmash]

Thanks :)

Re:manishs

[msmash]

Thanks for pointing it out.

Re:manishs

[Nidi62]

I appreciate the response. I mostly only block ads when I'm browsing at work. Giant banner ads on the top and side of the page makes discrete slacking rather difficult :)

Re:manishs

[Anonymous Coward]

I love that commentary on the article seems to be making a comeback.

More so when it is descriptive, even at a basic level, and especially more so when it discusses the validity of the story.

Keep it up guys.

Re:manishs

[Cut]

"Rm -rf" would mark the block as empty, and unless the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.

As a heads up, "unless" should be "if" or "hasn't written anything new" should be "has written something new."

Re:manishs

[Szeraax]

I agree. Some people ONLY want the data points of the news. If that is the case, I would read just ArsTechnica or one of the host of other tech news sites. I /. because I want to see what other tech people think about the news. Putting your opinion on a submission where it is CLEARLY not part of the article is something that I like and hope you will continue to do.

2 Big issues

[silas_moeckel]

Why in hell is is running scripts out of ansible? Why are those scripts not running on a QA system thats a block for block clone of production? Finally what idiot thinks that some mounted drives he copies stuff for is a backup system?

Tape disk I do not care just treat disk as tape, plenty of backup system are more than happy to do just that. Rsync is not nor will it ever be a backup, snapshots are not a backup, some script some guy wrote that works ok is not a backup. Now they can all help to meet your RTO but at the end of the day it's not backed up till the device is offline sitting offsite.

Re:2 Big issues

[jcdr]

man sync
[...]
-b, --backup
                            With this option, preexisting destination files are renamed as each file is transferred or deleted. You can control where the backup file goes and what (if any) suffix gets appended using the
                            --backup-dir and --suffix options.

Re:2 Big issues

[jcdr]

Err, please read 'man rsync' of course :-)

I also used the rsync batch mode to keep the last 6 months daily backup.

Re:2 Big issues

[silas_moeckel]

That's still not a real backup strategy. Look all my backups are on mounted disks/arrays. It's great for RTO and can be part of one but at the end of the day you still need to get that data offline and offsite. So it might be more correct that rysnc is not a complete backup system.

I've been down that road to many times it's far far too prevalent in the hosting and small business segments that think a single copy will be fine. Idiocy like a backup drive in a local system, because that shares no failure domains na. I sent it to this cloud provider what could go wrong.

Re:2 Big issues

[jcdr]

Then use rsync batch mode to maintain multiple offline and offsite copy. That's still simple.

Re:2 Big issues

[silas_moeckel]

It's still a piss poor backup. You have no central thing to orchestrate/monitor the backups. At the end of the day there are plenty of OSS backup tools that do the job right. Hell most of them use rsync or similar to do file level diffs. But that does not make rsync by itself a good or particularly viable backup strategy, when to get to even reasonable you have a pile of custom scripts.

Re:2 Big issues

[jcdr]

Well, mostly agree, this depend of the situation.
For some very simple situation a periodic rsync could be just enough (I agree this is a very light definition of 'backup').
For more demanding situations there could be exiting backup management software using rsync that work very well.
For a few others very unusual situations, not handled by existing backup software, a set of scripts around rsync could be a good solution.

What happened to NEWS for Nerds?

[Jack9]

This was a blatant troll on a forum and now because some idiot millennial wrote an op-ed piece, some idiot (manishs) put it on the /. frontpage?
Are the admins now supporting the things the moderation system fights on their own site?

This story is more of an embarrassment than the political vomit I've had to endure because _this_ story doesn't even qualify as news. e.g. What Company did he destroy exactly? You would think the incredibly obvious lack of facts would be a tipoff to someone.

Re:What happened to NEWS for Nerds?

[Anonymous Coward]

Nevertheless, it's a welcome respite from AGW and women in STEM.

Re:What happened to NEWS for Nerds?

[jschultz410]

This was a blatant troll on a forum ...

^^^^^^ THIS!!!

Re: What happened to NEWS for Nerds?

[Darinbob]

I make it a point to lump people into the category of "everyone". Then I can despise them all equally without picking and choosing favorites.

Re:What happened to NEWS for Nerds?

[Darinbob]

I noticed IBM was offline. Maybe it was them.

Re:What happened to NEWS for Nerds?

[HiThere]

If you had done that, would you have wanted to let people know who you were, or who you worked for?

I'm not saying you should believe the story, but pick legitimate points. There seems to be an improbable level of stupidity. (OTOH, I'm known some managers...let's just say most of the one's I'm thinking of wouldn't have thought to ask for help.)

Re: What happened to NEWS for Nerds?

[david_thornley]

It's not millennials who have excessive smugness, outrageous senses of entitlement, unjustifiable arrogance, and penchants for causing lots of problems for others. It's young people. Since millennials are currently young, they get all the blame. In twenty or thirty years, the millennials are going to be saying this about the currently young generation.

Re:What happened to NEWS for Nerds?

[twistedcubic]

What the fuck is up with everyone using "millennial" as a catch-all insult?

We apologize for this. Now get your millennial ass up off my lawn!

Re: What happened to NEWS for Nerds?

[Ginger Unicorn]

i remember getting the old fogey hate treatment in the early 90's when it was all "generation x'ers are zombified by their tv and their video games". as it is every generation..these delinquent kids with their Elvis and their Hula hoops...etc etc

Re:What happened to NEWS for Nerds?

[Jack9]

Obviously...

http://www.pcworld.com/article...

Hobbling the default rm command, etc.

[John Allsup]

Hobbling the default rm command slightly would make a sense, possibly having a second command (oblit or something) for the really nasty stuff, would make sense. Many commands can be unnecessarily destructive, and those destructive commands are too easy to invoke by accident. Possibly requiring a --really and a --reallyreally switch on rm to enable things like rm -rf crossing filesystems, would make sense. I did once make a quick hack so that rm -rf would require an environment variable to be set in order to work, but then various scripts broke.

Re:Hobbling the default rm command, etc.

[alvarogmj]

He says it was from Ansible, and having worked with it, I fail to see why in hell would you run 'rm -rf' from it when there are modules for that.

Re:Hobbling the default rm command, etc.

[Dahamma]

It already is hobbled. You can't delete / unless you add "--no-preserve-root".

If you read more about this post, it was an obvious troll. The problem is it got picked up by a "tech" journalist with just enough UNIX knowledge to understand what "rm -rf" does, but not enough to notice the obvious flaws in the story...

Not everything

[Melkman]

When an friend an I got started with Linux he wanted to remove his Slackware install from a dual boot PC. For fun he ran rm -rf / on that install. We had a good laugh when the message scrolled by of the OS trying and failing to remove files from the CDROM. That was until he realized that he had mounted his Windows partition too. It didn't fail to remove files there :-)
 

Bad Coding

[hattig]

This guy has learned the hard way about losing data because he wrote bad code.

He had a backup, but his code bypassed common sense and mounted the backup server's remote drives in the filesystem.
Which he then wrote a script that ran as root that in this case deleted everything because it didn't sanitise inputs before running the 'rm' command.
Ideally he would have built the path to be deleted, so he could check that it was sane, i.e., starts with "/users/" or similar, before passing it to 'rm'.
Worse, is if he was using an off-the-shelf-hosting-package that did this. But when you do, assume it is broken, keep a wall between the systems it can touch, and your last hope (your backups).

Backup systems should ideally be push only, with any delete action only occurring after validating the replacement. And let's be honest, in this day and age, you can keep multiple generations of backups, push them into Amazon Glacier to keep things cheap.

I truly hope he was a low end hoster that gave ftp access for website uploads only, so the customers can re-upload their sites, once he reprovisions his servers. Although that's the best option, any databases (for example) would have been wholly reliant on his backup strategy.

Sheesh, who makes their backups part of the system that is being backed up! It's meant to be isolated because of, frankly, the situation that occurred.

Re:beed there, done that

[zarr]

Oh, the old not-so-side-effect-free GET request story. Happens a lot, and makes me chuckle every time :-)

Re:Why is everything "trolling" to people like you

[s.petry]

You missed one.

-- Some anonymous coward complains about people calling out trolling, trolling.

The funny part is the person you responded to did not claim "trolling", they expressed a healthy skepticism. That last part is something more people should have. There are plenty of liars out there. Quite often they work for main stream media outlets and hold public offices.

set -eu

[bolt_the_dhampir]

So, he writes a bash script for an important backup job and doesn't "set -u" at the beginning to throw an error if you attempt to use an uninitialized variable? Really?

Re:set -eu

[allo]

{foo} is no bash variable, but some other template language.

Something similar

[CanadianMacFan]

I was working at a small development shop about 15 years ago and I came in one morning to find the main development server not working. Turned out that the previous night a developer on the other project ran "rm -rf" from the root directory on the Sun box and then tried to fix things before giving up and going home. No note, no call to the boss, nothing to indicate what had happened so I had to figure that out when I arrived around 8 AM. Oh, and no backups of their project. I at least had the latest version of the source code on mine on my laptop the but the history and development environments would have been toast.

Since he toasted the box and it was the only Sun box around (small company) I had to install the OS again. But the only thing that saved us was the fact that rm deletes files and folders alphabetically (or at least at that time on the Sun boxes it did). So as soon as it started removing everything in /dev the machine was toast and it couldn't delete anything else. I first found this out by booting off the CD so I was able to mount the drives. We had our development environments and source code on separate drives (thankfully) so I copied anything that was useful off of the drive used for / onto one of them, installed the OS onto that drive while leaving the other drives alone, and then manually putting things back to where they were. Took less than a day and the owner was very thankful. Never heard a word from the guy that caused the mess though.

Ever since then whenever I've created mount points for new drives on Linux/UNIX servers I've always made sure that they come after dev alphabetically.

Re:Something similar

[sabt-pestnu]

> rm -rf doesn't actually delete the files.
> you had it within your ability to recover the data.

For those of us who do not eat, drink, and sleep Linux (or UNIX...), would you please describe the built-in commands that you would use to recover the data?

Or the forensic tools you would need to use (name some examples)?

And does it change the recovery method/possibilities if the entire server is virtual, and that the blocks covering the deleted files may have been recovered by the host (dynamic VM storage allocation)?

Not trolling you. Want to know.

Deleted a critical file once. /vmunix. Most embarassing. 25 years ago. "Had to" restore from tape, not knowing an alternative.

Re:Something similar

[LQ]

rm -rf doesn't actually delete the files. It's the linux counterpart to windows "quick format" lets wipe out the table and index, but not the actual data

My understanding is that rm will unpick the daisy chain of linked blocks allocated to the files and move them to the free list. I don't think there is any easy way to reconstitute removed files. Do any of the fancy new *fs do anything different? Mind you, I once used unlink as root unwisely and was able to recover an entire directory tree from lost+found. But that's a whole different mechanism.

He just did a "rm -rf My_IT_Career"

[JoeyRox]

Joke or not he's voluntarily entered himself into the timeless database known as Google, viewable with the not-so-secret incantation "google Marco Marsala"

Re:Repeat after me...

[Hylandr]

Adding to AC's astute observation: ..."In the cloud" does not mean "Is backed up." ...'Deleted' from the cloud does not always mean 'deleted'. ;)

Re:Which command?

[Anonymous Coward]

How does one boot the cloud from a CD?

Still value as a troll

[Minupla]

I collect these stories for people who I mentor. Even if they're trolls, they work as cautionary tales, because lots of people have had similar smaller scale disasters (as evidenced by posts in this thread) and it's healthy for mentees to get a taste of what can happen when you (for example) forget to error check your script parameters.

In a big way it doesn't matter if it's true or not, it could be true which makes it a teachable moment. I'm sure everyone who reads the story will run a mental checklist to see if they have a script somewhere that could EVER do it. Do they have their backups mounted when they should be rsyncing, etc.

Min

Backup with rsync instead of a mount

[jcdr]

That's so simple and effective, work on local network as well as on remote networks thank's to ssh.

Re:Backup with rsync instead of a mount

[allo]

rsync --delete ......

Corrections

[ledow]

Man ALLOWS his entire company to be wiped out in one command.

Man DESIGNS his entire company to be wiped out in one command.

Man SETS UP his entire company to be wiped out in one command.

Hint: I work in schools. Once I had a teacher delete their entire planning folder. Then (and DO NOT ask me why, because I don't understand it either), they emptied that folder from Recycle Bin. They rang up in the more embarrassed panic.

And then it was explained that we still had copies of that folder in:

a) Shadow Copies of the profile on the client.
b) Network Copies of the profile that they were logged in as (and which fortunately hadn't logged off once they realised what they did).
c) Shadow Copies of the profile folder on the server.
d) Copies of the profile folder on all the other servers.
e) Copies of all the servers on replica servers.
f) Copies of the server VM's and storage in a primary backup location.
g) Copies of the server VM's and storage in a secondary backup location.
h) Copies of the server VM's and storage in a tertiary backup location.
i) Several off-line and off-site copies of the server VM's and storage .
k) Random, casual backups all over the place.

And that's just for the crap that teachers think is important (i.e. a lesson plan they have to write every two weeks and which they can't re-use anyway).

Fuck knows what this guy was thinking, but there's no one one command ANYWHERE should be able to do that many actions, let alone dangerous actions that you haven't evaluated properly. Honestly, some of those machines don't even TURN ON until the backup window, and even the backup devices have rollback and shadow-copy-like functionality on top of whatever the backup software gives (incrementals, etc.). And several are DELIBERATELY offline for almost their entire lives and have entirely disparate credentials so no one command could ever affect them.

Not being funny, but we're talking a small school of 400 5-14 year olds here. He actually has more customers than I have users. And you just can't fuck about like that, so if he thinks he can, I honestly have zero sympathy and can only laugh.

Re:Corrections

[Anonymous Coward]

but there's no one one command ANYWHERE should be able to do that many actions, let alone dangerous actions that you haven't evaluated properly.

This is really a problem with the Enter key, not with rm. If you randomly press Enter, you never know what will happen. I think Enter should be removed from all keyboards. It's far too dangerous to keep around.

Re:Corrections

[ledow]

You might want to go ask some IT guy somewhere.

In my country, you are legally obliged to provide attendance and visa and financial data for between four and ten years. No "Oh, but we lost the backup". You're legally obliged. Can't do it? No school.

You're legally obliged to hold enough work to prove these kids can pass the exam coursework the teachers say they can. No ifs, buts, maybes, you lose it, they lose their qualifications forever.

Add on testing, pupil tracking, MIS databases (including medical records, records of dealing with child abuse, etc. staff CRB checks, parental complaints, legal issues, etc.), staff salaries, pensions, decades of teacher planning, always-on web resources, VLE's, staff VPN's, mail, etc. and it quickly escalates.

I've worked in London primary schools for the last 20 years. Backups like this (shadow copies aren't really backups, but still, it takes two seconds to turn the option on) are standard. There was a time when you physically separated admin and curriculum networks to prevent data transfer - twice the servers and cabling and switching (but now you VLAN and/or just permission properly).

Secondary servers are the norm. Grandfather-father-son backups are the norm. Off-site backups are the norm. Every school I've ever worked in has them, with proper tape rotation already in place before I even walk in the door. And we're talking state schools with a couple of hundred kids below age 11.

Some small schools ask the Boroughs to help them out and they have somewhat centralised services. Which all do this too. But most primary handle their own IT. And probably have at least one IT suite, several class sets of tablet computers, and quite possibly interactive whiteboards or touchscreens in every classroom. It's the norm for teacher interviews to expect to be able to turn up with Smartboard files and get started immediately.

Now consider what that means in terms of daily expectations, and actually the stuff I've listed just about keeps you running.

The school I *actually* work for now, a large private boarding school, has IBM BladeCenters, iSCSI storage, off-site VM replication, MDM, Cisco Meraki networking and wireless throughout and double-redundant fibre backbones on leased lines. I don't expect everything school to have those.

But I assure you that every school I've ever worked in cycles tapes religiously and keeps several offsite, and has kit supplied which backs up and emails backup logs which school policy states must be checked and that the governors verify is taking place. And some of those literally didn't have enough money to buy exercise books.

The cost of data loss like that far outweighs anything to do with the education of your child, even if it doesn't actually make it into the public domain.

Re:Corrections

[phorm]

Where I worked previously, we would generally disclaim having backups, until an important situation arrives (we did have them, but if people don't think we don't they're a bit less likely to do something stupid with their data).

Extremely timely article!

[ErichTheRed]

I just got put on a project at work as "the systems guy" for a project being built in Azure. This is in support of a reasonably critical system, and the development staff are salivating over the chance to self-deploy code and infrastructure. It sounds like this problem was caused by the first thing I noticed as a risk -- if you don't limit what Azure users can do, it's just like giving them the keys to the data center. And this isn't in an "evil BOFH control freak" sense, this is just the fact that everything in Azure is virtual and easily changed either manually or through automation. So, someone who's having a bad day could easily make a mistake and get rid of things they have permissions on -- it's possible in AWS too.

It's a really different mindset than even a hosted IaaS service. There, if you do something stupid, at least the physical infrastructure doesn't get rolled up and carried off. Now hopefully you have backups if that happens and can just restore the VMs and storage as needed, but if developers are running the show I would highly doubt it. (In Marco's case, I would imagine this was caused by the classic "run as root, because I'm the boss" issue.

So, in summary, all the (good) sysadmins worrying about the cloud taking their jobs need not worry. The rules of designing a safe computing environment have changed, but they haven't gone away entirely! I'd be a little worried if I were a savant-level EMC or Cisco guru right about now, but generalists with good heads on their shoulders are still in demand.

Three steps

[frovingslosh]

Put backup copies in truck.

Drive them to the backup site.

Repeat regularly.

Re:Three steps

[Waffle Iron]

... and don't underestimate the bandwidth.

Re:Three steps

[fustakrakich]

You're just asking to get hit by a meteorite out there in the desert. Wouldn't it be safer to send two trucks out, in opposite directions for eight hours, turn around and come back for the next day's backups, and repeat?

Re:Three steps

[Triklyn]

no, asteroid or nuclear blast. safest would be dig a deep dark hole, and send it down with a guys and pack mules... as well as the 2 trucks going opposite directions.

duh.

Re:Three steps

[fyngyrz]

The zombies will get you either way. It's hopeless, really.

Re:Three steps

[Triklyn]

come on, did i ever say you didn't need to kit out your trucks road warrior style? and every dungeon needs some traps.

Re:Three steps

[david_thornley]

Remember the article a day or so ago about accelerating a small space probe to 0.2c? That would take out your hole, your mules, your trucks, and your truck destinations. You need offsite backup on another planet.

Re:Three steps

[Locke2005]

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." Andrew S. Tanenbaum

Re: Three steps

[Frankzy]

Or as an absolute minimum on the opposite side of the planet

Not buying this

[phishybongwaters]

Nope, not buying any part of this story, nope. No one is dumb enough to run that without a test. And how were the offsite backups even accessible? doesn't matter, because everything would be recoverable from the systems he "wiped". No, this is another bullshit story spread around by IT departments. this did not happen.

Re:Not buying this

[YukariHirai]

The story probably is a fabrication, but it's not implausible. If there's one thing I've learned about humanity in all my years, it's that there is no limit to the stupidity it's capable of.

No line of bad code

[aglider]

Just a bad brain!
Can you spell "test"?

Re:No line of bad code

[jittles]

Can you spell "test"?

Can you use it in a sentence please? Oh wait no never mind. I think I've got it! D-O-I-T-L-I-V-E???

Re:No line of bad code

[aglider]

No, I cannot.

Re:Which command?

[gmack]

That will work unless the drive was on an SSD and the OS told the drive to TRIM the deleted blocks.

Re:Why is everything "trolling" to people like you

[X-Ray Artist]

...Somebody accidentally transposes two letters when using the dd command...

Did you intentionally transpose your "dd"?

Re: Why is everything "trolling" to people like yo

[Megane]

If they weren't on alt.religion.kibology back around '92 or so, they don't know shit about trolling.

"trolling" word use

[fbobraga]

"trolling" are a very used adjective to claim someone is inferior, today. It's an Ad hominem heavy used tool, nowadays! The hell is, really, other people! :P

...and...??

[dentar]

He admitted it publicly?

Re:...and...??

[GlenRaphael]

> He admitted it publicly?

Not really, no.

The question was posed by username "bleemboy", whose profile has been made private. The allegedly associated truename "Marco Marsala" is kind of generic. (there are three on LinkedIn, none claiming to be in the hosting business. Not to mention, if there WERE a "Marco Marsala" in the hosting business wouldn't you expect him to have registered his own name?)

I find it deeply suspicious that none of the alleged "more or less 1535 customers" nor anybody USING any of the sites run by those customers has piped up in the comments at reddit or here or anywhere else to say "So THAT is what happened to my favorite site and the company hosting it!"

Re:Why is everything "trolling" to people like you

[s.petry]

Thanks! I don't always read the subject and missed it. Can I call it an out of "body" experience? :^X The skepticism part is still correct.

rsnap is popular. Should pull from read-only accou

[raymorris]

Rsnap is a very popular backup system which uses network mounted drive as it's default/most common configuration. I constantly remind people on the rsnap mailing list about the existence of cryptolocker type malware.

A much safer way to do it is to have the backup system PULL backups using a read-only account. That way no command on the live system can touch the backups, and the backup system can't change anything on the live system - either accidentally or maliciously.

One solid backup / hot spare system that does it safely by default is Clonebox.

Excellent troll ...

[jschultz410]

I mean, he made /.'s front page. That's pretty darn good!

Re:Excellent troll ...

[__aaclcg7560]

Front page exposure always results in a Slashdotting. Oh, wait. Never mind.

Re:Repeat after me...

[NatasRevol]

For most users the cloud is more reliable and more accessible than anything they'll ever be able to do in a SO/HO environment.

The problem is that medium sized (1000+ employees) seem to think this too.

And then have no backups, version control or anything else because some PHB said 'put it in the cloud, and stop arguing with me.' to the IT engineers who wanted local & remote backups, version controls, redundancy of hardware, network and power, etc, etc, etc.

Re:Why is everything "trolling" to people like you

[NatasRevol]

I don't like windows, but LOVE PowerCLI for managing VMware environments. And HATE using pyvmoni.

I get to troll everyone.

Perhaps time for a change to rm

[WindBourne]

The likely hood of this happening is slim, but I sometimes wonder if a minor change is really not that bad.
In this case, change rm to NOT allow /, until -t/--top is added. Then it is allowed.
With this minor change, it could save noobs from themselves and would like not be used that often in the first place.

Re:Perhaps time for a change to rm

[jittles]

The likely hood of this happening is slim, but I sometimes wonder if a minor change is really not that bad. In this case, change rm to NOT allow /, until -t/--top is added. Then it is allowed. With this minor change, it could save noobs from themselves and would like not be used that often in the first place.

They've got an even better one, and it's been in rm for a while. --no-preserve-root

Re:Perhaps time for a change to rm

[WindBourne]

just looked that up. I have been Linux since 1 floppy days and did not know about that. Thank you.

Re:Perhaps time for a change to rm

[Dahamma]

Yep, and apparently the troll who posted the original question to severfault didn't know about it, since when questioned how he managed to delete / without it... crickets.

(the whole thing was pretty much proven to be a troll...)

Re:Perhaps time for a change to rm

[jittles]

just looked that up. I have been Linux since 1 floppy days and did not know about that. Thank you.

Just note that if you do /* it will not require the flag. It'll still nuke your entire disk.

Chain of Mistakes

[Greyfox]

Recently the USPA was talking about stuff that kills skydivers. It's almost never just one mistake. It's a chain of mistakes where one single good decision anywhere in that chain would break the chain and prevent entirely preventable deaths. In the case of this story, if it had actually happened, which it didn't, the decisions made to violate best practices all along the chain (IE, running your bash scripts as root or as any user ID that has authority to delete anything on the file system, not pushing just pushing your backup data to isolated storage, not having numbered sequential backups, etc) would be so egregious that the story would simply be an example of Darwin at work. The conversation would go "Oh hey, did you hear about that guy who designed his system so badly that he was able to delete the whole fucking thing with one mistyped command? Yeah, the council of sysadmins voted to kill him. Said it was for the good of the species."

I feel for the guy

[jomcty]

Talk about a bad day at the office. As an IT professional, I feel for the guy.

Theology Lesson

[Miamicanes]

On a Linux system, root is God(*)

God is omniscient, omnipresent, and infallible.

Therefore, when root deletes files, it's never a mistake, and the files should be immediately destroyed forever without question.

(*) Unlike those heathen Windows systems, where there can be multiple gods, some of whom are more equal than others... and not necessarily in ways that are obvious to casual observers... ;-)

Old Saying

[Tablizer]

"To err is human. To really fuck things up, you need a computer."

I prefer that any bulk or query-based "delete" command ask for confirmation along with basic feedback. Example pseudo-code:

> delete *:*.*

You are about to delete 832 folders and 28,435 files.
Your choices are:
      1 - Proceed with deletion
      2 - List path details about the above folders and files
      3 - Cancel deletion
Your Choice: __

(end of example)

It may be slower and/or more resource intensive, but that's better than mass boo-boo's.

An optional command parameter could switch off verification, but verification should be the default. This is something Unix/Linux gets backward in my opinion: the default should be confirmation mode, not the other way around. In other words, a command switch should be required to switch off confirmation rather than requiring a command switch to turn confirmation on.

Typical SQL doesn't have a confirmation mode, so I usually do a verification query on the WHERE clause before running the actual:

-- check
SELECT count(*) FROM myTable
WHERE x > 7 AND foo='BAR'

-- actual, keeping same where-clause
DELETE FROM myTable
WHERE x > 7 AND foo='BAR'

I also often inspect at least some of the actual rows, not just the count. Thus, as a rule of thumb, do random spot-checks of actual data, and a total count before final command execution.

Re:Old Saying

[hankwang]

"This is something Unix/Linux gets backward in my opinion: the default should be confirmation mode, not the other way around."

1. All Ubuntu versions and derivatives (and I think Centos/RHEL as well) alias rm to "rm -i" out of the box. Drives me crazy; with every install I have to hunt down whether those aliases were defined in .profile, .bash_profile, .bashrc, /etc/profile, /etc/bashrc, or somewhere in /etc/bash/*.

2. Command-line tools that ask for confirmation suck for scripting. Especially if those prompts only occur under specific conditions (such as confirm overwrite).

Re:Old Saying

[iusty]

With SQL is even easier. BEGIN TRANSACTION, run the command, and check how many rows it deleted/updated/inserted. If not the number expected, ROLLBACK.

Re:Old Saying

[Tablizer]

That's a good suggestion, but if the actual target quantity is far larger than expected, then buffering the (pending) changes can bring the system to a crawl. Not as bad as an erroneous delete, but still unpleasant.

I might run a count first, then run it again using TOP N, LIMIT, or rownum n to sample some records before I do the final query. Steps vary per RDBMS vendor.

Re:Old Saying

[Anonymous Coward]

Unix:

You do know what you are doing

and

You meant what you said

Re:Old Saying

[Tablizer]

By default, rm will ask you to confirm you want to delete every single file, one at a time, before it's removed.

That's not practical when working with lots of files. The default should probably be summary statistics and/or review options, and then a confirmation. If the default is rarely useful, then people get into the habit of not using the default, which defeats the very purpose of a default. My description of the complaint could use better wording, I do admit.

Putting some kind of "sanity limit" on a command is also a useful idea. For example, the hypothetical "rm -s5000" could mean stop if the command affects more than 5000 files and/or folders. This is not the same as stopping after 5000, but cancelling the entire command if it will impact more than 5000 if continued.

A variation would be to prompt if more than the stated limit. If you run it in batch mode, then a prompt request would typically stop the batch process and/or wait for user confirmation. This is probably closer to what we really want.

Re:Old Saying

[La Gris]

One solution would be having prepared statements for shell commands instead of doing those risky string concatenations.

And I have stopped using rm to delete files because this command is way too stupid and can take options from file-names.

I use find command,
run it dry once to show what is found
then run find stuffs-to-trash -delete

Re:Old Saying

[AbRASiON]

I've read about rm for years and never seen this suggestion before.
I don't understand why it's not put in place, that's a great idea to be honest. A simple solution and easily bypassed by impatient admins with a -now or -y in the command line for auto approval.

Re:Old Saying

[lastman71]

Yes confirmation before doing something is dangerous, and often rm is alias to 'rm -i', which let it prompt to you if you want do proceed.

The point is: if you need to run it over 1500 server, you can't confirm one by one. And how many rm were in the script? So you'll be asked for 1500 x (number of rm comand in the script). No way.

The problem is confirmation help a bit, but not very much. When I was young, I worked on VMS, where del command ask for confirmation. And I have spent the day doing test, where I had to delete some *.dat files in a directory, to run a program, to see if the files are produced correctly, repeat till the file are ok.

What could possibly go wrong? Just change directory, where a lot of configuration file are (and you can immagine, for some reasons, many have a generic extension like .dat), and just restart to do test without thinking, and say yes out of habit.

If you think that confirmation is a good tool to save data, you're dangerously wrong. As many here have already said:

1) copy data somewhere else is not a backup, and not help you from corruption/deletion of data. As soon you syncronize the data on the copy, your data is lost.

2) real backup save the history of your data.

3) your backup data must not be accesible from the server, you should not be able to delete backup data from the same place where you can delete the same data.

So, I agree, that you should make difficult to make mistake. And it is already the default in unix (or at least in recent modern linux distro). But is not enough. You should also able to undo eventual mistake. As a sysadmin, I have learn, that every time you do something, you should thing about how you will undo if something go wrong. Backup is the most important tool.

Re:Old Saying

[LordWabbit2]

Any and all updates or deletes I ALWAYS do a BEGIN TRANSACTION and ROLLBACK, even when on the dev box. I have had one or two "oh fuck" moments before and this helps. If I am really paranoid or the data is difficult to restore / recover I do a full backup first.

Re:Old Saying

[Wolfrider]

--You do know you can override the alias by calling the path+exec?

$ alias hostname
alias hostname='hostname | cut -d '\''.'\'' -f 1'

$ hostname # calls the alias
thisbox

$ /bin/hostname # calls the exec direct
thisbox.fully.qualified.domain.name

--Plus, bonus:

$ for i in .profile .bash_profile .bashrc /etc/profile /etc/bashrc; do echo $i; grep 'rm -i' $i; done
$ grep -R 'rm -i' /etc/bash/*

> Command-line tools that ask for confirmation suck for scripting. Especially if those prompts only occur under specific conditions (such as confirm overwrite).

--Agreed, but you can usually get around those by ' echo y| annoyingscript ', unless you have sudo in the mix... Then you might have to (shudder) use parenthesis or something ;-)

Re: Why is everything "trolling" to people like yo

[VAXcat]

Or alt.folklore.urban. Some world class trolling occurred there, back in the day.

Re:manishs clearly has no idea...

[DutchUncle]

Disagree. At some level, a real command for a real delete has to happen. This should be like RDP being built on top of UDP. The script could just as easily be deleting prev.active.stuff, then renaming active.stuff to prev.active.stuff, rather than directly deleting active.stuff. Oh, wait, that would imply building a system with enough extra capacity to maintain a first generation hot backup - which is what the "undo" does anyway.

Same process had access to offsite and local data

[QuietLagoon]

Poor design of a backup system.

.
Offsite data should be offsite. Period.

It should never be accessible by the day-to-day operational processes, and that includes the routine backups.

He needs another backup step between the local and the offsite backups.

I back up local data to a onsite staging area. Then, in a totally separate process, I back up the staging area to offsite facilities.

I empathize with the guy....

Condolance cards

[Danathar]

Is there any address to send flowers?

When will the funeral be?

Been there, done that

[Mondorescue]

I did something similar to my server in 2005. It wouldn't have been so bad, except that (!) I hadn't made a backup in months and (2) the server hosted a disaster recovery project whose purpose was to guard against catastrophic data loss. A cobbler's bairns are aye the worst shod...

Re:rsnap is popular. Should pull from read-only ac

[mlts]

The best of all worlds is pull based backup software. However, the enterprise based programs are extremely pricy, well out of the range for a home user. The cheapest around would probably be Windows Fundamentals which is a descendant of Windows Home Server.

What I've wound up doing on a small scale (this won't scale up past a few machines) is having a hardware NAS appliance. It had a samba share and account for every machine. The Windows boxes use Veeam to dump their data onto the individual shares. Every 15 minutes, the NAS pops a snapshot of each share, where several are kept for each hour/day/week/month/year, and the rest get tossed after a while. Every eight hours, the NAS backs itself up to an external HDD. This protects against ransomware in several ways. If ransomware just zaps the share, restoring the snapshot and bare-metal loading the machine isn't too bad. If ransomware takes its time and zeros files over an interval, because I have weekly, monthly, and backups over a duration, there is a good chance that I will still have the file around, either in a snapshot, or on the backup drive. Because each machine dumps to a separate share via a separate account, ransomware on one box can't destroy or access another machine's data.

The ideal would be having the NAS maker writing an agent that sits on Windows and uses SSH or another time-tested protocol to pull backups. This would not just guarantee that backups are done, but are protected against ransomware.

rsync and LVM

[Mybrid]

LVM and rsync are your friend.

The easiest way to backup physical is with rsync, LVM and snapshots. No rm required. None of my backup scripts have /bin/rm in them?

If it is a VM just take a snapshot of the VM?

Why would any backup script use "/bin/rm"?

This sounds really fishy.

Or really incompetent.

Or both.

Use Shell File-name expansion

[iam_TJ]

The usual way to do it using the shell file-name expansion is:

bash/dash/sh: ls .[!.]*
bash/csh/zsh: ls .[^.]*

which will pick all the dot-files but not double-dot (link to parent directory). Only time this will be problematic is if there are files/directories named with the style "...three-dot-file".

Sodium per diem

[fyngyrz]

You must be on a salt restricted diet. Nobody likes unsalted earth, or earth that only has the salt you added while cooking.

Myself, I like to scorch it, salt it, scrape it into a squarish pile with four corners, dye it red, vibrate it until I'm sure the earth moved, and then chant "Tara! Tara! Tara!" I do all this wearing Birkenstocks. It's a thing.

Tastes much better that way. Really.

Re:Why is everything "trolling" to people like you

[HiThere]

You had made a block copy backup via direct connection first, right? (I'm just assuming you can still do that, since it's been several decades since I had someone do it for me.)

Backups are the best answer, but if you don't have backups, the first step should be to make a sector-by-sector backup. And I, personally, have had more success having a hardware guy do it for me in hardware. OTOH, I'm not sure I could even edit a disk anymore. The last time I needed to to that I was using MSWind95.

Re:rsnap is popular. Should pull from read-only ac

[fisted]

Then again, rsnap is a seven kilolines perl script around rsync...

Re:rsnap is popular. Should pull from read-only ac

[shess]

rsnapshot is 7klines of Perl script around rsync which does _pull_ backups.

Re: Why is everything "trolling" to people like yo

[DFarmerTX]

I remember you.

Why not change the behavior of rm?

[ndykman]

So, first, people figured out that rm -rf / is bad, so they added an option that would disallow it by default, but you can turn it off, and rm -rf /* may still work.

But, the command never makes sense. What does it mean to remove /proc? Or /dev? People quote the ideal that you could be able to do really dumb things to be able to do really great things, but why allow things that don't make any sense at all? "rm -rf /dev/null" for example. What does that mean? It's seems reasonable to say "rm -r /" or "rm -r /*" has no meaning, as it asks to perform an operation on filesystems where it has no defined semantics, and therefore, you can always forbid it. After all, you can still list all the directories and files you want manually to get the same effect.

Re:Why not change the behavior of rm?

[flargleblarg]

Agreed. There is no reason ever to "rm -rf /" the whole filesystem. If you want everything wiped, you reformat the disk and start over with a new filesystem on it. Not only is this safer, it's cleaner and faster.

One word.

[sarkeizen]

Snapshots.

Bad summary

[gweihir]

First, "Rm -rf gets you an error because the comman "Rm" is unknown. And second, Unix-filesystems are not moron compatible (different from Windows) and recovering files is generally very hard or infeasible.

Use "set -u"

[fnj]

Some good points made by various posters, amongst all the piling on.

Here's one suggestion which, curiously, I don't see having been made by anyone yet. This is a good illustration of how putting "set -u" at the top of the script can help you. It would have prevented this catastrophe, actually. "Set -u" causes any use of an undefined variable to abort before executing the command, and the script will be terminated on the spot with a diagnostic identifying the particular variable as an "unbound variable".

It certainly won't prevent ALL rm boo-boos. but it would have prevented this one. It works in POSIX/BSD sh as well as bash.

Re:Should've used Windows...

[Tablizer]

Clippy: "It looks like you just deleted your entire company. Would you like assistance crying?"

Should have used windows

[bugs2squash]

'rm' is not recognized as an internal or external command, operable program or batch file.

Nice rush of adrenaline ...

[jopet]

this must have been!

Re:beed there, done that

[Purity Of Essence]

One day my web host decided to go out of business and it sent me scrambling to fix the damage. Unfortunately, my forums were not preserved anywhere. I had basic user account information from registration emails. I attempted to restore the public posts from Google caches and a lot of manual MySQL data entry. It was rough few days of pure panic. Between Google banning me for bot-like behavior and them updating their caches faster than I could copy them, I barely pulled it off with only a few lost posts. I'm never letting that happen again.

Re:rofl

[alvarogmj]

yes, until you start doing "sudo x" instead of "x" for everything... At least for me, being root means to be in a "full alert" state of mind.

Also, this supposedly was done by Ansible, why in hell was he issuing "rm -rf" commands from there? that is what the 'file' module is for

Less destructive, but far too common

[Y2K is bogus]

Init 1 instead of init q, that was always my favorite line that someone used to run on the main server, far too often as it turns out. On a Sun box that requires serial terminal to type "go", to recover from.

The Horror!

[Dutchmaan]

There's that brief moment of confusion, followed by the heart clench and a burst of fear in realizing what has just happened. Then the mind quickly scrambles for any and every solution that could possibly put things back to normal. Then you're hit by the icy cold wave of fear like being submerged in frigid water as you realize the ramifications of what has happened and predictions of what your increasingly bleak future may hold. After you realize all hope is lost you hit "fuck it" mode and just try to escape the situation in the outside world, but it's too late... that icy water you're submerged in goes with you.

How binary

[Sivar]

Funny how is number of users is 2^10 + 2^9 - 1. 'Lot of code smell in this article.

strings /dev/sda

[iamacat]

If your livelyhood really depends on the data, you can invest time to recover it after rm -rf /. Come up with patterns to identify your database records on disk blocks. A few will be split between blocks which are non continuous, but most can be found and dumped from raw device. Then exploit relationship between different records to identify outdated data. Finally, offer suppliers and customers a healthy discount to log in and correct remaining errors. After all, the value of a business is in human relationships more than any single hard drive.

Man Trolls Internet with One Post on Serverfault

[Gumbercules!!]

I fixed the article heading for you...

Re:Why is everything "trolling" to people like you

[Dahamma]

It was a pretty obvious troll if you read the whole thread:

1. The guy claims to have made the most insanely improbable mistake to kill his entire set of servers. Possible, but unlikely. Most took the bait on this one.
2. He had no explanation as to why "--no-preserve-root" didn't save him - basically looked like he didn't know about it, and he was lying.
3. later on he responded to someone's suggestion to use dd to backup saying he reversed if and of - which is probably the second most joked about UNIX sysadmin error after "rm -rf".

So, either you are pretty clueless about any of this, or, like another poster suggested, you are also a troll. Based on the specifics of your moronic post, probably the latter.

-rf {foo}/{bar} - Fake story.

[erexx23]

-rf {foo}/{bar} = fubar foobär/ adjective adjective: fubar; adjective: foobar out of working order; seriously, perhaps irreparably, damaged. "the clock in the hall is fubar"

Re:Repeat after me...

[YukariHirai]

It's where the principles of Murphy's Law and Schroedinger's Cat intersect - Murphy Schroedinger's Data, if you will.

Data stored in the cloud both exists and doesn't exist simultaneously; if you need the data it got accidentally deleted and there's no way to retrieve it, and if you delete the data because you don't want it seen, it turns out there's another copy and it will be all over the open internet shortly.

In 1991, our secretary changed the tapes

[Baki]

As part of "cost savings" back in 1991, managers decided that the secretary would be able to rotate the backup tapes, instead of IT staff.
After 6 months of hacking and developing, we had a crash of 2 disks simultaneously.
It turned out, all backups, offsite, offline, had all been made on cleaning tapes. Error messages had been ignored.

There is nothing new here. People and managers keep taking shortcuts and hope for the best.

Re:Foolproof backup strategy

[FormOfActionBanana]

a hosts file, properly configured, will do that for you more easily.

Re:Hobbling the default rm command, etc. +1

[pruneau]

I totally second that: http://docs.ansible.com/ansibl... Moderate parent to top level. rm is a tool, ansible too, badly using them is asking for trouble.

Backup is indeed impt.

[Geanne]

Hopefully the company able to do backup atleast every month.

Is it ironic?

[freeschwag]

I saw this right away.... Lol

foo bar ??? Yup. It sure is.

ServerFault Meta Admins Have Confirmed It A Hoax

[celest]

ServerFault Meta admins have confirmed that it was a deliberate "Guerilla Marketing" promotional hoax. Seems it was highly effective. The original thread has been deleted, though admins are still deciding what to do, if/how to punish user account, etc.

Re:Why is everything "trolling" to people like you

[allo]

> 2. He had no explanation as to why "--no-preserve-root" didn't save him
Maybe because it's a BSD or other system without gnu tools?

Re:rsnap is popular. Should pull from read-only ac

[tigersha]

Yes. This. PULL PULL PULL.

The system should not have write rights on the backup system. The backup system should have read rights on the system.

ZFS would have saved him

[MichaelJackson123]

With ZFS you can make a snapshot, i.e. a freeze of the entire disk in time. This means you can rollback to any of these time points when you want. These snapshots does not use additional space. This also means that if some virus messes up your system disk, no worry, because you can just rollback to an earlier snapshot in time. I have made much mistakes with Solaris, but each time I am going to do something weird, I make a snapshot (takes one second) and if I mess up, I just reboot in GRUB to an earlier state where the system disk worked fine. ZFS is a godsend.

Error checking is for chumps

[ebvwfbw]

Guy needs to be a life member in the Error Checking is for Chumps club.
I've seen very close, a chown/chmod -R /$path/*. Well $path wasn't defined. The guy called it through sudo. I had a record. No more sudo for him. Could still login as root, took an hour or so to fix.

Re:rsnap is popular. Should pull from read-only ac

[Krishnoid]

I constantly remind people on the rsnap mailing list about the existence of cryptolocker type malware.

Along those lines, could one write or modify a filesystem driver to add mount options which only allowed creation of files and directories (on, say, a snapshot-style-style backup volume), to combat those kinds of attacks?

mount -o remount

[raymorris]

Mount options may not do much good, because an attacker (or malware) could remount. Also for efficiency you often don't want to write everything again, you want to clone the old through some efficient mechanism such as hard links or volume tricks, then update it by updated files which have been updated, deleting deleted files, and creating any new files. A pull with read-only access to the live system does this well, and read-only is very well supported by existing file systems.

Re:manishs clearly has no idea...

[flargleblarg]

Why would anyone ever want to rm -rf /? Much better to reformat the disk and run mkfs.

Re:manishs clearly has no idea...

[flargleblarg]

Actually, that's stupid filesystem design, not stupid operating system design.

Story is not true.

[Still Having Fun]

See http://www.trymodern.com/artic...

Re:Why is everything "trolling" to people like you

[Dahamma]

Maybe because it's a BSD or other system without gnu tools?

Maybe you could RTFA. It wasn't.

What I'm not seeing in the Slashdot echo chamber..

[Wolfrider]

--HOWTO safe rm on Linux (at least):

- Interactive:
o Use Midnight Commander. Works from a text terminal, no GUI needed.
Insert to mark files/dirs, F8 to delete. Never had a misfire, even as root.

- Script (for the paranoid, and want a log):
IF you have a known dir " destdir=/tmp/blah " with a subdir " /tmp/blah/1 "
$ cd $destdir && cd 1 && cd .. && rm -rv 1/* |tee /tmp/rm.txt

- REALLY safe rm, with find:
# find multiple (known) names of files > 40 days old in a given directory and delete them
bkpath="/mnt/bkpdrive"
pathh="$bkpath/work/bkpsys-laptop-p2400-thinkpad-xubuntu-14-04-LTS--64--sda7"
cd $pathh && \
    find $pathh/* \( -name "bkp*gz" -o -name "bkp*bz2" -name "bkp*lzop" -o -name "flist*" \) -type f -mtime +40 -exec /bin/rm -v {} \;

--If you need to delete multiple levels of subdirectories, or dotfiles -- do it interactively, and use MC.

Re:Why is everything "trolling" to people like you

[allo]

Maybe it was a hoax. Maybe it WAS.

Re:Why is everything "trolling" to people like you

[Dahamma]

Yes, it was a hoax. It WAS. :)

http://www.snopes.com/man-dele...

CONFIRMED HOAX...

[Dahamma]

I know this article is "old" by now, but, it's already been 100% proven hoax, confirmed by the original perpetrator:

http://www.snopes.com/man-dele... /. lack of any technical editors or researchers strikes again!