Developer Accidentally Deletes Production Database On Their First Day On The Job (qz.com)

An anonymous reader quotes Quartz: "How screwed am I?" asked a recent user on Reddit, before sharing a mortifying story. On the first day as a junior software developer at a first salaried job out of college, his or her copy-and-paste error inadvertently erased all data from the company's production database. Posting under the heartbreaking handle cscareerthrowaway567, the user wrote, "The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that I 'completely fucked everything up.'"
The company's backups weren't working, according to the post, so the company is in big trouble now. Though Qz adds that "the court of public opinion is on the new guy's side. In a poll on the tech site the Register, less than 1% of 5,400 respondents thought the new developer should be fired. Forty-five percent thought the CTO should go."

Comments

How the fuck

[Gojira Shipi-Taro]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding. The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Re:How the fuck

[Anonymous Coward]

The entire CXX staff level should be let go. Why a person fresh off the street had permission to even make a mistake of such magnitude is beyond me.

Re:How the fuck

[lucm]

The entire CXX staff level should be let go.

First, "CXX" level is not "staff". Second, firing the entire senior management team because of this incident would be completely reckless. I understand the outrage but that's not how companies work in real life. The last thing a company in that situation needs is more instability.

The proper way to address this is to stabilize the situation, then make sure the problem cannot occur again. And this typically doesn't simply mean firing people, because odds are that there are cultural or organizational factors that made this situation possible (crazy deadlines, shoestring budgets, etc.), and those would probably lead the replacement to making the same kind of mistakes down the road.

What is needed is new processes and controls. You start with a simple governance framework (like COBIT maybe) where each part of the IT ecosystem is linked to a specific business leader, then you let each of those leaders make sure that their area of responsibility is well managed from a risk perspective. That's how you make the company more resilient, not by firing people who maybe were not empowered to make the right decisions in the first place.

Re:How the fuck

[sexconker]

Additionally if the backups aren't working, that's something that could have saved the company - if the CTO had made sure that they worked properly.

I'd probably think twice before hiring the new guy as well over this, although at the same time I'd probably limit access to just what he needed to do his job. I'd also make sure I have proper backups.

But I'd avoid the CTO at all costs, at least as a CTO. (OR C- anything if this is how he operates.)

I reconfigured our backups recently. It's a huge pain in the ass because I had to wait for the end of the month, move everything to other storage, recreate new jobs from scratch, wait for them to run, copy to a second location, and perform automated validation checks in both locations, then do it all again (making sure version chaining worked), then actually go in and open up every single new backup, in both locations, to ensure they all worked, the correct passwords were used, etc.

But I know they fucking work.

Re:How the fuck

[dbIII]

The article said the business had a couple hundred people at most and 40+ developers.

Not exactly a small business deluded sig guy.

Re: How the fuck

[Tomahawk]

Agreed. Has they said it was a month in then maybe it could be believed. But onlyâ day one you're only getting down around, here's your PC, here's your password, Joe will show you around the product later and explain what we do, the toilets are that way...

On day one, there'd be no production access. Nor access to source code. And if he did have that of the was access, and changes would be scrutinised.
So, yeah, it's a fake.

An interesting hypothetical scenario, though -- if it did happen, given the info available, then who's to blame?

The new dev? No: first day, fresh out of college, mistakes should be expected. He shouldn't have had that level of access, and his immediate superiors should have been keeping a close eye.

His immediate superiors? Possibly. How did they let someone new make such a big error? Why did they allow him to do anything at all?

The DBAs? No working backups on a production database? No transaction logs that could be rolled back? No DR solution in place? Basic stuff here that were all missed. So definitely some blame sits her.

CTO? Hard to say. Certainly policies should be in place to ensure this shouldn't happen, so why did it under their watch? Were the staff too overworked that they didn't have time to get the basics right?

To properly sign any blame, more information is needed. We don't have all the facts. To many questions remain unanswered.

There is a general human flaw that this story highlights, though, which is that the majority will sign blame based on too few facts. This comes up time and time again, both here and elsewhere. Take, for example, "Making A Murderer" -- how many petitioned to have Stephen released based on 10 hours of a TV show that really only showed one side of the story? There were more than 200 hours' of testimony to try to show all of the available facts for the jury to make a decision.

This is a similar case -- one reddit post describing things from the point of view of developer who was on his first day ever working. There's tonnes of missing information here.

Still, and interesting scenario to ponder as a thought experiment.

Re:How the fuck

[Calydor]

If it wasn't a pain in the ass it wouldn't be called work.

Re: How the fuck

[ColaMan]

I read the thread over on reddit and basically he turned up, was given his dev machine, and then given the instructions for creating the dev environment + db's & etc on his machine.

Then after a while he was like, "OK, let's see, step 17: Enter these commands to create your dev DB."

So he copied and pasted the commands from the document, like he'd been doing for the last 16 steps.

Unfortunately, the commands had the production db's connection in them. And they also had queries like, "drop database mainDB; create database mainDB; create table etc etc"

Whoopsie!

Re: How the fuck

[war4peace]

I'll tell you how. (names below are fictional)
Netadmin or Sysadmin team has no content writer, so Prakash is told to create an induction manual for new hires. Prakash hates this assignment (understandably, he's a net admin, not a fucking content writer) so he does the bare minimum: takes screenshots of prod environments and enters credentials for that generic admin account everyone was using (because fuck processes) as example.
New hire comes, proceeds going through the document, and with his lacking attention to details (or because he's overzealous, or been told to beat the best time for new hires of 30 minutes 15 seconds) doesn't pay attention to the step saying "enter YOUR given connection details" and copy/pastes the ones shown as example, which incidentally are absolutely valid and point out to the PROD DB.

Disaster happens.

Moral of the story?

1. Get a proper content writer to create documentation: responsibility falls to management to ask for it and senior management to approve it. Blame falls on whoever didn't ask or didn't approve it.
2. Review the documentation: responsibility falls to line manager Prakash is under. Blame that line manager and hang him (figuratively) from a tall sturdy branch.
3. Publish the document as a controlled document in a knowledge management environment. If such an environment doesn't exist, blame everyone who didn't ask for it or approve it.

The CTO is hardly to blame, it's not his business to handle such processes, that should fall under a line manager or whichever dedicated person was supposed to handle it.

I personally wouldn't have kept a netadmin/sysadmin who can't follow basic instructions or a manager who didn't review the training document. Everyone else is off the hook because they were either not aware of the risk or did what they could with a task that wasn't in their job description.

Re: How the fuck

[Defakto]

As a manager and supervisor I wouldn't put you out on the floor without a senior to ensure you could handle and to QA the work. The first 6-12 months are hour learning and trust building phase. I expect mistakes during this time and I expect your peers to work with you on new projects to ensure those mistakes don't happen .

Re: How the fuck

[Anne Thwacks]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Hell, even on a one man business, development should not be done on the same machine as production activities.

Even my home machines have off-site table backups - and yes I have tested recovery works - cross platform - on different hardware and OS.

if your data is worth money, you should be prepared to spend money to protect it. If its not worth anything, then just delete it yourself and save the disk space.

Disclaimer: I do not have an MBA.

Re: How the fuck

[whoever57]

The CTO is hardly to blame, it's not his business to handle such processes

But it is his job to ensure appropriate security and backups for the production database.

Re:How the fuck

[bigtiny]

I'm always amazed when something like this happens, a lot of people's first reaction is 'who gets fired'? It's really not a productive attitude to take in a case like this.
I find it problematic that
- companies take an agile 'quick turnaround' approach to development without seeming to understand the risks, until they get bitten. This is an example.
- seems that whoever's managing the dev team should have had a break-in period/mentoring system in place to make sure new hires (especially newbies straight out of college) know their way around before pushing production code.
- whoever's managing the place is REALLY on the hook if the data cannot be recovered! If a manager/company is not willing to provide the personnel and tools necessary to insure backup and restore integrity, then they have only themselves to blame.

Firing this kid, in this situation is stupid. In a way, he did them a favor....now they know what they need to fix.

Re: How the fuck

[lgw]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Welcome to DevOps. It's the fad sweeping the industry. Yes, it's exactly as stupid as you think it is. It's so obviously wrong, so mind-bogglingly stupid that only a manager with an MBA could possibly have thought it up. So of course it's the latest fad.

"Their backups weren't working."

[Chas]

Okay, the guy fucked up ROYALLY.

It happens. And he SHOULD get in a bit of trouble for it. That's how you learn "don't do that". I don't think they deserve to lose their job though.

The CTO and all the people in charge of the backups need to be on the street YESTERDAY though. That the dev COULD do something like this is a major fuckup on their part. They simply didn't have their production system locked down properly.

The fact that their backup system was non-functional is double-plus unforgiveable. The dev is merely the highlight for their massive cluster-fuck of a setup.

Re:"Their backups weren't working."

[Just Some Guy]

Okay, the guy fucked up ROYALLY.

I don't think he did. I actually RTFA this time, and the guy was following the onboarding directions he was given. Where it went south was that he copied-and-pasted the wrong database credentials. He was supposed to use the username and password that a command had spit out, but he instead used the ones from the onboarding docs.

I'll pause for a moment to let that sink in.

Some jackass had put actual prod root creds in the onboarding docs, then gave them to a new graduate fresh on his first day of his first job, then walked away while he onboarded himself without supervision.

This poor kid did absolutely nothing wrong except misreading some instructions. The engineering team responsible for the chain of events that led to this colossal fuck are completely and wholly to blame.

Re:"Their backups weren't working."

[lucm]

The fact that their backup system was non-functional is double-plus unforgiveable.

In my experience, continuous SAN replication is often to blame for a poor backup strategy. It creates the illusion of security - yes, your DR site is synchronized with production within seconds or milliseconds, but guess what, mistakes are also replicated.

Replication -> floods, fire and similar disasters
Backups -> oops my bad

Both are needed.

Re:"Their backups weren't working."

[Just Some Guy]

But his mistake was easy to make and should have resulted in an "access denied" error message. If you give a five year old a hammer and say "don't whack shit with this", and they whack shit with it, you're the one done goofed.

Re:Why Was He Mucking With It In The First Place?

[sanosuke001]

He was given a worksheet with information regarding how to clone a test database onto his workstation. He ran a script when printed out the test DB's URL and login information but he just copied the info off the worksheet instead. The info on the worksheet was the production database with an account containing full write access. He overwrote the production database with the test data he was attempting to clone.

Whoever gave a new hire a worksheet with a URL for the production server and login with full access should be the one fired. This new kid showed the company that their new hire setup is completely insecure and that their backup infrastructure doesn't work. It's not this kids fault.

If backups are not working and this is known...

[nomad63]

If a company looks at a non-working backups as a minor inconvenience, I think the CTO's ass should be on the firing line before this poor guy's. Yes, wat he did is inexcusable and in some cases, firing might be justifiable (as in, a junior developer on his first day in his/her job doing what on a production database case) but if someone assigned him/her to perform anything on the company-critical data, that person should be the one getting fired, not this guy's. I am a 20+ year experienced sysadmin and not too long time ago, when I started at a new position, I was not able to touch any system other than few development machines for 2+ months of my start date and I know/knew my shit. This company management shows their incompetency in more ways than one. Yet they are making this person the scapegoat. Good riddance to them, as their days are numbered.

Some Company Information Please

[mykepredko]

I'm surprised that the firm has not been named - while I would think that any company that had this happen to them would want to keep this confidential, I would think that somebody would talk about it separately. I suspect that the "company" is some podunk startup in which the CTO is the CEO, CFO, head of development and probably the HR head and they've just hired a developer without thinking about access restrictions (or verifying that backups are actually happening).

Some more information would help clarify these questions and maybe better explain how such a situation could happen.

Re:Some Company Information Please

[ryen]

The author in the reddit thread (in a buried comment) describes it as a 100+ company with at least 40 devs. Pretty sad

We need to have medals to give out.

[clovis]

I say if he succeeded in putting that company out of business, then he should get a medal for sacrificing himself to destroy the company.

My belief is when he saw on his first day, the badly written docs they handed him, with a printed (!) account/password having RW access, he instinctively threw himself on that grenade by destroying their production database. Only the most cowardly IT worker would have done otherwise.

Thank you, selfless IT worker from saving us from the horror of whatever product they were trying to produce.

be happy

[ooloorie]

You don't want to work at a company where the backups don't work and where a new hire can accidentally delete all their data. Don't beg to stay, instead be happy that you found out quickly how incompetent that company actually is.

Re:Why Was He Mucking With It In The First Place?

[Kjella]

Having production databases that can be reached from developers workstations is always a bad idea.

Welcome to DevOps ;)

Re:Why Was He Mucking With It In The First Place?

[Shadyman]

Having production databases that can be reached from developers workstations is always a bad idea. They were lucky, in a way, that the developer deleted the data and didn't just alter it slightly.

You'd also have to pray they did not alter it any further.

Re: Old News...

[KGIII]

Retired boss, here. If a junior dev can push to prod AND can delete data, it isn't their fault. Okay... I am still going to try to salvage my hire and see if they fit in QA, or see if the debs will stop fucking with him. Seriously, not his fault. Nobody should be able to push to prod, withou someone signing off. Ask me how I fucking know. I'm not even a programmer. I just paid a lot of you and shut the hell up and listened.

So, if I am wrong then they were wrong. You don't push code to prod without someone signing off. The person signing off has ultimate authority. You sure as fuck don't let a junior do it, without oversight. Not now, not ever.

If this happened in my shop, some titles would have been changed.

Recent grad given full production access

[DukeLinux]

What even marginally competent IT manager would give an inexperienced person the ability to modify or delete production objects? I feel sorry for this guy, but his management is completely at fault here. We know the story. Nothing will happen to them and one or more will likely get promoted.

Re: Old News...

[KGIII]

I gotta stop trying to type on a tablet.

Anyhow, I'd not want to fire anyone, without more information at hand. Firing people has large costs associated with it.

How to describe this?

As stated, I am retired. I've been retired for over ten years. Back in my day, we trained people and paid for them to continue their education. Don't laugh, that really used to happen.

So, I'd probably try to salvage this as a teaching moment. I'd also be changing someone else to the lead position. If I were to fire anyone, it might be the person who enabled this to happen and probably wouldn't be the junior.

Our shop was kinda laid back. It is unlikely that I'd fire anyone, but someone is going to catch a whole ration of shit. If this is a habit, then someone is getting fired, but if this is a habit then it speaks to a larger problem.

I'd start with the junior dev and work my way up. I'd keep digging until I found the problem in the process. I'd then take time to listen and find out what we need to do to make sure it never happens again. There's gonna be a meeting. Hell, there are going to be several meetings. Chances are, this is going to have a bunch of little teaching moments.

Junior devs, autocorrect hates that word and thinks it should be debs, sure as hell shouldn't be able to push to production. Even if they could, those backups damned well better be timely and functional. The IT staff, I guess you call them ops today, were great at their jobs, thankfully. For reasons, we had a pretty robust backup methodology. I am still flummoxed when I hear stories about not backing up properly.

I haven't done this for a while, so it is time to do it again.

I used a computer because I had to. I actually kinda hated them, for the longest time. I am not a programmer, but I programmed because I had a task that needed to get done. I'm not an admin, but I did that job because it also needed to get done.

Then, I was able to hire capable people. I was able to hire competent developers and IT staff. It was those people who did much of the heavy lifting. I learned a lot from them. I learned what best practices were and why they were that way.

So, as I said, I've not taken the time to do so lately. Here's a tip of the hat, and a nod, to those folks in the admin side and in the developer side. Here's a tip of the hat to the database admins and to the jerks who secured my network. Here is a tip of the hat to the programmers and to the QA. Here's a tip of the hat to those who spent long hours beside me, enabling me and teaching me.