Developer Accidentally Deletes Production Database On Their First Day On The Job (qz.com)

An anonymous reader quotes Quartz: "How screwed am I?" asked a recent user on Reddit, before sharing a mortifying story. On the first day as a junior software developer at a first salaried job out of college, his or her copy-and-paste error inadvertently erased all data from the company's production database. Posting under the heartbreaking handle cscareerthrowaway567, the user wrote, "The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that I 'completely fucked everything up.'"
The company's backups weren't working, according to the post, so the company is in big trouble now. Though Qz adds that "the court of public opinion is on the new guy's side. In a poll on the tech site the Register, less than 1% of 5,400 respondents thought the new developer should be fired. Forty-five percent thought the CTO should go."

Comments

How the fuck

[Gojira Shipi-Taro]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding. The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Re:How the fuck

[Anonymous Coward]

The entire CXX staff level should be let go. Why a person fresh off the street had permission to even make a mistake of such magnitude is beyond me.

Re: How the fuck

[Tomahawk]

Agreed. Has they said it was a month in then maybe it could be believed. But onlyâ day one you're only getting down around, here's your PC, here's your password, Joe will show you around the product later and explain what we do, the toilets are that way...

On day one, there'd be no production access. Nor access to source code. And if he did have that of the was access, and changes would be scrutinised.
So, yeah, it's a fake.

An interesting hypothetical scenario, though -- if it did happen, given the info available, then who's to blame?

The new dev? No: first day, fresh out of college, mistakes should be expected. He shouldn't have had that level of access, and his immediate superiors should have been keeping a close eye.

His immediate superiors? Possibly. How did they let someone new make such a big error? Why did they allow him to do anything at all?

The DBAs? No working backups on a production database? No transaction logs that could be rolled back? No DR solution in place? Basic stuff here that were all missed. So definitely some blame sits her.

CTO? Hard to say. Certainly policies should be in place to ensure this shouldn't happen, so why did it under their watch? Were the staff too overworked that they didn't have time to get the basics right?

To properly sign any blame, more information is needed. We don't have all the facts. To many questions remain unanswered.

There is a general human flaw that this story highlights, though, which is that the majority will sign blame based on too few facts. This comes up time and time again, both here and elsewhere. Take, for example, "Making A Murderer" -- how many petitioned to have Stephen released based on 10 hours of a TV show that really only showed one side of the story? There were more than 200 hours' of testimony to try to show all of the available facts for the jury to make a decision.

This is a similar case -- one reddit post describing things from the point of view of developer who was on his first day ever working. There's tonnes of missing information here.

Still, and interesting scenario to ponder as a thought experiment.

Re:How the fuck

[Calydor]

If it wasn't a pain in the ass it wouldn't be called work.

Re:Why Was He Mucking With It In The First Place?

[sanosuke001]

He was given a worksheet with information regarding how to clone a test database onto his workstation. He ran a script when printed out the test DB's URL and login information but he just copied the info off the worksheet instead. The info on the worksheet was the production database with an account containing full write access. He overwrote the production database with the test data he was attempting to clone.

Whoever gave a new hire a worksheet with a URL for the production server and login with full access should be the one fired. This new kid showed the company that their new hire setup is completely insecure and that their backup infrastructure doesn't work. It's not this kids fault.

Re:"Their backups weren't working."

[Just Some Guy]

Okay, the guy fucked up ROYALLY.

I don't think he did. I actually RTFA this time, and the guy was following the onboarding directions he was given. Where it went south was that he copied-and-pasted the wrong database credentials. He was supposed to use the username and password that a command had spit out, but he instead used the ones from the onboarding docs.

I'll pause for a moment to let that sink in.

Some jackass had put actual prod root creds in the onboarding docs, then gave them to a new graduate fresh on his first day of his first job, then walked away while he onboarded himself without supervision.

This poor kid did absolutely nothing wrong except misreading some instructions. The engineering team responsible for the chain of events that led to this colossal fuck are completely and wholly to blame.

We need to have medals to give out.

[clovis]

I say if he succeeded in putting that company out of business, then he should get a medal for sacrificing himself to destroy the company.

My belief is when he saw on his first day, the badly written docs they handed him, with a printed (!) account/password having RW access, he instinctively threw himself on that grenade by destroying their production database. Only the most cowardly IT worker would have done otherwise.

Thank you, selfless IT worker from saving us from the horror of whatever product they were trying to produce.

Re:"Their backups weren't working."

[lucm]

The fact that their backup system was non-functional is double-plus unforgiveable.

In my experience, continuous SAN replication is often to blame for a poor backup strategy. It creates the illusion of security - yes, your DR site is synchronized with production within seconds or milliseconds, but guess what, mistakes are also replicated.

Replication -> floods, fire and similar disasters
Backups -> oops my bad

Both are needed.

Re: Old News...

[KGIII]

Retired boss, here. If a junior dev can push to prod AND can delete data, it isn't their fault. Okay... I am still going to try to salvage my hire and see if they fit in QA, or see if the debs will stop fucking with him. Seriously, not his fault. Nobody should be able to push to prod, withou someone signing off. Ask me how I fucking know. I'm not even a programmer. I just paid a lot of you and shut the hell up and listened.

So, if I am wrong then they were wrong. You don't push code to prod without someone signing off. The person signing off has ultimate authority. You sure as fuck don't let a junior do it, without oversight. Not now, not ever.

If this happened in my shop, some titles would have been changed.