Developer Accidentally Deletes Production Database On Their First Day On The Job (qz.com)

An anonymous reader quotes Quartz: "How screwed am I?" asked a recent user on Reddit, before sharing a mortifying story. On the first day as a junior software developer at a first salaried job out of college, his or her copy-and-paste error inadvertently erased all data from the company's production database. Posting under the heartbreaking handle cscareerthrowaway567, the user wrote, "The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that I 'completely fucked everything up.'"
The company's backups weren't working, according to the post, so the company is in big trouble now. Though Qz adds that "the court of public opinion is on the new guy's side. In a poll on the tech site the Register, less than 1% of 5,400 respondents thought the new developer should be fired. Forty-five percent thought the CTO should go."

Comments

How the fuck

[Gojira Shipi-Taro]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding. The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Re:How the fuck

[Anonymous Coward]

The entire CXX staff level should be let go. Why a person fresh off the street had permission to even make a mistake of such magnitude is beyond me.

Re:How the fuck

[blindseer]

How the fuck does a new hire have that kind of access?

Having worked at some small businesses before it seems to me to be pretty common. The article said the business had a couple hundred people at most and 40+ developers. Quite likely the people there were there for a long time and they hired a handful of people since they now had a need for some help and finally some money to pay them. This is how things were done for years and not remembering what can happen if a newbie fucks up once in a while they thought nothing about handing over the documentation they used themselves, and never really having to test the documentation against a newbie before they didn't consider the consequences of what might happen if the doc was followed to the letter and/or with a minor mistype like inputting the production servers instead of the developers servers.

The lack of a proper backup is inexcusable. I learned that in college after having lost the only copy to a semester project a week before the end of the semester. After that anything I deemed "important" I kept a backup where it was not likely to be deleted accidentally.

I worked for a larger organization doing some development and I was never given access to the production servers. Once I could prove that I had something working I would hand my files over to a manager and he'd copy the files over himself. This was always a bit of a tense moment because, due to budget constraints for the project, I was using MacOS/Apache/MySQL/PHP for testing while the production servers were Windows/IIS/MSSQL/PHP. It seems everything was close enough because I didn't hear anyone complain about broken code. If something didn't work then it would have been very time consuming to roll back and also very difficult to debug.

This also makes me think a bit on why developers, even newbies, would have access to the production servers. If something broke between the transition between the development system and production then they'd want it fixed right away. This would be easier to do if the developers had access to the production servers. This would be avoided if the production and development systems were identical but like in that project I mentioned it might be that budget constraints prevented them from doing so.

Re:How the fuck

[lucm]

The entire CXX staff level should be let go.

First, "CXX" level is not "staff". Second, firing the entire senior management team because of this incident would be completely reckless. I understand the outrage but that's not how companies work in real life. The last thing a company in that situation needs is more instability.

The proper way to address this is to stabilize the situation, then make sure the problem cannot occur again. And this typically doesn't simply mean firing people, because odds are that there are cultural or organizational factors that made this situation possible (crazy deadlines, shoestring budgets, etc.), and those would probably lead the replacement to making the same kind of mistakes down the road.

What is needed is new processes and controls. You start with a simple governance framework (like COBIT maybe) where each part of the IT ecosystem is linked to a specific business leader, then you let each of those leaders make sure that their area of responsibility is well managed from a risk perspective. That's how you make the company more resilient, not by firing people who maybe were not empowered to make the right decisions in the first place.

Re:How the fuck

[sexconker]

Additionally if the backups aren't working, that's something that could have saved the company - if the CTO had made sure that they worked properly.

I'd probably think twice before hiring the new guy as well over this, although at the same time I'd probably limit access to just what he needed to do his job. I'd also make sure I have proper backups.

But I'd avoid the CTO at all costs, at least as a CTO. (OR C- anything if this is how he operates.)

I reconfigured our backups recently. It's a huge pain in the ass because I had to wait for the end of the month, move everything to other storage, recreate new jobs from scratch, wait for them to run, copy to a second location, and perform automated validation checks in both locations, then do it all again (making sure version chaining worked), then actually go in and open up every single new backup, in both locations, to ensure they all worked, the correct passwords were used, etc.

But I know they fucking work.

Re:How the fuck

[dbIII]

What is needed is new processes and controls

Yes, but I think what people are getting at here is that current policies appear to be so fucked up that the team who implemented them may be unlikely to implement new processes and controls that are any better than they have already done.
I've seen that sort of stuff in a few places, (notably government owned corporations - worst of both worlds) where management have been chosen for reasons other than ability.

As shown by this incident that sort of mismanagement is often self-correcting, but it's a shame that so many others have to go down with the ship when the place goes under.

Re:How the fuck

[dbIII]

The article said the business had a couple hundred people at most and 40+ developers.

Not exactly a small business deluded sig guy.

Re: How the fuck

[Tomahawk]

Agreed. Has they said it was a month in then maybe it could be believed. But onlyâ day one you're only getting down around, here's your PC, here's your password, Joe will show you around the product later and explain what we do, the toilets are that way...

On day one, there'd be no production access. Nor access to source code. And if he did have that of the was access, and changes would be scrutinised.
So, yeah, it's a fake.

An interesting hypothetical scenario, though -- if it did happen, given the info available, then who's to blame?

The new dev? No: first day, fresh out of college, mistakes should be expected. He shouldn't have had that level of access, and his immediate superiors should have been keeping a close eye.

His immediate superiors? Possibly. How did they let someone new make such a big error? Why did they allow him to do anything at all?

The DBAs? No working backups on a production database? No transaction logs that could be rolled back? No DR solution in place? Basic stuff here that were all missed. So definitely some blame sits her.

CTO? Hard to say. Certainly policies should be in place to ensure this shouldn't happen, so why did it under their watch? Were the staff too overworked that they didn't have time to get the basics right?

To properly sign any blame, more information is needed. We don't have all the facts. To many questions remain unanswered.

There is a general human flaw that this story highlights, though, which is that the majority will sign blame based on too few facts. This comes up time and time again, both here and elsewhere. Take, for example, "Making A Murderer" -- how many petitioned to have Stephen released based on 10 hours of a TV show that really only showed one side of the story? There were more than 200 hours' of testimony to try to show all of the available facts for the jury to make a decision.

This is a similar case -- one reddit post describing things from the point of view of developer who was on his first day ever working. There's tonnes of missing information here.

Still, and interesting scenario to ponder as a thought experiment.

Re:How the fuck

[Calydor]

If it wasn't a pain in the ass it wouldn't be called work.

Re: How the fuck

[ColaMan]

I read the thread over on reddit and basically he turned up, was given his dev machine, and then given the instructions for creating the dev environment + db's & etc on his machine.

Then after a while he was like, "OK, let's see, step 17: Enter these commands to create your dev DB."

So he copied and pasted the commands from the document, like he'd been doing for the last 16 steps.

Unfortunately, the commands had the production db's connection in them. And they also had queries like, "drop database mainDB; create database mainDB; create table etc etc"

Whoopsie!

Re:How the fuck

[tburkhol]

The proper way to address this is to stabilize the situation, then make sure the problem cannot occur again.

You're expecting rational behavior from a business? Good luck. /s

I'd add that the size of the business probably matters a lot here. If this happened in a shop of 20 people with 3 computer guys, one of whom is "CTO," it's a very different situation than a billion dollar, multi-site company. It's probably smart to send the new guy home in the immediate aftermath - he's not going to be helpful in the recovery and emotions will be very high. After that, bring him back in, sort out what happened, and how to prevent similar failures of process. Dude has the cost of a fuck-up burned into his brain, and he's going to be a lot more careful and deliberate in the future.

Now, if he does it a second time...

Re: How the fuck

[JWW]

How the heck did the copy/paste text include valid credentials to even access the production database?!?

Re: How the fuck

[war4peace]

I'll tell you how. (names below are fictional)
Netadmin or Sysadmin team has no content writer, so Prakash is told to create an induction manual for new hires. Prakash hates this assignment (understandably, he's a net admin, not a fucking content writer) so he does the bare minimum: takes screenshots of prod environments and enters credentials for that generic admin account everyone was using (because fuck processes) as example.
New hire comes, proceeds going through the document, and with his lacking attention to details (or because he's overzealous, or been told to beat the best time for new hires of 30 minutes 15 seconds) doesn't pay attention to the step saying "enter YOUR given connection details" and copy/pastes the ones shown as example, which incidentally are absolutely valid and point out to the PROD DB.

Disaster happens.

Moral of the story?

1. Get a proper content writer to create documentation: responsibility falls to management to ask for it and senior management to approve it. Blame falls on whoever didn't ask or didn't approve it.
2. Review the documentation: responsibility falls to line manager Prakash is under. Blame that line manager and hang him (figuratively) from a tall sturdy branch.
3. Publish the document as a controlled document in a knowledge management environment. If such an environment doesn't exist, blame everyone who didn't ask for it or approve it.

The CTO is hardly to blame, it's not his business to handle such processes, that should fall under a line manager or whichever dedicated person was supposed to handle it.

I personally wouldn't have kept a netadmin/sysadmin who can't follow basic instructions or a manager who didn't review the training document. Everyone else is off the hook because they were either not aware of the risk or did what they could with a task that wasn't in their job description.

Re:How the fuck

[Ol Olsoc]

What is needed is new processes and controls.

What is needed is a backup system that actually works, and is used.

I never trusted our official backup system, having seen it not work on several occasions. So I installed one of my own for the group. Sure enough, around a year later, a group member called in a total panic - she'd written a script that was supposed to perform a find then print the find results. What it managed to do was delete the whole database.

Calls me in a tearful freakout - the IT folks backup didn't work.

Mine did. There were a few things here that needed addressed. She should have been working on a copy of the database, not the main one. Shit does happen, and if she accidentally deletes a copy due to a derpy moment, its no problem at all. Fix the script, and work on a new copy.

Second is don't ever trust a backup that you can't personally verify will work.

Third is don't be afraid to have backups for backups for backups. People called me Mister OCD or Monk (referring to the television show about a detective with serious OCD issues) but after that the little jabs mysteriously went away.

Re: How the fuck

[Defakto]

As a manager and supervisor I wouldn't put you out on the floor without a senior to ensure you could handle and to QA the work. The first 6-12 months are hour learning and trust building phase. I expect mistakes during this time and I expect your peers to work with you on new projects to ensure those mistakes don't happen .

Re: How the fuck

[Anne Thwacks]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Hell, even on a one man business, development should not be done on the same machine as production activities.

Even my home machines have off-site table backups - and yes I have tested recovery works - cross platform - on different hardware and OS.

if your data is worth money, you should be prepared to spend money to protect it. If its not worth anything, then just delete it yourself and save the disk space.

Disclaimer: I do not have an MBA.

Re:How the fuck

[slappynipsy]

Testing the backups is a huge pain in the ass but in the immortal words of Don Draper, that's what the money's for!

Re: How the fuck

[subanark]

I'm a developer for Azure. The service I develop doesn't have a DBA or any test team. Could I screw things up? Yes, quite easily, however since the DBs are on XStore, we can just pick a date and time within the last week and snap the DB back to that state. I don't run a big enough or impactful enough service to warrant this level of support.

Re: How the fuck

[whoever57]

The CTO is hardly to blame, it's not his business to handle such processes

But it is his job to ensure appropriate security and backups for the production database.

Re: How the fuck

[HornWumpus]

No _competent_ DBA is not the same as no DBA...

Re:How the fuck

[bigtiny]

I'm always amazed when something like this happens, a lot of people's first reaction is 'who gets fired'? It's really not a productive attitude to take in a case like this.
I find it problematic that
- companies take an agile 'quick turnaround' approach to development without seeming to understand the risks, until they get bitten. This is an example.
- seems that whoever's managing the dev team should have had a break-in period/mentoring system in place to make sure new hires (especially newbies straight out of college) know their way around before pushing production code.
- whoever's managing the place is REALLY on the hook if the data cannot be recovered! If a manager/company is not willing to provide the personnel and tools necessary to insure backup and restore integrity, then they have only themselves to blame.

Firing this kid, in this situation is stupid. In a way, he did them a favor....now they know what they need to fix.

Re: How the fuck

[lgw]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Welcome to DevOps. It's the fad sweeping the industry. Yes, it's exactly as stupid as you think it is. It's so obviously wrong, so mind-bogglingly stupid that only a manager with an MBA could possibly have thought it up. So of course it's the latest fad.

Re: How the fuck

[war4peace]

The fact that the behavior you're presenting seems normal to you scares me, because it shows how widely accepted this has become.
While I agree that the admin should start writing the draft, their job would end there. Technical people rarely can produce proper documentation, and whether the documentation is internal or customer-facing is irrelevant. Standards should be the same. the mindset you're presenting is similar to that of people writing in bad English online because "fuck it, it's not important, I can write good when I have to".

Writing note for tasks != procedures. Also procedures != processes. Also induction manuals != processes or procedures (albeit they often overlap, still they are not identical).

At the very least, you need a dedicated person (or team, depending on how big the organization is) to verify, review and vet all documentation (except for private notes, comments and such) and store it in a controlled, unique repository. This could be as simple as a collection of printed brochures for small companies, or a wiki, or a more complex Knowledge Management System. The person or team assigned to that bears the ultimate responsibility. The CTO (of a larger organization, of course) has better things to do than review all documentation. Responsibility is then compartmentalized.

I've seen my fair share of horrible documentation over the years. often I have to go and learn what the processes inside a certain group or team are built, in order to understand the data generators (I develop analyses and reports) as well as why that data is generated, how is it generated and what does it mean from a business perspective. It greatly helps me provide correct, helpful and thorough analyses and reports. The problem is that the documentation is most times a mess, usually a collection of disparate word and excel files of various versions circulating through e-mail back and forth like feces in a sewer. Open one and you'll be hit by a jumble of acronyms I'm sure the tech people understand, but I have no idea what they mean.

Yes, from your point of view, that little information bubble you're in, it all makes perfect sense and you're used to it. Introduce a stranger to it and they'll stare at you blankly while you're thinking "how stupid is this guy?"

TL;DR version: Isolated communities develop their own unintelligible language, while isolated dev teams develop their own unintelligible documentation.

"Their backups weren't working."

[Chas]

Okay, the guy fucked up ROYALLY.

It happens. And he SHOULD get in a bit of trouble for it. That's how you learn "don't do that". I don't think they deserve to lose their job though.

The CTO and all the people in charge of the backups need to be on the street YESTERDAY though. That the dev COULD do something like this is a major fuckup on their part. They simply didn't have their production system locked down properly.

The fact that their backup system was non-functional is double-plus unforgiveable. The dev is merely the highlight for their massive cluster-fuck of a setup.

Re:"Their backups weren't working."

[Just Some Guy]

Okay, the guy fucked up ROYALLY.

I don't think he did. I actually RTFA this time, and the guy was following the onboarding directions he was given. Where it went south was that he copied-and-pasted the wrong database credentials. He was supposed to use the username and password that a command had spit out, but he instead used the ones from the onboarding docs.

I'll pause for a moment to let that sink in.

Some jackass had put actual prod root creds in the onboarding docs, then gave them to a new graduate fresh on his first day of his first job, then walked away while he onboarded himself without supervision.

This poor kid did absolutely nothing wrong except misreading some instructions. The engineering team responsible for the chain of events that led to this colossal fuck are completely and wholly to blame.

Re:"Their backups weren't working."

[lucm]

The fact that their backup system was non-functional is double-plus unforgiveable.

In my experience, continuous SAN replication is often to blame for a poor backup strategy. It creates the illusion of security - yes, your DR site is synchronized with production within seconds or milliseconds, but guess what, mistakes are also replicated.

Replication -> floods, fire and similar disasters
Backups -> oops my bad

Both are needed.

Re:"Their backups weren't working."

[Just Some Guy]

But his mistake was easy to make and should have resulted in an "access denied" error message. If you give a five year old a hammer and say "don't whack shit with this", and they whack shit with it, you're the one done goofed.

Re:Why Was He Mucking With It In The First Place?

[sanosuke001]

He was given a worksheet with information regarding how to clone a test database onto his workstation. He ran a script when printed out the test DB's URL and login information but he just copied the info off the worksheet instead. The info on the worksheet was the production database with an account containing full write access. He overwrote the production database with the test data he was attempting to clone.

Whoever gave a new hire a worksheet with a URL for the production server and login with full access should be the one fired. This new kid showed the company that their new hire setup is completely insecure and that their backup infrastructure doesn't work. It's not this kids fault.

If backups are not working and this is known...

[nomad63]

If a company looks at a non-working backups as a minor inconvenience, I think the CTO's ass should be on the firing line before this poor guy's. Yes, wat he did is inexcusable and in some cases, firing might be justifiable (as in, a junior developer on his first day in his/her job doing what on a production database case) but if someone assigned him/her to perform anything on the company-critical data, that person should be the one getting fired, not this guy's. I am a 20+ year experienced sysadmin and not too long time ago, when I started at a new position, I was not able to touch any system other than few development machines for 2+ months of my start date and I know/knew my shit. This company management shows their incompetency in more ways than one. Yet they are making this person the scapegoat. Good riddance to them, as their days are numbered.

Allowing it to happen is wrong to begin with

[sentiblue]

I've been in production management for 15 years... The first 5 years as a Sr. datacenter engineer. The next 7 years as a staff engineer and the past few years as a cloud architect. One thing that has NEVER changed is that developers are NOT allowed to touch production... and I mean not allowed to even log into any host at all...They don't even have a damn clue where the machines physically sit.

Allowing a junior developer fresh out of college to log into production with privilege that makes even a minor change is already in violation of industry protocols. I wouldn't blame the guy too hard. He's fresh out of college remember? The first person to blame is operations management who doesn't have a clear separation of duties protocol. Even if ops management allows him to log in, the next person to blame is the lead datacenter engineer who allows his account to issue intrusive commands.

If company leadership asks me to handle this fall out... I will fire those two guys first. Then the Jr. developer will get only a warning but that would be his only warning.

Some Company Information Please

[mykepredko]

I'm surprised that the firm has not been named - while I would think that any company that had this happen to them would want to keep this confidential, I would think that somebody would talk about it separately. I suspect that the "company" is some podunk startup in which the CTO is the CEO, CFO, head of development and probably the HR head and they've just hired a developer without thinking about access restrictions (or verifying that backups are actually happening).

Some more information would help clarify these questions and maybe better explain how such a situation could happen.

Re:Some Company Information Please

[ryen]

The author in the reddit thread (in a buried comment) describes it as a 100+ company with at least 40 devs. Pretty sad

Re:Why Was He Mucking With It In The First Place?

[lucm]

Having production databases that can be reached from developers workstations is always a bad idea. They were lucky, in a way, that the developer deleted the data and didn't just alter it slightly.

Firewalls, people, they're not just there to block wordpress exploit scanners. Danger can come from insiders, and that includes hostile employees as well as clueless morons.

We need to have medals to give out.

[clovis]

I say if he succeeded in putting that company out of business, then he should get a medal for sacrificing himself to destroy the company.

My belief is when he saw on his first day, the badly written docs they handed him, with a printed (!) account/password having RW access, he instinctively threw himself on that grenade by destroying their production database. Only the most cowardly IT worker would have done otherwise.

Thank you, selfless IT worker from saving us from the horror of whatever product they were trying to produce.

Re:bullshit click bait ...

[lucm]

just like the "supposed" rm -rf / last year ...

Something similar happened at Columbia Internet a while ago...

http://ars.userfriendly.org/ca...

be happy

[ooloorie]

You don't want to work at a company where the backups don't work and where a new hire can accidentally delete all their data. Don't beg to stay, instead be happy that you found out quickly how incompetent that company actually is.

Re:be happy

[hyades1]

Wish I had a mod point for you, my friend. That is exactly, 100% correct. Rookies make mistakes...sometimes even stupid mistakes. It happens.

If a rookie can wreck a company this badly, it's hardcore proof that the problem is a long, long way up the food chain.

THAT is where heads should roll.

Re:Why?

[__aaclcg7560]

That's funny. When I started reading Slashdot back in 1999 or so, it was called a website. Someone probably slapped on the "social media" label a few years ago to get some VC money.

Re:bullshit click bait ...

[__aaclcg7560]

In an IT department a long, long time ago...

Re:Why Was He Mucking With It In The First Place?

[Kjella]

Having production databases that can be reached from developers workstations is always a bad idea.

Welcome to DevOps ;)

Back in the day....

[GerryGilmore]

...we were a small company making the transition from a proprietary (TI/990 - DX/10 OS) system to a Prime UNIX system and I was the guy A) learning UNIX and B) writing shell scripts to replicate basic DX/10 stuff. One of my first scripts was to delete a user. Beyond the basic password deleting stuff, I thought it would be cool to delete the user's home directory also. You know, delete user "gerry" and also delete user's home dir "/usr/gerry". Guess what my brilliant script writing failed to anticipate? Yep, someone entering an empty user name. So, the script - running under SUID perms proceeded to delete "/usr/*", including "/usr/data". Oops! Fortunately, my backup script was better written and the customer only lost a few hors' worth of work, but....Whew! That's also how we learn!

Re:Back in the day....

[rvw]

> You know, delete user "gerry" and also delete user's home dir "/usr/gerry".

Who puts a user home folder in /usr/? Isn't that bad practise in the first place? The usr directory is for system stuff.

Re:Why Was He Mucking With It In The First Place?

[Shadyman]

Having production databases that can be reached from developers workstations is always a bad idea. They were lucky, in a way, that the developer deleted the data and didn't just alter it slightly.

You'd also have to pray they did not alter it any further.

Fake. Made up story.

[McBooCZech]

full stop.

I did this too

[riverat1]

I can relate to this. I wiped out the production database for our ERP system when I was trying to create a copy of it. Fortunately I had good backups and was able to restore the DB with minimal loses but it took all day (this was back in 1997 and the computer wasn't particularly fast, a SparcServer 1000 with 2 CPUs and 500 MB of RAM). In my case I wasn't fired and retired from the job last year after 31 years.

Re: How is it reckless?

[lucm]

I don't fail at civics. You fail at understanding what "at the same time" means.

Here's a hint.

https://en.wikipedia.org/wiki/...

Re:Why Was He Mucking With It In The First Place?

[JaredOfEuropa]

Some people thrive in that kind of environment. Working under pressure on something that really matters to the company, and with fluid roles and a lack of procedures, you have a lot of freedom as well as an opportunity to make a meaningful and unique contribution to the product as well as to the way the company will do things in the future. I know I perform better in such an environment than in the corporate one, with entrenched interests, highly compartmentalised roles, stifling procedures and generally terrible middle management. Sure, there's downsides to each environment as well as advantages. Personally I found a nice middle ground by working on innovation in larger organisations.

Re: Old News...

[KGIII]

Retired boss, here. If a junior dev can push to prod AND can delete data, it isn't their fault. Okay... I am still going to try to salvage my hire and see if they fit in QA, or see if the debs will stop fucking with him. Seriously, not his fault. Nobody should be able to push to prod, withou someone signing off. Ask me how I fucking know. I'm not even a programmer. I just paid a lot of you and shut the hell up and listened.

So, if I am wrong then they were wrong. You don't push code to prod without someone signing off. The person signing off has ultimate authority. You sure as fuck don't let a junior do it, without oversight. Not now, not ever.

If this happened in my shop, some titles would have been changed.

Re: Old News...

[thegreatbob]

This is more like they had a spurious cable attached between some random door below deck, and some critical lever in the control room... The point people are trying to make is: this guy is being given the shaft as a direct result of the fallout from idiots putting privileged/critical information into widely accessible documentation.

Let me guess

[nospam007]

The company is named 'British Airways'?

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

I worked as a DevOp quite often recent years.
As a DevOps I never had access to prod or preprod.
You must have a wiered idea what a DevOp is actually doing.

Recent grad given full production access

[DukeLinux]

What even marginally competent IT manager would give an inexperienced person the ability to modify or delete production objects? I feel sorry for this guy, but his management is completely at fault here. We know the story. Nothing will happen to them and one or more will likely get promoted.

Re: Old News...

[KGIII]

I gotta stop trying to type on a tablet.

Anyhow, I'd not want to fire anyone, without more information at hand. Firing people has large costs associated with it.

How to describe this?

As stated, I am retired. I've been retired for over ten years. Back in my day, we trained people and paid for them to continue their education. Don't laugh, that really used to happen.

So, I'd probably try to salvage this as a teaching moment. I'd also be changing someone else to the lead position. If I were to fire anyone, it might be the person who enabled this to happen and probably wouldn't be the junior.

Our shop was kinda laid back. It is unlikely that I'd fire anyone, but someone is going to catch a whole ration of shit. If this is a habit, then someone is getting fired, but if this is a habit then it speaks to a larger problem.

I'd start with the junior dev and work my way up. I'd keep digging until I found the problem in the process. I'd then take time to listen and find out what we need to do to make sure it never happens again. There's gonna be a meeting. Hell, there are going to be several meetings. Chances are, this is going to have a bunch of little teaching moments.

Junior devs, autocorrect hates that word and thinks it should be debs, sure as hell shouldn't be able to push to production. Even if they could, those backups damned well better be timely and functional. The IT staff, I guess you call them ops today, were great at their jobs, thankfully. For reasons, we had a pretty robust backup methodology. I am still flummoxed when I hear stories about not backing up properly.

I haven't done this for a while, so it is time to do it again.

I used a computer because I had to. I actually kinda hated them, for the longest time. I am not a programmer, but I programmed because I had a task that needed to get done. I'm not an admin, but I did that job because it also needed to get done.

Then, I was able to hire capable people. I was able to hire competent developers and IT staff. It was those people who did much of the heavy lifting. I learned a lot from them. I learned what best practices were and why they were that way.

So, as I said, I've not taken the time to do so lately. Here's a tip of the hat, and a nod, to those folks in the admin side and in the developer side. Here's a tip of the hat to the database admins and to the jerks who secured my network. Here is a tip of the hat to the programmers and to the QA. Here's a tip of the hat to those who spent long hours beside me, enabling me and teaching me.

Re: Old News...

[KGIII]

I owned the company. We were also pretty relaxed. I'm not sure we'd have used those titles, but I'm pretty sure someone is getting demoted - and it's probably not going to be the junior developer. Hell, I might have taken Mr. Junior Dev out to eat, or given them a small bonus - for having discovered the glaring hole in our process that enabled them to do this and for demonstrating that our backups were worse than useless.

Fishy

[DarthVain]

Not only exactly what you said, but the fact that they have no backups (or they were not working) is also pretty scathing. I've worked with some experienced developers that know our systems for 15 years, and they never ever have any access to production! Never mind they should have DEV and TEST instances as well, probably logging depending on what they are using...

Makes me wonder if their IT was that inept, or if there was some serious financial or other more fundamental technical issues that they needed to go away, and this guy is simply the scapegoat... LIke the CTO was embezzling from the system and needed a reason for all evidence to disappear. Or someone senior with real access screwed up very badly, and again this guy is getting the blame. Seems pretty unreasonable and unrealistic that something like this could even happen. Anyway it all sounds very fishy to me, I'd bet there is something else going on.