Developer Accidentally Deletes Production Database On Their First Day On The Job (qz.com)

An anonymous reader quotes Quartz: "How screwed am I?" asked a recent user on Reddit, before sharing a mortifying story. On the first day as a junior software developer at a first salaried job out of college, his or her copy-and-paste error inadvertently erased all data from the company's production database. Posting under the heartbreaking handle cscareerthrowaway567, the user wrote, "The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that I 'completely fucked everything up.'"
The company's backups weren't working, according to the post, so the company is in big trouble now. Though Qz adds that "the court of public opinion is on the new guy's side. In a poll on the tech site the Register, less than 1% of 5,400 respondents thought the new developer should be fired. Forty-five percent thought the CTO should go."

Comments

How the fuck

[Gojira Shipi-Taro]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding. The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Re:How the fuck

[Anonymous Coward]

The entire CXX staff level should be let go. Why a person fresh off the street had permission to even make a mistake of such magnitude is beyond me.

Re:How the fuck

[Anonymous Coward]

Additionally if the backups aren't working, that's something that could have saved the company - if the CTO had made sure that they worked properly.

I'd probably think twice before hiring the new guy as well over this, although at the same time I'd probably limit access to just what he needed to do his job. I'd also make sure I have proper backups.

But I'd avoid the CTO at all costs, at least as a CTO. (OR C- anything if this is how he operates.)

Re:How the fuck

[Anonymous Coward]

easy. it is entirely FICTION. it's reddit. this same shit's been done there before, and it shall be done there again.

congratulations, you've been bamboozled, along with most everybody else.

Re:How the fuck

[Anonymous Coward]

It's in the article. He wasn't directly given full access. The company's getting started guide happened to use production login information in the examples.

Pretty dysfunctional, yes.

Story may be real or fictional.

Re:How the fuck

[blindseer]

How the fuck does a new hire have that kind of access?

Having worked at some small businesses before it seems to me to be pretty common. The article said the business had a couple hundred people at most and 40+ developers. Quite likely the people there were there for a long time and they hired a handful of people since they now had a need for some help and finally some money to pay them. This is how things were done for years and not remembering what can happen if a newbie fucks up once in a while they thought nothing about handing over the documentation they used themselves, and never really having to test the documentation against a newbie before they didn't consider the consequences of what might happen if the doc was followed to the letter and/or with a minor mistype like inputting the production servers instead of the developers servers.

The lack of a proper backup is inexcusable. I learned that in college after having lost the only copy to a semester project a week before the end of the semester. After that anything I deemed "important" I kept a backup where it was not likely to be deleted accidentally.

I worked for a larger organization doing some development and I was never given access to the production servers. Once I could prove that I had something working I would hand my files over to a manager and he'd copy the files over himself. This was always a bit of a tense moment because, due to budget constraints for the project, I was using MacOS/Apache/MySQL/PHP for testing while the production servers were Windows/IIS/MSSQL/PHP. It seems everything was close enough because I didn't hear anyone complain about broken code. If something didn't work then it would have been very time consuming to roll back and also very difficult to debug.

This also makes me think a bit on why developers, even newbies, would have access to the production servers. If something broke between the transition between the development system and production then they'd want it fixed right away. This would be easier to do if the developers had access to the production servers. This would be avoided if the production and development systems were identical but like in that project I mentioned it might be that budget constraints prevented them from doing so.

Re:How the fuck

[davester666]

HR is responsible for what? Hiring someone who makes a mistake? Geez, might as well not have a HR department at all...

But the CTO and other people in charge of IT should be fired or demoted for failing to handle the most basic issue, how to handle the production data being lost. This isn't the only way, perhaps a significant electrical spike takes the data out, or a hacker breaks in and deletes or corrupts it all.

Once you actually have a "production" setup, you MUST have a backup (well, backups over time) that must be periodically checked and tested. Even just having a very expensive backup setup, but never testing that you can actually get back everything you need from it, is worthless.

Re:How the fuck

[lucm]

The entire CXX staff level should be let go.

First, "CXX" level is not "staff". Second, firing the entire senior management team because of this incident would be completely reckless. I understand the outrage but that's not how companies work in real life. The last thing a company in that situation needs is more instability.

The proper way to address this is to stabilize the situation, then make sure the problem cannot occur again. And this typically doesn't simply mean firing people, because odds are that there are cultural or organizational factors that made this situation possible (crazy deadlines, shoestring budgets, etc.), and those would probably lead the replacement to making the same kind of mistakes down the road.

What is needed is new processes and controls. You start with a simple governance framework (like COBIT maybe) where each part of the IT ecosystem is linked to a specific business leader, then you let each of those leaders make sure that their area of responsibility is well managed from a risk perspective. That's how you make the company more resilient, not by firing people who maybe were not empowered to make the right decisions in the first place.

Re:How the fuck

[ma1wrbu5tr]

The trainer, the CTO, the Managed Services Director, and whoever let that doc with the PROD database URL out into the training materials. Those are the people I would hold responsible.

Re:How the fuck

[sexconker]

Additionally if the backups aren't working, that's something that could have saved the company - if the CTO had made sure that they worked properly.

I'd probably think twice before hiring the new guy as well over this, although at the same time I'd probably limit access to just what he needed to do his job. I'd also make sure I have proper backups.

But I'd avoid the CTO at all costs, at least as a CTO. (OR C- anything if this is how he operates.)

I reconfigured our backups recently. It's a huge pain in the ass because I had to wait for the end of the month, move everything to other storage, recreate new jobs from scratch, wait for them to run, copy to a second location, and perform automated validation checks in both locations, then do it all again (making sure version chaining worked), then actually go in and open up every single new backup, in both locations, to ensure they all worked, the correct passwords were used, etc.

But I know they fucking work.

Re:How the fuck

[dbIII]

What is needed is new processes and controls

Yes, but I think what people are getting at here is that current policies appear to be so fucked up that the team who implemented them may be unlikely to implement new processes and controls that are any better than they have already done.
I've seen that sort of stuff in a few places, (notably government owned corporations - worst of both worlds) where management have been chosen for reasons other than ability.

As shown by this incident that sort of mismanagement is often self-correcting, but it's a shame that so many others have to go down with the ship when the place goes under.

Re: How the fuck

[Anonymous Coward]

So he was given a getting started guide that contained full access info, maybe without even stating that it was prod and risky to play with. It's even worse than being explicitely given full access. For all he knew, he was in a playground (that's what most companies use to train their new hire)

Re:How the fuck

[dbIII]

The article said the business had a couple hundred people at most and 40+ developers.

Not exactly a small business deluded sig guy.

Re: How the fuck

[Tomahawk]

Agreed. Has they said it was a month in then maybe it could be believed. But onlyâ day one you're only getting down around, here's your PC, here's your password, Joe will show you around the product later and explain what we do, the toilets are that way...

On day one, there'd be no production access. Nor access to source code. And if he did have that of the was access, and changes would be scrutinised.
So, yeah, it's a fake.

An interesting hypothetical scenario, though -- if it did happen, given the info available, then who's to blame?

The new dev? No: first day, fresh out of college, mistakes should be expected. He shouldn't have had that level of access, and his immediate superiors should have been keeping a close eye.

His immediate superiors? Possibly. How did they let someone new make such a big error? Why did they allow him to do anything at all?

The DBAs? No working backups on a production database? No transaction logs that could be rolled back? No DR solution in place? Basic stuff here that were all missed. So definitely some blame sits her.

CTO? Hard to say. Certainly policies should be in place to ensure this shouldn't happen, so why did it under their watch? Were the staff too overworked that they didn't have time to get the basics right?

To properly sign any blame, more information is needed. We don't have all the facts. To many questions remain unanswered.

There is a general human flaw that this story highlights, though, which is that the majority will sign blame based on too few facts. This comes up time and time again, both here and elsewhere. Take, for example, "Making A Murderer" -- how many petitioned to have Stephen released based on 10 hours of a TV show that really only showed one side of the story? There were more than 200 hours' of testimony to try to show all of the available facts for the jury to make a decision.

This is a similar case -- one reddit post describing things from the point of view of developer who was on his first day ever working. There's tonnes of missing information here.

Still, and interesting scenario to ponder as a thought experiment.

Re: How the fuck

[h33t l4x0r]

I can guarantee there was no DBA. A copy+paste error that can't be rolled back *or* restored from backup means that nobody is on the ball over there.

Re:How the fuck

[Calydor]

If it wasn't a pain in the ass it wouldn't be called work.

Re: How the fuck

[ColaMan]

I read the thread over on reddit and basically he turned up, was given his dev machine, and then given the instructions for creating the dev environment + db's & etc on his machine.

Then after a while he was like, "OK, let's see, step 17: Enter these commands to create your dev DB."

So he copied and pasted the commands from the document, like he'd been doing for the last 16 steps.

Unfortunately, the commands had the production db's connection in them. And they also had queries like, "drop database mainDB; create database mainDB; create table etc etc"

Whoopsie!

Re:How the fuck

[tlhIngan]

Well, depending on the size of the company, it could be possible. Maybe not if you're a 1000 person enterprise, but if you're a 10-100 person SME, it's definitely possible, especially a startup.

However, the lack of backups is more damning - it means the entire company is one mistake away from losing it all. I don't care if it's a new guy - it could be the rockstar developer making a typo and deleting the entire database. It could be anyone.

Even though I work for a company of less than 100 people, no one other than the IT guy has full permissions. Even me as the backup IT guy I don't have full permissions. I've been offered, and I always refuse it - if I don't see a need to have it, I don't want it. (It basically evolved down to what I should do - and the vast majority of what is needed involves users, so access to the domain controller isn't required (about the only thing I'd need it was for resetting locked out accounts, but it resets after 5 minutes anyways, so my response is "go for a coffee and it'll be fixed").

Even then, I'm pretty sure the only access he has to delete all the files is via the file server console - his user may have read only to everything, and read-write to what he needs, but not full read-write for obvious reasons.

But we have backups, too, and because it's all shared files, we have volume shadow copies enabled, so even if you do delete it all, there's a snapshot of it from a few hours back. How many asses this feature has saved, who knows? It's saved mine several times (not that I'd tell anyone :) ).

And that's why everyone believes the CTO should go - it's not hard to protect, and have backups and ensure they all work. If a newbie could delete the production database, so can anyone else. Perhaps the rockstar developer has an oops with a mouse and deletes it?

Accidents happen, and there is no excuse for not having basic, simple insurance policies (permissions, backups, etc) in place and operational. I say the shareholders should hold the management team to account for this - something so basic should not be possible.

Re:How the fuck

[William Robinson]

What is needed is new processes and controls. You start with a simple governance framework (like COBIT maybe) where each part of the IT ecosystem is linked to a specific business leader, then you let each of those leaders make sure that their area of responsibility is well managed from a risk perspective. That's how you make the company more resilient, not by firing people who maybe were not empowered to make the right decisions in the first place.

Well, that is what they have been paid for so long. If processes and controls are not in place, to avoid this kind of disaster, the leaders have clearly failed their duties. I might agree that entire staff need not be fired, but it is for sure that blame goes to somebody higher up sitting on his ass, instead of the poor kid.

Re:How the fuck

[geekmux]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding.

This has little to do with noob status as an employee, or even technical experience. The real question is why the fuck a developer has access to the Production system. We call the Non-Production environment Development for a fucking reason.

The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

The CTO should get the shitcan for not ensuring backups were working, as well as not implementing proper security policy that prevents developers from fucking around in the Production system without assistance and a documented approval process.

Regarding HR, they're fucking clueless at this technical level, so don't even really know what could be pinned on them. Their involvement started with processing payroll and ended with him signing the HR handbook, which probably doesn't have jack shit to address this particular fuck-up. That would be IT policy and procedure.

Re:How the fuck

[Z00L00K]

Everyone screws up now and then, most often small, sometimes big. There's sometimes a small screw-up where you accidentally reboot the wrong server. Once is just an oops. It's when people screw up frequently and tries to get away with it you should bury them somewhere safe or remove them. If they admit their mistake then it's better to work on salvaging the pieces and patch together the remains or recover from a backup.

Companies that don't have backups that they test - they are toast as it doesn't even have to be a mistake that causes the need for a backup, it can as well be a hardware failure or thunderstorm.

Good Employees Make Mistakes. Great Leaders Allow Them To. From the article: ...As John Wooden once said, “If you’re not making mistakes, then you’re not doing anything.”...

Re:How the fuck

[Interfacer]

When I started as the new systems engineer in our company, I was hired to become the guy in charge of the process control infrastructure running the entire pharma plant. Keep in mind we were already running operational for tens of millions per year. I was taking over from the consultants who built the place (during a transition that took a year or two iirc). First real day on the job, he created my user account, granted me enterprise domain admin and full application / db admin rights and spoke the immortal words: You're a full admin now. Don't break anything.

Re:How the fuck

[tburkhol]

The proper way to address this is to stabilize the situation, then make sure the problem cannot occur again.

You're expecting rational behavior from a business? Good luck. /s

I'd add that the size of the business probably matters a lot here. If this happened in a shop of 20 people with 3 computer guys, one of whom is "CTO," it's a very different situation than a billion dollar, multi-site company. It's probably smart to send the new guy home in the immediate aftermath - he's not going to be helpful in the recovery and emotions will be very high. After that, bring him back in, sort out what happened, and how to prevent similar failures of process. Dude has the cost of a fuck-up burned into his brain, and he's going to be a lot more careful and deliberate in the future.

Now, if he does it a second time...

Re: How the fuck

[JWW]

How the heck did the copy/paste text include valid credentials to even access the production database?!?

Re: How the fuck

[war4peace]

I'll tell you how. (names below are fictional)
Netadmin or Sysadmin team has no content writer, so Prakash is told to create an induction manual for new hires. Prakash hates this assignment (understandably, he's a net admin, not a fucking content writer) so he does the bare minimum: takes screenshots of prod environments and enters credentials for that generic admin account everyone was using (because fuck processes) as example.
New hire comes, proceeds going through the document, and with his lacking attention to details (or because he's overzealous, or been told to beat the best time for new hires of 30 minutes 15 seconds) doesn't pay attention to the step saying "enter YOUR given connection details" and copy/pastes the ones shown as example, which incidentally are absolutely valid and point out to the PROD DB.

Disaster happens.

Moral of the story?

1. Get a proper content writer to create documentation: responsibility falls to management to ask for it and senior management to approve it. Blame falls on whoever didn't ask or didn't approve it.
2. Review the documentation: responsibility falls to line manager Prakash is under. Blame that line manager and hang him (figuratively) from a tall sturdy branch.
3. Publish the document as a controlled document in a knowledge management environment. If such an environment doesn't exist, blame everyone who didn't ask for it or approve it.

The CTO is hardly to blame, it's not his business to handle such processes, that should fall under a line manager or whichever dedicated person was supposed to handle it.

I personally wouldn't have kept a netadmin/sysadmin who can't follow basic instructions or a manager who didn't review the training document. Everyone else is off the hook because they were either not aware of the risk or did what they could with a task that wasn't in their job description.

Re: How the fuck

[Cesare Ferrari]

Agreed, i've worked places where stuff like this is possible.

Remember, if the place was a windows shop, and the DB is, say, SQL Server, then the permissions to do dangerous stuff may just be group membership of the user account in AD, so you could get production read/write access without typing a password, just by connecting to a db server.

I've worked on linux systems where the production access is locked to a given user, but it was common to sudo to this user for certain tasks. If the user in question isn't called 'prod' or something like that, but say, named after the product, it's not clear what permissions, and in fact, damage you can do.

So yes, it's possible, and i've been places where this can happen. Actually, sitting at my desk at work, I can screw the system in many ways without typing a password. I can delete binaries, or promote something broken to production, or screw up network configuration, or well, just endless ways really. The upside is that I can fix stuff quickly when it goes wrong, and each company needs to evaluate the risk vs reward tradeoff to determine what is right for them. When the stuff I produce goes wrong, only the company is harmed, and they have no external users who would be aware. If I were working on some web facing system with millions of users, clearly a different approach is appropriate!

Re:How the fuck

[drinkypoo]

The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Yep. When I read this:

"The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss.

My first reaction was yes, yes they will, and the CTO is the one responsible for that kind of stuff, and the one who will get sued if anyone does. And if his boss has any sense, he'll be sacked, too. If you haven't verified that your backups are working, you are worse than useless. You are a cancer that must be cut out before everything goes wrong, as it has here. That company is probably going to die now, and it's going to die the death of the shit CTO.

Re: How the fuck

[rgbatduke]

I mostly agree with your remarks, but I think there is a question of scale. I've done several startups, and in a startup, one person is often the CTO (that would have been me, twice), the primary developer of the core software (ditto), and the system manager for the entire network of computers owned by the company, which on occasion have been pretty much my cluster of computers plus eventually a "company owned" server or two when we had enough capital or cash flow to afford them.

From this level of "my basement plus your basement" startup, there are a set of scaling steps that lead through VC incubators to getting actual VC to hiring a skeleton staff (cheap, quite possibly fresh out of school and paid in part with options or the prospect of options) to making money but not as fast as you burn it to making money (one hopes) faster than one burns it and moving on to fame and fortune and early retirement.

At intermediate steps in this process, IT is not the polished gem that it might be for a fully developed and capitalized and profitable company. Backups might have been set up by the original founders (and done correctly) but all it takes is one hire that necessarily is given serious responsibility with little oversight in a tiny startup who doesn't completely understand backup to make a small change that fucks it up without even realizing it. I've been around a long time, and trust me, it happens, and if you are LUCKY you discover it when some tiny unimportant file is overwritten and you try to restore it and can't.

If this was a startup in the stage just past profitability -- small but important database of actual customers and/or their data, team of maybe three or four IT people still wearing many hats, so the database person was also the systems manager and the primary software developer was also the web manager and the original CTO/developer/general factotum was distracted by the demands of sales and corporate politics with a board made up partly of the VC people who want rapid growth to a liquidity event and so on, this scenario isn't that unlikely. DB person says I'm going nuts adding all these new workstations to sales desks, CTO says let's hire a DB person, CEO/COO says we can't afford a $150-200K position right now (duh!) so CTO says we'lll hire some kid straight from college with DB chops, kid comes in, they show kid the DB and sit him or her down, he/she tries a few tentative SQL commands but has no experience with whatever actual DB they are using and has credentials that are more talk than substance, tries to make a copy of DB to play with without breaking original and fucks it up. In the meantime, they have been backing up to a RAID, and the RAID started throwing errors but the same DB person who left the kid to play and orient themselves had JUST STARTED the rebuild or was trying to fix the problem and rerun the failed backup when...

Life is a comedy of errors like that more often than one might think. Most times they are non-fatal, but every now and then the ENT surgeon slashes the carotid the first day of his/hir first surgery and the patient dies on the table, or the kid working on steel 200 feet above the ground loses their balance on their fist day trying to emulate people who have been doing it for years and walk a narrow beam over nothingness. Sometimes people can get back up on the horse that threw them and sometimes they end up living under an overpass or dead at their own hand.

Humans are highly error prone information processing systems. We deliberately design systems that are critical -- as much as we can -- to have multiple levels of mutual auditing to catch and prevent errors before they occur, but it simply isn't possible to idiot-proof every process, and accidents can happen even to those who are not idiots. Few are the people who have root privileges who have NEVER EVER entered rm *.junk in some directory, but accidentally entered an extra space before the ".junk" (blush, been there, done that, done WORSE than that). If you are the only systems man

Re: How the fuck

[TWX]

No person in the company should be able to do that without peer review. Backups should work and be tested. Clearly he's not the one to blame !

I really could see him being fired from the job. If I were a mechanic and I accidentally broke some important active piece of machinery on my first day I would not find it unreasonable to be fired.

Now, that out of the way, there's a whole host of other people that built the conditions that allowed this to happen that should also be fired. That CTO that suggested the Legal department should be involved is among them, and HR should take exception for how he threatened a new-hire for his own mistake.

Re: How the fuck

[mixed_signal]

"The Staff" or "staff level" certainly does refer to the C level in many tech companies.

Re:How the fuck

[Ol Olsoc]

What is needed is new processes and controls.

What is needed is a backup system that actually works, and is used.

I never trusted our official backup system, having seen it not work on several occasions. So I installed one of my own for the group. Sure enough, around a year later, a group member called in a total panic - she'd written a script that was supposed to perform a find then print the find results. What it managed to do was delete the whole database.

Calls me in a tearful freakout - the IT folks backup didn't work.

Mine did. There were a few things here that needed addressed. She should have been working on a copy of the database, not the main one. Shit does happen, and if she accidentally deletes a copy due to a derpy moment, its no problem at all. Fix the script, and work on a new copy.

Second is don't ever trust a backup that you can't personally verify will work.

Third is don't be afraid to have backups for backups for backups. People called me Mister OCD or Monk (referring to the television show about a detective with serious OCD issues) but after that the little jabs mysteriously went away.

Re: How the fuck

[mixed_signal]

Sorry, ensuring business critical data is correctly backed up is obviously critical. No excuses pass muster here, as it's a well known disaster waiting to happen from any number of causes.

Re: How the fuck

[mixed_signal]

Exactly. Backups are business critical.

Re: How the fuck

[Defakto]

As a manager and supervisor I wouldn't put you out on the floor without a senior to ensure you could handle and to QA the work. The first 6-12 months are hour learning and trust building phase. I expect mistakes during this time and I expect your peers to work with you on new projects to ensure those mistakes don't happen .

Re:How the fuck

[tommeke100]

It just depends how big a "production database" is. We have a relational database that is maybe 100 GB big. Daily Dumps are a bit less than 18 GB and like 4 compressed. It takes maybe 15 minutes to backup which can be done at night (and even during operations really, it's not such an heavy load on the system anyway). Since the data is used often to implement new functionality or check on things, we deploy the data on test servers often so we know it works. The backups are also copied to other locations. If something drastic happens, we don't lose more than a day of work.
Of course, I can imagine big transaction systems where you only have a couple months worth of data and you need some clever process to mount and unmount the older data to make space for the new. The older data must than be kept somewhere safe yet accessible obviously.
Indeed, if the junior developer managed to wipe out the whole DB, and no fail-safe mechanism is in place, it's not really his fault.

Re:How the fuck

[Kjella]

This also makes me think a bit on why developers, even newbies, would have access to the production servers. If something broke between the transition between the development system and production then they'd want it fixed right away. This would be easier to do if the developers had access to the production servers.

Yes, 99.9% of the time that's what happens. Then there's the 0.1% where our most senior "I know what I'm doing" developer was just updating a status (not through a stored procedure, because why add another layer of indirection) and forgot the WHERE clause. To the global log table that every process is required to use. Or the house consultant (he had more system experience than better skills than most our hires so wasn't that) that was going to scratch a database on the test server and rebuild it from a script, who didn't check what server the query window was connected to. Of course these are totally hypothetical examples that never happened *cough*.

Re: How the fuck

[Anne Thwacks]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Hell, even on a one man business, development should not be done on the same machine as production activities.

Even my home machines have off-site table backups - and yes I have tested recovery works - cross platform - on different hardware and OS.

if your data is worth money, you should be prepared to spend money to protect it. If its not worth anything, then just delete it yourself and save the disk space.

Disclaimer: I do not have an MBA.

Re: How the fuck

[tommeke100]

I reckoned this was the only possible scenario. They gave him some code to look at, with DB creation SQL statements and such. And they just had the Production DB in the connection string and user/pwd.
That's basically a huge mistake of the senior people who need to train him. That's the person you need to fire. I've had to train many devs to work with our databases, and I've never given them code to access the production DB ever. Only after maybe months of working on test servers, they get to run their stuff in prodution, and that's after it's been vetted by different people.

Re: How the fuck

[Anne Thwacks]

If you hire someone that seems like they know what their doing they should know what their doing

Are you trying to justify voting Trump?

Re:How the fuck

[JustAnotherOldGuy]

How the fuck does a new hire have that kind of access?

My thought exactly, which is why I'm a little skeptical of this story. Who hands out this level of access on Day 1?

If this super-newbie was given this level of access on his first day, the CTO and HR and whoever was responsible for making backups should all be fired like a cannon.

Re: How the fuck

[JustAnotherOldGuy]

I did a contract job years ago where they had me work on a mission-critical production database. In MS Access.

Well there's your problem. FFS, who does that??

Re:How the fuck

[slappynipsy]

Testing the backups is a huge pain in the ass but in the immortal words of Don Draper, that's what the money's for!

Re: How the fuck

[subanark]

I'm a developer for Azure. The service I develop doesn't have a DBA or any test team. Could I screw things up? Yes, quite easily, however since the DBs are on XStore, we can just pick a date and time within the last week and snap the DB back to that state. I don't run a big enough or impactful enough service to warrant this level of support.

Re: How the fuck

[whoever57]

The CTO is hardly to blame, it's not his business to handle such processes

But it is his job to ensure appropriate security and backups for the production database.

Re: How the fuck

[lgw]

If you hire someone to clean your house and they break an expensive vase on their first day, do you say they should have had a more senior cleaner shadow them for a while?

You know that's how it usually works, right?

Re: How the fuck

[HornWumpus]

No _competent_ DBA is not the same as no DBA...

Re: How the fuck

[HornWumpus]

They've moved onto MySQL...

Re:How the fuck

[140Mandak262Jamuna]

First, "CXX" level is not "staff"

This is military terminology. The "staff" here is the General Staff, officers who form the core of administration of the armed forces. Officers who has achieved "staff" rank get to wear some special markings on their uniforms, like a red band around their caps, or red velvet under their insignia or red markings under the stars on their collar etc.

It is like the word "secretary". The secretary of the board is a senior administrative officer. These terms secretary, staff etc often refer to the low ranking as well as high ranking officers.

Re: How the fuck

[Anonymous Coward]

It was legit.

world came round from other people who worked for that company. who no longer have work.

the company fucked up completely by having the actual real admin/pass listed on the training sheet the noob was given. 99.9999% company fault.

Re: How the fuck

[140Mandak262Jamuna]

Netadmin or Sysadmin team has no content writer, so Prakash is told to create an induction manual for new hires.

What the hell? It is not some user manual for an application sold to customers. It is an internal document of procedures. Admins routinely write such procedure documents every day of the week. Often for their own references, so that they dont miss important tasks. Many admin jobs are done one a week, once a month, once a quarter, or on triggers like off boarding, on boarding, going on leave, coming back from vacation etc. They should document what is done. Often admins will history > this_session ; emacs this_session and add notes to the commands and make that the document.

Re:How the fuck

[Just Some Guy]

The last thing a company in that situation needs is more instability.

It sounds to me like they've already maximized instability, and any change whatsoever could only lessen it.

Re:How the fuck

[bigtiny]

I'm always amazed when something like this happens, a lot of people's first reaction is 'who gets fired'? It's really not a productive attitude to take in a case like this.
I find it problematic that
- companies take an agile 'quick turnaround' approach to development without seeming to understand the risks, until they get bitten. This is an example.
- seems that whoever's managing the dev team should have had a break-in period/mentoring system in place to make sure new hires (especially newbies straight out of college) know their way around before pushing production code.
- whoever's managing the place is REALLY on the hook if the data cannot be recovered! If a manager/company is not willing to provide the personnel and tools necessary to insure backup and restore integrity, then they have only themselves to blame.

Firing this kid, in this situation is stupid. In a way, he did them a favor....now they know what they need to fix.

Re:How the fuck

[theNetImp]

you apparently didn't read the article, where it says the credentials were in the onboarding documentation....

Re: How the fuck

[JustAnotherOldGuy]

They've moved onto MySQL...

And that would be a good move, if they did in fact do that. I know people deride MySQL, but it's light-years ahead of MS Access.

You might not want to run your bank on MySQL, but it's a perfectly suitable database for 98% of the kind of stuff that most companies do on a day-to-day basis. Unless you're sending people to the Moon or doing financial services, MySQL is fine. Tesla, Netflix, Youtube, Spotify, Facebook, Twitter, and the NASA Jet Propulsion Lab all use MySQL on a daily basis and they seem to be doing just fine.

If you don't like MySQL, use Postgresql or SQL Server or IBM DB2. Whatever floats your boat and gets the job done.

If you want to keep your recipe collection in MS Access, great, but for anything more than that, ewwwwwww.

Re: How the fuck

[HornWumpus]

MySQL is in a league with DBase 3+. Routine reindex.

Re: How the fuck

[lgw]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Welcome to DevOps. It's the fad sweeping the industry. Yes, it's exactly as stupid as you think it is. It's so obviously wrong, so mind-bogglingly stupid that only a manager with an MBA could possibly have thought it up. So of course it's the latest fad.

Re: How the fuck

[lgw]

That's the CIO, BTW. CTOs have nothing to do with internal IT operations, their job is to guid the tech that the company creates, not uses. Well, mostly it's to tell large sales prospects that the company is working on really cool stuff. The CIO runs IT.

Re: How the fuck

[lgw]

It's a real step up from MS Excel, which is shockingly common. The huge tech company I work for keeps all it's pricing stuff in freaking Excel sheets: I blame the mandatory lobotomy that accompanies an MBA.

Re: How the fuck

[Entrope]

The "script" command is awesome for that kind of thing, as it captures intra-program I/O as well as command lines.

Re: How the fuck

[war4peace]

The fact that the behavior you're presenting seems normal to you scares me, because it shows how widely accepted this has become.
While I agree that the admin should start writing the draft, their job would end there. Technical people rarely can produce proper documentation, and whether the documentation is internal or customer-facing is irrelevant. Standards should be the same. the mindset you're presenting is similar to that of people writing in bad English online because "fuck it, it's not important, I can write good when I have to".

Writing note for tasks != procedures. Also procedures != processes. Also induction manuals != processes or procedures (albeit they often overlap, still they are not identical).

At the very least, you need a dedicated person (or team, depending on how big the organization is) to verify, review and vet all documentation (except for private notes, comments and such) and store it in a controlled, unique repository. This could be as simple as a collection of printed brochures for small companies, or a wiki, or a more complex Knowledge Management System. The person or team assigned to that bears the ultimate responsibility. The CTO (of a larger organization, of course) has better things to do than review all documentation. Responsibility is then compartmentalized.

I've seen my fair share of horrible documentation over the years. often I have to go and learn what the processes inside a certain group or team are built, in order to understand the data generators (I develop analyses and reports) as well as why that data is generated, how is it generated and what does it mean from a business perspective. It greatly helps me provide correct, helpful and thorough analyses and reports. The problem is that the documentation is most times a mess, usually a collection of disparate word and excel files of various versions circulating through e-mail back and forth like feces in a sewer. Open one and you'll be hit by a jumble of acronyms I'm sure the tech people understand, but I have no idea what they mean.

Yes, from your point of view, that little information bubble you're in, it all makes perfect sense and you're used to it. Introduce a stranger to it and they'll stare at you blankly while you're thinking "how stupid is this guy?"

TL;DR version: Isolated communities develop their own unintelligible language, while isolated dev teams develop their own unintelligible documentation.

Re:How the fuck

[modmans2ndcoming]

Depending on how large of an enterprise you are dealing with, personally verifying the backup of the database is functioning might not be possible.

Re: How the fuck

[Brockmire]

You either work for the city or you're in a union.

Re: How the fuck

[lgw]

How do you think people get into the job? Some minor skill is required, and must be demonstrated. Usually one of the older women will bring someone new along with her for a little while, until she seems to get the hang of it.

Re: How the fuck

[lgw]

Facebook, Google, Microsoft, and Amazon all do DevOps to some extent (Amazon especially). So everyone else copies them. Damn all management fads.

Re:How the fuck

[lucm]

Maybe. But usually when talking about the enteprise world, C-level executives are called corporate officers, and everyone else is the staff.

Re:How the fuck

[lucm]

It sounds to me like they've already maximized instability, and any change whatsoever could only lessen it.

And you say that based on 2 paragraphs posted by an anonymous junior developer who deleted a production database without knowing it?

Re: How the fuck

[minstrelmike]

Most mechanics have "backups" of all the tools junior mechanics will use. In a normal environment, deleting the production database isn't really that big a deal. About the same as ordering the wrong part for a repair or having the electricity shut down.

Backups are supposed to work. Shit happens. All the time.

The CTO should have had the junior devs working off backups so 1) nothing goes wrong and 2) even better, you know those backups work. Load a new one each day.

Re: How the fuck

[minstrelmike]

I remember the very first SQL stuff in the Oracle 4 manuals. First instruction was how to list out tables. The next one was how to drop them. So the first thing you do is drop the training tables you're supposed to use in the next step of the tutorial.

Whoopsie. That wasn't user error.

Re:How the fuck

[minstrelmike]

I'm one of those senior developers. I only delete all the data about once or twice a decade now. Live and learn.

Re: How the fuck

[lgw]

They may have started the fad, but they at least turn important stuff over to pure Ops people. Unlike Amazon, where a dev makes a typo in a command line and takes all of S3 down for a day.

Re: How the fuck

[lgw]

There's a real difference between DevOps and "small company, so I wear both hats". The latter is a forced error, the former insanity.

Re:How the fuck

[martinfb]

Also, the IT staff that failed to implement and verify backups needs to go.
PLEASE be sure these idiots are put on the national "IT offender" list!

Re:How the fuck

[sh00z]

Don't be an arsehole with the "CXX" not being "Staff". If you work for the company, you are simply staff / employees. You know exactly what he meant.

Quite possibly not. I work for a medium-tech company that gives all sorts of authority to the IT and Procurement organizations. Unfortunately, all of the accountability is assigned to the Science and Engineering teams. Not having to be responsible for their own f@ck-ups has created a culture where these groups work without oversight, and project managers have to pad schedules about 40% for the inevitable "you wanted a cable--what's wrong with this perfectly good rope I ordered?" moments.

Re:How the fuck

[mongothesecond]

Someone on the management team is accountable to the board for disaster recovery and other functions including data backup. How is not safeguarding corporate assets and intellectual property not a management level responsibility?

Re: How the fuck

[thesandtiger]

I worked for a start up that handed all new developers the keys to the kingdom on day 1, regardless of experience level.

On day 1 you were expected to deploy code changes to production (minor stuff - usually adding yourself to the about our team page) AND republish the production DB w/backup.

It was every bit as stupid as it sounded, and yes, in the very short time I was there before I got the hell out of dodge, at least 2 or 3 times the site was taken down completely by a day 1 newbie fucking this process up. At least they had backups, though.

Re:How the fuck

[lucm]

Only the CEO is accountable to the board. The CTO is accountable to the CEO. The VP-IT (or whatever) is accountable to the CTO. And down it goes until it gets to the idiot who couldn't follow instructions. Where do you draw the line? The executive who derives priorities based on the overall business strategy? The manager who is supposed to enforce policies? Risk management people who didn't foresee or implement safeguards? Team lead who didn't train or supervise the newb properly?

This is a witch hunt and it won't bring back the lost data. The blame game is never the best way to move forward. Processes and controls, that's how you tame the beast.

Re: How the fuck

[easyTree]

No person in the company should be able to do that without peer review. Backups should work and be tested. Clearly he's not the one to blame !

Perhaps the new-hire was also be responsible for putting their backup strategy in place? :D

Re: How the fuck

[cthulhu11]

Lemme guess -- you consider yourself a developer and look down your nose at "ops"?

What's stupid is segregating "A&E" and "ops". The former fiddle while Rome burns, like they're working on a thesis, pisisng away person-months on deadends, LLD' s that defy the laws of physics, and wheel reinvention. The latter are left to get the service to actually run, so they do their own dev in a vacuum.

What's also stupid is dedicated scrum masters who enforce dogma despite reality, counterproductive stuff like planning poker and droning on about how everyone on the team should be able to do any job.

What works is having a variety of complementary people in each team, providing perspectives and cross-pollination. May of us were doing this years before anyone invented the term DevOps.

Re:How the fuck

[NockPoint]

Where are my Mod points when I need them? CTO should be out the door. +5

"Their backups weren't working."

[Chas]

Okay, the guy fucked up ROYALLY.

It happens. And he SHOULD get in a bit of trouble for it. That's how you learn "don't do that". I don't think they deserve to lose their job though.

The CTO and all the people in charge of the backups need to be on the street YESTERDAY though. That the dev COULD do something like this is a major fuckup on their part. They simply didn't have their production system locked down properly.

The fact that their backup system was non-functional is double-plus unforgiveable. The dev is merely the highlight for their massive cluster-fuck of a setup.

Re:"Their backups weren't working."

[Just Some Guy]

Okay, the guy fucked up ROYALLY.

I don't think he did. I actually RTFA this time, and the guy was following the onboarding directions he was given. Where it went south was that he copied-and-pasted the wrong database credentials. He was supposed to use the username and password that a command had spit out, but he instead used the ones from the onboarding docs.

I'll pause for a moment to let that sink in.

Some jackass had put actual prod root creds in the onboarding docs, then gave them to a new graduate fresh on his first day of his first job, then walked away while he onboarded himself without supervision.

This poor kid did absolutely nothing wrong except misreading some instructions. The engineering team responsible for the chain of events that led to this colossal fuck are completely and wholly to blame.

Re:"Their backups weren't working."

[lucm]

The fact that their backup system was non-functional is double-plus unforgiveable.

In my experience, continuous SAN replication is often to blame for a poor backup strategy. It creates the illusion of security - yes, your DR site is synchronized with production within seconds or milliseconds, but guess what, mistakes are also replicated.

Replication -> floods, fire and similar disasters
Backups -> oops my bad

Both are needed.

Re:"Their backups weren't working."

[ooloorie]

The CTO and all the people in charge of the backups need to be on the street YESTERDAY though.

I think the company is hopelessly lost. It's not only the CTO that screwed up but the CEO who hired him. At that point, you run out of people to fire, and the company just goes out of business.

And he SHOULD get in a bit of trouble for it.

He should thank his lucky stars that he found out so quickly what a poorly run company had hired him.

Re:"Their backups weren't working."

[Chas]

He's a new guy. And the onus is on him for attention to detail.

However, he's a new guy. Fuckups are to be expected.

I've had fuckups at new jobs myself. This is how we LEARN.

And yes, in that environment he was set up to fail.

Re:"Their backups weren't working."

[Chas]

The entire company should be fired and cease to exist. You should also be fired, chazz.

Thanks! Now I can go home and catch up on my sleep! Later!

*Burnout noises*

Re:"Their backups weren't working."

[Just Some Guy]

You raise a lot of perfectly valid points that I'd expect the storyteller to have caught and reacted to. That is, if it weren't their first job out of school.

I'm senior, but I've taken jobs at places where I was given a few pages of setup instructions that could've been a shell script, and I had to work through the steps one at a time. I didn't know how their environments were arranged, or what systems called "cat-kicker" were supposed to do, or why on Earth their git repos were carved up like they were, but I slogged through and got stuff going. I have enough experience to fill in the missing pieces and tell when things don't feel right.

If you or I saw "Step 3: run sudo rm -rf /", we'd stop and ask someone. But if it was literally our first day at work ever, and we didn't have the experience to know that just because someone started before you doesn't mean they know what they're doing, it might not even register. You or I see "echo 'drop table users;' | psql --host prod", we're finding a coworker to see what's up. New guy sees "echo ... | psql --host rds-32494032" instead of "--host rds-1005828322", he might not think anything of it as he plows ahead. That no one looked over his shoulder to make sure he didn't actually use the dang ol' prod creds they gave him is entirely, 100% on them.

Re:"Their backups weren't working."

[Just Some Guy]

But his mistake was easy to make and should have resulted in an "access denied" error message. If you give a five year old a hammer and say "don't whack shit with this", and they whack shit with it, you're the one done goofed.

Re:"Their backups weren't working."

[Just Some Guy]

That's the modern version of "we have RAID backup!"

Re:"Their backups weren't working."

[Maxo-Texas]

A new guy may not be capable of distinguishing between details yet.

I've been hired before because a developer deleted a production library. My onboarding test consisted of asking me how I would perform the task. lol.

Re:"Their backups weren't working."

[rossz]

Yep. I have a hot standby for our production database server. Everything done on the primary is almost instantly duplicated to the standby. That would include fucking shit up, e.g. drop table foo; Which is why I make a weekly backup to disk and keep the entire WAL history until the next full backup. The backup and WALs are kept on a filer, which is also replicated to another filer.

Re:"Their backups weren't working."

[AmiMoJo]

How could someone in this position recover from it?

Suing the company for giving you in a potentially career ending position through no fault of your own might generate a one-off lump sum, but will be expensive and risky and likely make other companies not want to employ you in future.

Doing nothing and hoping they don't sue might be the best thing. Just keep quiet, never mention that you ever worked there in your job history and try to move on. If it brings the company down or they do sue and your name becomes public though...

Re:"Their backups weren't working."

[squiggleslash]

The engineering team responsible for the chain of events that led to this colossal fuck are completely and wholly to blame.

It's so over the top it's hard for me to believe any of it. If the CTO who fired him was described as a "Fat woman stuffing her face with a donut", then we'd know for sure it was just another Reddit story, but alas, no obvious signs beyond it being close to impossible to believe a department could be that stupid.

Re:"Their backups weren't working."

[HornWumpus]

You just never mention 'one day' jobs, for whatever reason they happen.

The better question: How will the CTO recover from this?

My recording school instructor/head...

[Anonymous Coward]

...supposedly erased an entire tape's worth of recordings (the only copy) at the Nashville studio he interned at.

Bullshit

[Anonymous Coward]

I have a feeling this is a dup of the bullshit story /. tried to feed us a couple of years ago, which after some investigation turned out to be 100% pure made-up bullshit. (No, I don't feel like searching /. history, but I'm sure it will turn up later in the comments.)

Anyhow, as we all commented in the previous story: If this was possible, then the company deserves to go under.

Re:Bullshit

[_merlin]

There was a script in Steam for Linux that did just that and wiped out some home directories when the variable managed to be empty. I didn't hear about it happening to anyone on their first day of work.

bullshit click bait ...

[Hugh Jorgen]

just like the "supposed" rm -rf / last year ... And if anyone is given access on their first day to production data, eh the CTO should be whacked first.

Re:bullshit click bait ...

[lucm]

just like the "supposed" rm -rf / last year ...

Something similar happened at Columbia Internet a while ago...

http://ars.userfriendly.org/ca...

Re:bullshit click bait ...

[__aaclcg7560]

In an IT department a long, long time ago...

No worries

[Anonymous Coward]

It would seem that the junior programmer is in line for a big payout, you see firing him, using profanity and implying legal action is not permitted. The poor guy doesn't even have to hire a lawyer in some states (like California) that have a Department of Industrial Relations (Labor Board) which he's _required_ to use instead of a lawyer (unless they eventually grant him permission to pursue with a lawyer). The way the Law will view it is that the company is totally responsible for the consequences of giving him too much responsibility.

Now you can argue the Law back and forth for all the good it will do you, this is nothing novel and this has happened many many times before.

Re:No worries

[HornWumpus]

CA is an 'at will' state. He's fired and that's all there is to it, unless they talked about him in public (slandered or defamed him).

Nothing new

[Tourney3p0]

Pretty much any comment that might be posted in this thread was already posted in the original Reddit thread over a week ago. Nothing insightful or interesting will come from this being posted here now.

Re:Why Was He Mucking With It In The First Place?

[sanosuke001]

He was given a worksheet with information regarding how to clone a test database onto his workstation. He ran a script when printed out the test DB's URL and login information but he just copied the info off the worksheet instead. The info on the worksheet was the production database with an account containing full write access. He overwrote the production database with the test data he was attempting to clone.

Whoever gave a new hire a worksheet with a URL for the production server and login with full access should be the one fired. This new kid showed the company that their new hire setup is completely insecure and that their backup infrastructure doesn't work. It's not this kids fault.

"Fire you? We just spent $2M educating you"

[Anonymous Coward]

http://the-happy-manager.com/articles/characteristic-of-leadership/

A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, “I suppose after that set of mistakes you will want to fire me.” Watson was said to have replied, “Not at all, young man, we have just spent a couple of million dollars educating you.”

Re:Why Was He Mucking With It In The First Place?

[charlesbakerharris]

Mebbe read the article?

S/He should sue...

[YVRGeek]

for wrongful dismissal. The fact that this F'd company's CTO left their *entire operation* vulnerable to such a simple and possible blunder means he should be fired, not the new hire. Someone tell that asshat, it's 2017... there is absolutely NO excuse for what happened being even remotely possible. And the fact that the new hire was following *verbatim* instructions in the setup doc and caused the issue is beyond laughable... I would fire the CTO for extreme incompetence. Whatta douchebag.

If backups are not working and this is known...

[nomad63]

If a company looks at a non-working backups as a minor inconvenience, I think the CTO's ass should be on the firing line before this poor guy's. Yes, wat he did is inexcusable and in some cases, firing might be justifiable (as in, a junior developer on his first day in his/her job doing what on a production database case) but if someone assigned him/her to perform anything on the company-critical data, that person should be the one getting fired, not this guy's. I am a 20+ year experienced sysadmin and not too long time ago, when I started at a new position, I was not able to touch any system other than few development machines for 2+ months of my start date and I know/knew my shit. This company management shows their incompetency in more ways than one. Yet they are making this person the scapegoat. Good riddance to them, as their days are numbered.

Re:If backups are not working and this is known...

[Njorthbiatr]

Yeah, they were doomed to fail without backups.

What if their server failed irreparably? What if some code went rogue and overwrote it? What if the server burns down because someone somewhere in the building left the stove on.

The CTO should be fired for total incompetency. You can have read-write access to database servers without having access to schema changes on the database. Personally, even though one of my creds actually gives me this access to some of our production databases, I never do it myself and always ask our admin to do it for me. This should be the default policy. Not even senior devs should be allowed to make schema changes on production databases without having limited credentials/sysadmin oversight.

Re:If backups are not working and this is known...

[thegarbz]

Yes, wat he did is inexcusable

Following the onboarding process on a sheet of paper that was given with him that some numbnuts decided should have the address and administrator credentials to the production database on it, on his first day, unsupervised?

What this guy did is 100% excusable. The CTO and whoever created the onboarding process should see themselves out.

I mean fuck I was on a visitors badge fully escorted for a whole week at my first job. That's right, they wouldn't even let me simply walk around the building unsupervised to say nothing of getting mission critical credentials on my first day on an instruction document that could potentially seriously fuck up said mission critical system.

Allowing it to happen is wrong to begin with

[sentiblue]

I've been in production management for 15 years... The first 5 years as a Sr. datacenter engineer. The next 7 years as a staff engineer and the past few years as a cloud architect. One thing that has NEVER changed is that developers are NOT allowed to touch production... and I mean not allowed to even log into any host at all...They don't even have a damn clue where the machines physically sit.

Allowing a junior developer fresh out of college to log into production with privilege that makes even a minor change is already in violation of industry protocols. I wouldn't blame the guy too hard. He's fresh out of college remember? The first person to blame is operations management who doesn't have a clear separation of duties protocol. Even if ops management allows him to log in, the next person to blame is the lead datacenter engineer who allows his account to issue intrusive commands.

If company leadership asks me to handle this fall out... I will fire those two guys first. Then the Jr. developer will get only a warning but that would be his only warning.

Re:Allowing it to happen is wrong to begin with

[Just Some Guy]

So much this. It's a major PITA requiring permission from a director for me to get access to a machine that can access a production database. I am perfectly fine with this arrangement!

Re:Allowing it to happen is wrong to begin with

[rossz]

One thing that has NEVER changed is that developers are NOT allowed to touch production... and I mean not allowed to even log into any host at all...They don't even have a damn clue where the machines physically sit.

In my office, only one developer has limited production access, the senior guy. He's the only developer who can do code releases and he has RW access to the DB, but he can't mess with the system's configuration. If he's out of the office, I have to do the code releases as the senior system administrator.

Some Company Information Please

[mykepredko]

I'm surprised that the firm has not been named - while I would think that any company that had this happen to them would want to keep this confidential, I would think that somebody would talk about it separately. I suspect that the "company" is some podunk startup in which the CTO is the CEO, CFO, head of development and probably the HR head and they've just hired a developer without thinking about access restrictions (or verifying that backups are actually happening).

Some more information would help clarify these questions and maybe better explain how such a situation could happen.

Re:Some Company Information Please

[ryen]

The author in the reddit thread (in a buried comment) describes it as a 100+ company with at least 40 devs. Pretty sad

Re:Some Company Information Please

[wvmarle]

Sound like your run-of-the-mill startup that has "the great next thing" and in the process got more investor money to burn than is good for them, so starts hiring more people than they can properly manage.

Re:Some Company Information Please

[CaptnCrud]

100 strong doesn't mean much, if its a sales heavy parasite company.

Probably every one of those devs are contractors with less than 6 months at the company....I've seen revolving door companies like that before, they are pure meat grinders. The pure essence of evil, they have no incentive to put good procedures in place or make sure backups are working because they have no incentive of existing beyond milking their current situation dry.

Re:Why Was He Mucking With It In The First Place?

[Excelcia]

He wasn't mucking about with prod, per se. He was following the instructions he was given for creating a sandbox copy of prod, and those instructions at one point had an example login he was supposed to replace, and that example login was a valid admin login for prod. He used the example login, admitedly by mistake, and ran the scripts that were supposed to scrub the database and give him an empty sandbox copy of the prod db structure to play with. Those scripts, however, because of that bad example login, ended up running on the real prod with actual admin access. Small mistake on his part. Huge mistake on the documentation's part.

Despite the number of catastrophic failures of the system, they fired him. Likely so the CTO could go and claim it was malice or vandalism on the part of the new hire, instead of owning that they had documentation that included admin prod access as their example.

What I find is interesting, is that despite what this company has done to him, he has never once named them. This is an example in loyalty, professionalism and discretion that is rare today. I would hire him in a heartbeat. I want that kind of professionalism.

Re:Why Was He Mucking With It In The First Place?

[lucm]

Having production databases that can be reached from developers workstations is always a bad idea. They were lucky, in a way, that the developer deleted the data and didn't just alter it slightly.

Firewalls, people, they're not just there to block wordpress exploit scanners. Danger can come from insiders, and that includes hostile employees as well as clueless morons.

Sounds like a poorly run IT system

[quietwalker]

Seriously, a day 1 dev has direct production access? Hell, any dev has direct production access? No QA, no release management, no integration or functional test suite if they're doing some sort of continuous deployment?

It's a pain in the ass, but if they've got any sort of actual real database, they'll have had a real database admin, running it with archive logs they can use to restore their data? ... plus their backups are gone?

What sort of fly-by-night operation is this?

Re:Sounds like a poorly run IT system

[dbIII]

Hell, any dev has direct production access

I agree with that - different mindset. Devs want to get stuff done ASAP and usually don't seem to get the concept of multiuser systems. Even very experienced devs do shit like reboot servers in working hours leaving dozens of staff twiddling their thumbs and unable to work unless months of effort has been put into changing their attitude to production.

We need to have medals to give out.

[clovis]

I say if he succeeded in putting that company out of business, then he should get a medal for sacrificing himself to destroy the company.

My belief is when he saw on his first day, the badly written docs they handed him, with a printed (!) account/password having RW access, he instinctively threw himself on that grenade by destroying their production database. Only the most cowardly IT worker would have done otherwise.

Thank you, selfless IT worker from saving us from the horror of whatever product they were trying to produce.

I accidentally the whole database

[rpresser]

now I'm too full

be happy

[ooloorie]

You don't want to work at a company where the backups don't work and where a new hire can accidentally delete all their data. Don't beg to stay, instead be happy that you found out quickly how incompetent that company actually is.

Re:be happy

[hyades1]

Wish I had a mod point for you, my friend. That is exactly, 100% correct. Rookies make mistakes...sometimes even stupid mistakes. It happens.

If a rookie can wreck a company this badly, it's hardcore proof that the problem is a long, long way up the food chain.

THAT is where heads should roll.

Re:be happy

[Anonymous Coward]

This exactly, if it literally happened on day one just walk the fuck away from this dumpster fire of a company and look for another job. Do not put this on your resume. If the company tries to publicly name and shame you in any fashion, immediately consult a lawyer.

Re:Why?

[__aaclcg7560]

That's funny. When I started reading Slashdot back in 1999 or so, it was called a website. Someone probably slapped on the "social media" label a few years ago to get some VC money.

Re:Why Was He Mucking With It In The First Place?

[Kjella]

Having production databases that can be reached from developers workstations is always a bad idea.

Welcome to DevOps ;)

Re:Why Was He Mucking With It In The First Place?

[Darinbob]

Welcome to startups. It's very good experience straight out of school to know just how screwed up startups are and to avoid them for the rest of the career.

Re:Why?

[Anonymous Coward]

You do realize that 99.999999999999% of "social media" is week-old posts...so why come to any "social media" sites at all? Why not cut out the middle man and experience reality directly? You know? Like humanity used to do....?

Back in the day....

[GerryGilmore]

...we were a small company making the transition from a proprietary (TI/990 - DX/10 OS) system to a Prime UNIX system and I was the guy A) learning UNIX and B) writing shell scripts to replicate basic DX/10 stuff. One of my first scripts was to delete a user. Beyond the basic password deleting stuff, I thought it would be cool to delete the user's home directory also. You know, delete user "gerry" and also delete user's home dir "/usr/gerry". Guess what my brilliant script writing failed to anticipate? Yep, someone entering an empty user name. So, the script - running under SUID perms proceeded to delete "/usr/*", including "/usr/data". Oops! Fortunately, my backup script was better written and the customer only lost a few hors' worth of work, but....Whew! That's also how we learn!

Re:Back in the day....

[rvw]

> You know, delete user "gerry" and also delete user's home dir "/usr/gerry".

Who puts a user home folder in /usr/? Isn't that bad practise in the first place? The usr directory is for system stuff.

Re:Back in the day....

[david_thornley]

That's what the /usr directory is for now. Back when I started using Unix, we had /usr/home rather than /home.

Ya no shit

[Sycraft-fu]

When we bring someone on, they do NOT get root/admin to critical servers their first day. They have to be off probation first, which is 6 months where I work. Even then, credentials for things are not on a document. That is just asking for them to get lost or stolen. They are given on a wallet sized card, written specially for that person, and they are instructed to keep them safe until memorized.

The reason is, of course, to prevent fuckups, as well as to make sure we trust them fully. The idea of giving someone full access to critical stuff on day one is stupid. Shit it sometimes takes more than a day for them to get access to e-mail and all that just because of all the other things they need to do.

This is 100% on the company. Have working backups, CHECK YOUR BACKUPS, and don't give a new hire a sheet with access to your critical data.

This was inevitable

[gweihir]

I mean, come on:

- No working backup
- Excessive access for the new person
- CTO is incompetent and cannot admit to mistake

This is an accident waiting to happen. And the new person has zero responsibility for it. Might be better off to be out of that fucked up company though.

Re:Why Was He Mucking With It In The First Place?

[Just Some Guy]

We've had very different experiences. The startups I've been at have been packed with smart, competent, diligent coworkers. I've had much worse luck with large corporations where employment was effective for-life and it was so easy to blameshift that no one ever got fired for anything.

Re:Why Was He Mucking With It In The First Place?

[Shadyman]

Having production databases that can be reached from developers workstations is always a bad idea. They were lucky, in a way, that the developer deleted the data and didn't just alter it slightly.

You'd also have to pray they did not alter it any further.

Re:My opinion

[lucm]

This is what the anonymous poster actually said:

The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that i "completely fucked everything up".

It's not clear that the CTO was threatening legal action against the developer. For all we know, it could simply mean that the company would have to issue some kind of statement to shareholders or to customers based on some contractual commitment.

There's nothing in the entire story that clearly indicates that the CTO or HR acted improperly. Is it normal to have production credentials in documentation handed out to junior developers? Of course not. But the developer was supposed to configure a local instance using the output of a specific configuration tool, but opted instead to copy-paste from somewhere else a connection string that must have been obviously not a local instance. Then the developer, after being fired, left with his company-issued laptop, and started sending messages on the company chatroom. Would you want that person on your team?

Back Assward

[Tablizer]

Corporate cheapskatism fucking them in ass: newbies handle key data and no backup system in place.

Sue/fire the CEO, not the grad.

Shit happens

[Camel Pilot]

Hey I had a security guy come into our data room to do a drive inventory and while waiting for me his curiosity got the best of him and he popped open a drive on 70 TB raid.

Lousy Management/Leads

[oldgraybeard]

The CTO and all the managers/leads that created the opportunity for this to happen are the true failures. And to think, they are all still there, so that place is still really messed up.

Seems to me this is not a single point of failure, it is multiple points of failure acting in unison. There was a lot of passing the buck and ducking all responsibility going on. Not to mention several individuals not doing their jobs.

A final thought, I sure would not want to work in a corrosive, petty and vindictive environment like that. I know it does not help now, but drive on. Not working there may turn out to be a, not totally bad thing for his sanity and future in the long run.

The real test is not how well one handles success, but in how one responds to failure.

et tu, RAID

[lucm]

RAID (especially those with parity) can be terrifying. Just think of it: you have a group of disks probably acquired at the same time and probably coming from the same vendor (or even same production batch) serving the same workload in the same environment. That implies a fairly similar MTTF for all the disks.

Then one of the disks fail; this causes the other disks in the array to first handle a higher load, then to be brutally impacted by the rebuild process. That's like playing Russian roulette with a gattling gun.

Re:et tu, RAID

[Carewolf]

RAID (especially those with parity) can be terrifying. Just think of it: you have a group of disks probably acquired at the same time and probably coming from the same vendor (or even same production batch) serving the same workload in the same environment. That implies a fairly similar MTTF for all the disks.

Then one of the disks fail; this causes the other disks in the array to first handle a higher load, then to be brutally impacted by the rebuild process. That's like playing Russian roulette with a gattling gun.

Yeah. I had a RAID-5 at home. When one disk starting failing I didn't notice because the system kept running, and being a home setup it didn't actually have any lights or warning unless I manually opened the RAID manager and checked. I noticed when a second disk failed this time completely for all sectors, during the recovery two more disks started failing.

Re:et tu, RAID

[ColaMan]

About 15 years ago I was looking after an old Compaq ProLiant server that had a 6 disk SCSI raid array in some configuration I cn't recall.

Oh, a drive just failed with an "Exceeded power on hours" error? Well, that's ok, there's a hot spare in the array, no problem.

Next day, two other disks went offline, because they were all powered up at the same time when they were new, weren't they? And they were perfectly usable, it's just that the array controller noticed that their smart attributes had exceeded a threshold that equaled "failure", powered down the disk, and that was that.

It wasn't a very pleasant week after that.

Re:et tu, RAID

[angel'o'sphere]

You have a point, but that has nothing to do with MTTF, note M means 'mean'.
In other words: if one of the disk fails 15 days after purchase, and the last one after 30 years, the mean can still easily be an acceptable 15years.

So, what you want to imlly is that they come from the same batch, and might all have the same flaw or faulty component and hence might fail in short time of each other. So yes, that again leads to 'same MTTF' but that is not what MTTF means or describes.

Re:et tu, RAID

[Asgard]

Was that a default configuration of the controller?

Re:et tu, RAID

[lucm]

The MTTF is provided by most disk vendors, it's not a calculation one makes by hand. The point here is that if a disk shits itself way before it should, odds are that disks from a same production batch running in the same conditions are likely to follow.

Re:et tu, RAID

[fateblossom]

I'm runnign a RAID 6 at home.
5 disk from one batch and 1 from a nother (Thats how I got them)
I have set up my NAS to E-Mail me with any warnings.
So when one disk failed. (One of the 5 from same batch) I order a new disk, and shutdown my NAS when I got home. And the next day I got my new disk. So I replaced it. And startet the NAS up and it used a LONG time to do the recovery.

So.
1) Alway set up your NAS/Server to send your warnings.

2) Just because the disks are from the same batch do not mean that they will fail at the same time. (The other 4 from that batch is still running)

And the reason I shutdown my NAS even when I could lose one more disk. Was it was my Home NAS. I could live without it for a day. And just in case more disk was about to fail. I could still lose one doing recovery without losing date. So just me being extra carefull.

Fake. Made up story.

[McBooCZech]

full stop.

Test backups by actually restoring

[Maxo-Texas]

We backed up daily, weekly, and then monthly.

Fortunately decided to restore an older copy of a file to work with and found despite the IBM software saying the backup was occurring and was good- it wasn't.

None of the backups were any good. IBM had to come out and figure out what was going on and after that we also retired a file from backup every single day. We had previously used to disk to disk backup but went to tape when the data got too big.

If you do backups- you need to actually restore a file and confirm the file isn't empty.

Re:Test backups by actually restoring

[stoatwblr]

I can tell you a story like that about a NEAX61M telephone exchange with 30,000 subscribers on the other end.

Which was writing crap to backups.
For at least 2 years before it was discovered.
It was discovered when the system was restarted and came up braindead.
It had been restarted after loading in y2k software to prevent any crashes and stepping back to the old software wasn't an option.

Mobile phones weren't a big thing in 1999

People get upset when they can't make phone calls. They get even more upset if they have no dialtone and 911 isn't working.

It took over a day for a 2.5 year old backup to be located and restored - but the problem with phone switches is that people move house regularly, so thousands of phone lines didn't have the number they were supposed to have.

People get even more upset that they can't _receive_ phone calls than when they dial Fredd and end up talking to a complete stranger on Fredd's number and someone dialling Bill ends up talking to Fredd, etc.

It took more than 6 weeks to replay every single transaction into the switch to bring it up to date - and even after that had supposedly been done properly there were still hundreds of errors because on every played back transaction worked.

We don't care. We don't have to. We're the PHONE company (Since the market opened up to dialtone competition in 2011, this company has dropped from 100% of the enduser market to something less than 20%. I wonder why?)

Managing to simultaneously piss off everyone in an entire city takes some doing.

Re:Why Was He Mucking With It In The First Place?

[lucm]

Not to mention - the very premise of just copying production database for use in testing is already terrible.

In theory, yes. But honestly, I've seen countless situations where there was no realistic alternatives given the available resources, especially if the volume or distribution of data could have an impact on the system. For instance, testing a new address autocomplete feature for an order form; if you only have a few hundred clients, maybe you can generate some decent test data, but if you have millions of customers it can be very difficult to make sure that the feature will work with the real data set. Same goes for specialized reporting (e.g quantitative analysis features in a trading platform).

There are a few interesting data obfuscation tools out there if data privacy is a concern, but they can be quite labor-intensive to set up. Sometimes an ironclad NDA and a copy of real prod data is the only available solution.

Multiple levels of fail

[rossz]

Lot's of blame to spread around.

The code used in production should have been reviewed by someone before execution in production. No exceptions. Especially because it's a new guy on his first day. The code should have been run in a staging environment first. How long was it known that the backup system was broken? This mistake was obviously not the newbies fault.

If my production DB backup was hosed, I would be dropping just about everything else to get it healthy again. A deleted database would mean some lost time since loading up the last backup and doing a PITR from WALs isn't fast, but at least I know I can get production back up no matter what someone does to the active DB.

Re:Why?

[AHuxley]

Freedom of speech and freedom after speech.

Re:Why Was He Mucking With It In The First Place?

[Darinbob]

Yes, but with startups you can have brilliant people, rarely, but almost always there's little coordination as procedures haven't been set up yet. Plus a constant non-ending rush to meet short deadlines, with demos needing to be done often and no time to slow down and take stock of what's going on.

Fire the CIO and CISO

[Opportunist]

If a junior developer can FUBAR the company, these are the two that fucked up royally. Very obviously processes are crap and gross negligence is running rampart.

Fire these two bozos. Out of a cannon.

Ah, the memories

[shanen]

Actually reminds me of two fellows I worked for briefly. Both of them were actually too small to have a separate CTO. In the first case I decided to leave as soon as I figured out the legal liability their main customer had incurred due to pirated software. There were actually two packages, and one of them was a database. The second case was a total shoestring operation and one of the first things I discovered was that their so-called daily backup processes were not actually backing up anything.

Now that reminds me of a third case, where they actually did have a CTO, in title if not in competence. While they did fire him, it was too late to save the company... It was one of the few things they'd done right while I was working there, but I decided to leave when one of the actually competent managers was fired, and they disappeared not long after that. (That was actually my last stint in a startup, and after that I just went back to safety in a big company for the rest of my career...)

Re:Ah, the memories

[sheramil]

... The second case was a total shoestring operation and one of the first things I discovered was that their so-called daily backup processes were not actually backing up anything.

Ah, that moment... months into it, when you decide to check the backup media and find it's empty. Heh heh heh.

Re:Ah, the memories

[HornWumpus]

It's actually pretty simple to make the last step of a backup be: Restore to a particular development database server.

And it's handy for devs to have near current data to test against, as long as it's not the only dev database.

Re:Ah, the memories

[shanen]

I feel obliged to note that one of my highest priorities before messing with anything was to figure out the state of the backup system, even though the boss was screaming about other emergencies. It still took me a couple of days to figure out what was going on, or more precisely not going on as regards backups. Various parts of the system had been done by various people with various degrees of competence, all of whom left for various reasons.

Re:Ah, the memories

[minstrelmike]

I wrote a cassette tape backup system for pcs a long time ago and was also on the help desk. I was talking to one of the users about something else and they just mentioned that the new backup procedure (which handled multiple kinds of units) was really nice and fast. It finished in less than 2 seconds!?!

If the tape unit was not found at all, not plugged in, then the software failed with a bright Success! message. I did not think to test not plugging in the unit.

Always test your disaster recovery plan!

[No Longer an AC]

At least know that your backups work.

In the mid-90s I had just started working at some place and the guy I was replacing (he left for a better opportunity) was showing me how everything was set up and as a demonstration he deleted his own account. I guess he felt he didn't need it anymore, but then he says there might be some useful stuff in there and tells me it would be a good exercise for me to learn how to restore from backup.

Not a big deal, it was actually documented and I had done that before at a previous job...but every single tape was bad.

When I told him, his reaction was that I must be doing it wrong, but he was similarly unsuccessful. I think they had been using the same tapes ever since they bought their system and they were all kept in a shoebox in the same small office with all the servers.

After 6 months I left for a better opportunity too.

I have seen so many unbacked up production DBs

[EmperorOfCanada]

It is scary how few companies backup their crap. I would be willing to bet that less than 30% of billion dollar companies (that really depend on their computers for day to day operations) could be back in operation in under 24 hours if they lost all their servers at once.

I wouldn't be surprised to find that a good percentage would be very screwed in the long term.

This is not only their data but the system as a whole. When I am consulting at most companies that are retiring servers I often suggest that they box them and put them in a safe place as a super emergency backup. The idea being that those servers are probably close to what they need to struggle along. This is after I have usually laid out a solid backup and disaster recovery plan. Seeing that most companies won't do a good backup then having some 2 year out of date servers might be better than nothing.

I did this too

[riverat1]

I can relate to this. I wiped out the production database for our ERP system when I was trying to create a copy of it. Fortunately I had good backups and was able to restore the DB with minimal loses but it took all day (this was back in 1997 and the computer wasn't particularly fast, a SparcServer 1000 with 2 CPUs and 500 MB of RAM). In my case I wasn't fired and retired from the job last year after 31 years.

Re:How is it reckless?

[lucm]

The C-level executives are the people in charge of executing the company strategy as established by the board (who are representing the owners). If you remove them, you basically have a headless chicken. If you replace them all at once, then you have months of instability as they have to rebuild their teams and puts their systems in place.

That's why elections are staggered in the USA. Could you imagine the mess if everyone was elected at the same time?

Re: How is it reckless?

[Anonymous Coward]

Um, the elections in the USA are all at the same time. Local, city, state, federal, all on the same day. The four year presidential cycle sees 200% of congressman and 67% of the senate cycled too...

You fail at civics.

Re:Why Was He Mucking With It In The First Place?

[No Longer an AC]

Having production databases that can be reached from developers workstations is always a bad idea.

I don't disagree but sometimes it's impractical with smaller teams especially if that team consists of only one person and they are also expected to provide support for production systems.

It's just VERY important to make sure you're aware of which system you're on.

The worst I ever did was delete all the inventory in an entire aisle in one of our customer's warehouses. Thank God I didn't delete the entire warehouse. I felt really bad as I called up the warehouse manager and told him about it.

He took it surprisingly well. I was expecting to be harshly chastised but he was very understanding for some reason. Possibly because they do physical inventories all the time and I just gave him a heads up on where his inventory team should focus next. Or maybe it's because I had dealt with him many times before and usually I fixed his problems instead of creating them.

That was about 20 years ago and as best as I can recall I had both production and development windows open at the same time and I adjusted inventory in the production environment instead of the development environment by mistake. I quickly realized what I had done and in a minor panic to correct it I compounded my mistake by more than 10-fold.

Since then I think most places have a change management process which involves more than a developer recklessly altering data in production on the fly without any oversight.

Even at the largest company I've worked for primarily as a developer I had full access to any production system my software ran on and there really wasn't a distinction between developer and sysadmin. Some were stronger in one area than the other, but we all did both.

Eventually they split us up and took away the developer's root privileges from all the machines forcing us to use sudo instead...and we grumbled a bit about that, but we accepted it. It really was the right thing to do.

Re:Why Was He Mucking With It In The First Place?

[lucm]

Eventually they split us up and took away the developer's root privileges from all the machines forcing us to use sudo instead...

sudo bash

Problem solved!

Re: How is it reckless?

[lucm]

I don't fail at civics. You fail at understanding what "at the same time" means.

Here's a hint.

https://en.wikipedia.org/wiki/...

Re:Why Was He Mucking With It In The First Place?

[JaredOfEuropa]

Some people thrive in that kind of environment. Working under pressure on something that really matters to the company, and with fluid roles and a lack of procedures, you have a lot of freedom as well as an opportunity to make a meaningful and unique contribution to the product as well as to the way the company will do things in the future. I know I perform better in such an environment than in the corporate one, with entrenched interests, highly compartmentalised roles, stifling procedures and generally terrible middle management. Sure, there's downsides to each environment as well as advantages. Personally I found a nice middle ground by working on innovation in larger organisations.

Re: Old News...

[KGIII]

Retired boss, here. If a junior dev can push to prod AND can delete data, it isn't their fault. Okay... I am still going to try to salvage my hire and see if they fit in QA, or see if the debs will stop fucking with him. Seriously, not his fault. Nobody should be able to push to prod, withou someone signing off. Ask me how I fucking know. I'm not even a programmer. I just paid a lot of you and shut the hell up and listened.

So, if I am wrong then they were wrong. You don't push code to prod without someone signing off. The person signing off has ultimate authority. You sure as fuck don't let a junior do it, without oversight. Not now, not ever.

If this happened in my shop, some titles would have been changed.

Re:How's the budget now?

[Calydor]

If you think hiring a professional is expensive, just wait until you hire an amateur.

Not His Fault

[StormReaver]

This kid will have no problem getting a new job. What happened was not his fault in any way, shape, or form. The fault lies squarely with the CTO (in no particular order):

1) He allowed a new hire to have superuser, unsupervised access to the production database without an ounce of training.
2) He allowed an unvetted new-hire script to contain actual superuser credentials that could be used to wipe out the production database with a simple copy and paste error.
3) He allowed a new hire to run that script unsupervised.
4) He set a new hire loose on their production database with no mentor looking over his shoulder.
5) He did not require the backups to be verified and/or tested to make sure they work.

I'm sure there are more (feel free to expand on it), but those are the ones I can think of off the top of my head.

Re:Not His Fault

[StormReaver]

By the way, I did something similar on my second day on the job (no, I didn't get fired). I'll skip the details, but yes the management fucked up and had a hard time coming to grips with their fuckup. Fortunately, circumstances in my case were slightly different; but it was close enough to this story that I don't blame the new hire one bit.

Systemic issue resolved....

[neurosine]

Although obviously the new guy fucked up...unless his job was to be solely responsible for the repository...he probably shouldn't be fired. He's pointed out a systemic issue with the system...and punishing him for it isn't reasonable. I don't know all of the details...but on first glance...the people responsible are looking anywhere else for the blame. Typical.

Re:Old News...

[K. S. Kyosuke]

Be more considerate; /. has just recovered its deleted database. It did take some time, obviously.

Seems pretty obvious...

[DaveAtWorkAnnoyingly]

They gave an employee a procedure that deleted their database. The only mistake the employee made was following the procedure and not using his own credentials, instead using the credentials in the procedure. That was a standard human error, no big deal. Humans make mistakes, it's all of our's jobs to reduce the probability of this, with good quality procedures, questioning attitudes etc. To provide an employee with a procedure that contains credentials to the production database is ridiculous, this was going to happen at some point with a procedure as poor as that. To then blame that employee immediately and fire him is knee jerk and immature. To be fair, the guy is probably better off not working for that company anyway.

Not having backups etc, well, that's the companies fault. All the employee has done is to uncover a poor working culture within that company, and expose a bad CTO. This is not the failing of the employee, it's clearly a failing of the company. They deserve what they got.

The person fired can make many positives from this. This is an excellent case study about business culture, procedures, strong catastrophe planning and testing, threat and error management, how to treat fellow humans. Your experience in this can be valuable to other companies, and the good ones will recognise this. Use this to your advantage, let that company fail, it was going to happen at some point, you were simply unlucky...

To give a personal experience, I work in a nuclear power station, in the control room. One shift I made a mistake, I rushed a job. I followed my procedure to the letter, I didn't make any mistakes, but because I chose to complete a certain activity early before an additional check was performed I broke our operating rules (the law effectively). I owned up to it as soon as I was conscious of it, and the investigation started. My authorisation was pulled and I went for drugs testing etc.

The first thing that occurred once management found out was that I received a phone call. It was the Operations Manager calling for me, and he called me to thank me for raising this report and owning up to it. He understood that without this honest reporting, the problem couldn't get fixed, and would happen again in the future. It has probably already occurred in the past too. My authorisation was recovered that same shift, and I carried on. There was no detriment to my career (quite the opposite infact) and life goes on, but now with better procedures...

Industry Standard Best Practices Ignored

[zifn4b]

- Insufficient IT resources (inadequate or no backups)
- Hiring developers to do everything in the company and giving them broad access to everything

These are two industry standard best practices that many companies ignore and they have no one to blame but themselves.

backups

[AndyKron]

Backups weren't working? Who's fault was that? That's who needs a talking to.

I'll believe this...

[Evil Kerek]

This post smells a bit. I'd like to some proof it actually occurred like this or that it occurred at all. The fact we've heard nothing indicates it was a small company - this isn't the sort of thing one keeps quiet. The lack of a functioning backup tends to also point at a small company.

The description is such a slam dunk in favor of the new guy and making out the CTO to be such an idiot...I dunno....I'm just skeptical at this point.

The rancor coming from what amounts to utter hearsay is...well actually, no, it's not surprising these days. Any sort of proof for people to pile on stopped being a requirement long ago - it won't surprise me if this is someone doing a social media experiment.

My 0.02
EK

This sucks so bad for that kid it's not even funny

[tomxor]

Fuck you incompetent business, this will go down badly for you in the media: Anyone hiring a junior straight out of college knows not to throw the crown jewels of the company into their lap to play with, and if you don't have working backups then you're double to blame...

Kid: Don't worry, notice how every single comment here will put the blame squarely on your terrible employer. It's just luck getting a first good employer, you will find more. Remember the hill climbing algorithm... your marble just rolled straight into the deepest pit, it's not your fault it's just bad luck, you've got a lot of random rolling to do, keep your head up and you will eventually find some nice reasonable people that appreciate you.

Re: Old News...

[Highdude702]

I dont think people like it when you take away their job title and replace it with 'fuckwit' and 'moronic fuck who leaves permissions open'

Re: Old News...

[volodymyrbiryuk]

Exactly, this is why the people who are in higher positions are payed more. What else could justify the higher salary if not the greater responsability. Because usually they don't do low level stuff, the actual work that drives a company.

Re: Old News...

[Anonymous Coward]

If you were his/her boss then you should dig a big deep hole for yourself, jump in and ask the junior very very politely to put the dirt back into the hole. Really. I would be very very ashamed of myself as a CTO if a junior dev would have access to a production database. I'm a very old dev and I still categorically refuse to do ANYTHING on production.

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

When DevOps exists, DevOps has responsibility for what's in prod.

The guys staffed, trained and assigned to cover emergencies 24x7x365 and answering that call at 4:59pm on Friday should be the only guys who can mess with production code.

Re:How is it reckless?

[someone1234]

But this company had a very erroneous fault control strategy and/or execution. The CTO was definitely not on the ball.

Perhaps it will make more sense this way

[dbIII]

It's not about good or bad, it's priorities and a lack of understanding of what the implications of disturbing a production environment are. They are not going to get that understanding by magic, they only get by looking at how a system is used over time, and if that's not related to what they are supposed to be doing for a living it's hard to put in the time.
Like the above problem it is a management issue. If you want a dev to act like a sysadmin you have to schedule time for it to happen.

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

"Sometimes an ironclad NDA and a copy of real prod data is the only available solution."

When you're getting executives to sign off on risk and only handing the data to specific named trusted developers, then it's not "just copying production databases into testing". It's a calculated risk by the executives, as informed by their staff.

Most of the time, when a developer complains about not having access to prod, the request disappears when I tell them to get their boss to go to their boss and pitch the case. They usually find a scrubbed set of data or another method to reproduce.

Shoulder surfing operations (with operations hands on keyboard), often solves things too, but everyone hates it. It has a huge benefit though in that operations needs to know what the developer is thinking as much as the developer needs to know whats' going on in production.

no backup? i've no sympathy at all..

[zr]

none at all..

Re: Old News...

[thegreatbob]

This is more like they had a spurious cable attached between some random door below deck, and some critical lever in the control room... The point people are trying to make is: this guy is being given the shaft as a direct result of the fallout from idiots putting privileged/critical information into widely accessible documentation.

The company's backups weren't working,

[QuietLagoon]

When the root cause of a problem presents itself so clearly, it is difficult to understand why the new person was fired. The person in charge of doing the backups should be the one who was fired.

Let me guess

[nospam007]

The company is named 'British Airways'?

Everyone calm down

[jon3k]

Let's all relax, this is obviously fakey. Slashdot is now citing reddit posts from self-proclaimed throwaway accounts as news? Yeesh.

Process, process, process..

[malkavian]

The company is responsible for process.. They should be continually having it as a background task (even in the back of their minds) of "how do we keep the wheels on this thing".
Just as important for a startup as it is for a big company (maybe moreso).
Starting point of everything is backups, backups and more backups! They're your lifeblood if you're a company using data for doing real things (hint, they all do).
If a backup isn't tested, it's not a backup. If you've not given your systems team time and resources to test backups, then the problem lies with a lack of understanding (or listening) further up the management chain.
Once you know you can recover, you know you can use your data safely... Then it's the process for letting someone loose in your systems...
If they're not authorised for prod, then they don't get the credentials for prod. And credentials should be stored in a nice encrypted file on a VERY well backed up area. Credentials should NEVER be in plain, or embedded in unencrypted documents.. That''s so much a 'no' that it fails the basic test of "do you understand what the word security means?".
Rough analysis of the situation is that there's a marked problem with process, and large problem with security and information governance, and an insane problem with backup processing. An accident waiting to happen..
The developer made a mistake.. Yep, a big telling off for not engaging the brain and being careful with the credentials is well in order. You need to treat systems carefully.. But us old grizzled systems farts got our caution from somewhere, and it's called experience.. And experience involves coming up through the ranks and seeing stuff go wrong. Even stuff that's almost guaranteed never to go wrong. I'm all for having young blood in a place.. They do bring new approaches, and new ideas.. But us old farts have seen many of the approaches before and can say why they're broken and/or dangerous.. It's just that little gap of new idea where we thing "Hey, that's actually new.." that means things move forward... But with us old farts around, it happens in a controlled and safe manner..
So, I'd say this guy is actually MORE valuable than he was before he made the screwup.. I can be fairly sure that they'll be MUCH less likely to be cavalier in the future, and put an awful lot of thought into safety of systems..
From the way it's set up, I'd actually say they're better off out of that environment anyway with the CTO having that lack of understanding of the root cause of the problem and/or management also not understanding/standing up for the dev and explaining root cause.
Alas, I'm not hiring at the mo, otherwise I'd have a serious chat with them and pick them up.. People getting through extremely bad screwups AND openly admitting they screwed up is a good sign of a person you want on your side.. Screwups happen.. It's the people that HIDE them that are dangerous..

Re:Process, process, process..

[angel'o'sphere]

In the environments I work, developers can not even connect to any prod and most of the time not to any preprod system either.
They only can access dev and test environments, and often they have no access to the test filesystem, hence admins have to set up test dbs, test systems and basic test data and developers or their software can only access the running DB or other software via REST etc.

Likewise admins that actually can work on PROD have usually no access to test or dev. Lets call them 'operators', old school :)

Toddler with a gun?

[Theovon]

The security measures we put in place in IT systems aren’t just about keeping out malicious users. They’re also about granting access only to those who understand how to use them properly. You don’t give guns and knives to toddlers, and people who do are the ones responsible for the carnage.

That being said, I recall one time that a friend wanted to back up his hard drive before upgrading his OS or something. He had lots of music and photos and stuff, so that made sense. So he asked our sysadmin to help, and the sysadmin used some disk imaging software to do it, to copy from one drive to the other. Unfortunately, he got the drives backwards.

Irony

[Shoten]

You know, a few weeks ago there was a post on reviewing code before pushing it to production, and some people seemed appalled by the idea that another person...who would invariably be a coder, lest they be unable to understand what they were looking at...would be in the process flow for committing a change. I pointed out that this was actually an industry best practice in enterprise organizations, and an inherent part of any SDLC. In any large environment, people with access to development environments do not have the rights to push changes to production. This concept was not well-received.

And now, I see an awful lot of people saying that this problem above was the CTO's fault, for giving a developer sufficient access to change the production environment. I agree with that point...but it's amusing to me how a lot of us seem to live in a fantasy world where we must have access to everything, but when we screw up it's someone else's fault for giving it to us. We have to choose one of the other; either we are justified in having full autonomy and accept the consequences of our actions...for good and for bad...or we accept safeguards that protect both us and the organization we support.

Re: Why Was He Mucking With It In The First Place?

[mixed_signal]

Wrong conclusion. Rather, avoid startups staffed with inexperienced people.

Re: Why Was He Mucking With It In The First Place?

[mixed_signal]

Or at 2am Sunday...

Re: Old News...

[Applehu Akbar]

That newbie going the wrong way on a copy operation could even more easily have been a disk error or a ransomware infection. The problems was not having backups. I'm with the Register readers on this one.

I fail to see how they have the legal ground to do

[waspleg]

anything. You would have to prove malice. This was clearly an accident and something they should have been able to recover from.

Firing? Whatever, I'm assuming this is America we're talking about where I can fire you because I don't like your fucking ankle socks and you have 0 recourse.

The root cause

[XB-70]

Forget the kid, forget the CIO etc. The root cause here is that setting up an easy backup system is a right, royal pain in the ass.

Backup protocols should be integral to every database vendor's product. Db creation should not be allowed until a backup system is in place FIRST.

Secondly, Information Systems Audit and Control Association, ISACA standards (or similar) should apply and vendors should work to integrate those standards into their database creation protocols.

In the end, your database represents your organization's livelihood. Why would you not protect it as such?

Re:The root cause

[Wolfrider]

> Backup protocols should be integral to every database vendor's product. Db creation should not be allowed until a backup system is in place FIRST.

--I find what you say to be fascinating, and would like to subscribe to your newsletter. :b

--Veeam bare-metal backup does a pretty good job of this, there's a prompt to generate a recovery ISO before setting up/performing the 1st backup.

Learning From Others

[Anonymous Coward]

Guys, THIS, and crypto locker, and wanna cry, and fat-finger (human brain virus that causes people to hit return way to fast) is WHY you pay for ENTERPRISE STORAGE that takes snapshots that are efficient, and fast.

There is a difference between logical protection (snapshots on the same array) and physical protection (backup to different devices).

While we are at it, WHERE is the DR copy? If a sprinkler went off in the datacenter or drive/array data loss occurred, where would the production data be?

WHO has money for a CIO/CTO type and can't afford proper Enterprise Storage?

I work in the Enterprise Storage industry. Use a modern enterprise storage array - take snapshots, replicate to another array, take snapshots there, too. Take backups from the snapshots (so as to not affect production). Use a backup SW that support array off-loading so you can make your backup windows (instead of all the host side copying).

Duh. There is a reason these products exist.

Re: Old News...

[Zontar The Mindless]

-10, Gratuitous XKCD Link That Is Not Actually Relevant To Story

Re: Old News...

[Zontar The Mindless]

"Great news, everyone! You're all fired!"

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

What has that to do with DevOps?

Re:Old News...

[ganjadude]

well, we only got the backups back online today!!!!

Everybody has admin rights

[PPH]

p.And this is what happens. How about dev, test and production accounts or environments? And a QA process in place to make sure code from the devs works properly before promoting it? And scripts prepared (and tested) for every maintenance and operation function. So nobody has to log on as admin and fat-finger anything on the command line.

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

I worked as a DevOp quite often recent years.
As a DevOps I never had access to prod or preprod.
You must have a wiered idea what a DevOp is actually doing.

Recent grad given full production access

[DukeLinux]

What even marginally competent IT manager would give an inexperienced person the ability to modify or delete production objects? I feel sorry for this guy, but his management is completely at fault here. We know the story. Nothing will happen to them and one or more will likely get promoted.

Re:Why Was He Mucking With It In The First Place?

[Darinbob]

Lack of procedure is a major problem. People who want that may not really be experienced, or as much as they think they are... Lack of organization is a major flaw...

Re: Old News...

[KGIII]

I gotta stop trying to type on a tablet.

Anyhow, I'd not want to fire anyone, without more information at hand. Firing people has large costs associated with it.

How to describe this?

As stated, I am retired. I've been retired for over ten years. Back in my day, we trained people and paid for them to continue their education. Don't laugh, that really used to happen.

So, I'd probably try to salvage this as a teaching moment. I'd also be changing someone else to the lead position. If I were to fire anyone, it might be the person who enabled this to happen and probably wouldn't be the junior.

Our shop was kinda laid back. It is unlikely that I'd fire anyone, but someone is going to catch a whole ration of shit. If this is a habit, then someone is getting fired, but if this is a habit then it speaks to a larger problem.

I'd start with the junior dev and work my way up. I'd keep digging until I found the problem in the process. I'd then take time to listen and find out what we need to do to make sure it never happens again. There's gonna be a meeting. Hell, there are going to be several meetings. Chances are, this is going to have a bunch of little teaching moments.

Junior devs, autocorrect hates that word and thinks it should be debs, sure as hell shouldn't be able to push to production. Even if they could, those backups damned well better be timely and functional. The IT staff, I guess you call them ops today, were great at their jobs, thankfully. For reasons, we had a pretty robust backup methodology. I am still flummoxed when I hear stories about not backing up properly.

I haven't done this for a while, so it is time to do it again.

I used a computer because I had to. I actually kinda hated them, for the longest time. I am not a programmer, but I programmed because I had a task that needed to get done. I'm not an admin, but I did that job because it also needed to get done.

Then, I was able to hire capable people. I was able to hire competent developers and IT staff. It was those people who did much of the heavy lifting. I learned a lot from them. I learned what best practices were and why they were that way.

So, as I said, I've not taken the time to do so lately. Here's a tip of the hat, and a nod, to those folks in the admin side and in the developer side. Here's a tip of the hat to the database admins and to the jerks who secured my network. Here is a tip of the hat to the programmers and to the QA. Here's a tip of the hat to those who spent long hours beside me, enabling me and teaching me.

Re: Old News...

[KGIII]

I owned the company. We were also pretty relaxed. I'm not sure we'd have used those titles, but I'm pretty sure someone is getting demoted - and it's probably not going to be the junior developer. Hell, I might have taken Mr. Junior Dev out to eat, or given them a small bonus - for having discovered the glaring hole in our process that enabled them to do this and for demonstrating that our backups were worse than useless.

Re:Why Was He Mucking With It In The First Place?

[sootman]

> The info on the worksheet was the production database
> with an account containing full write access.

Then whoever wrote the docs should be fired. You should NEVER write docs with actual credentials. Not only should they be fake, they should be DESCRIPTIVE and fake, like YOUR_NAME_HERE and YOUR_PASSWORD.

Re:Why Was He Mucking With It In The First Place?

[Just Some Guy]

But too much procedure is also a major problem. I've been in shops where I had to hold a serious of meetings to use open source library A to accomplish a task because an unrelated group already started using open source library B several years ago (even though no one uses B anymore because A is clearly better in every way). "It's important that we standardize everything!" was the logic given. We were in a very competitive industry and competing with shops that weren't spending their days in endless meetings to "build consensus" on every single little detail of every task. Sure, you can't allow total chaos, but it's probably OK for the engineering department to start using Flask even though the marketing department also has some Django deployments.

At one place, I got in several arguments with a manager who insisted I should be using an IDE (when I was already using Emacs). Him: "We'd be more efficient if everyone was using a similar work environment." Me, finally: "OK, I'll be happy to help them all upgrade to Emacs. No, I'm not switching. Stop asking."

Too little procedure is bad. I think it's less bad than having too much, especially when most of it is cargo cult stuff that's only done because "that's the way we do things here" even though no one remembers why.

Don't Worry

[cstacy]

Don't worry about this guy, he got a job at Verelox the next day!

Bad analogy

[s.petry]

If you hire someone to clean your house and they break an expensive vase on their first day, do you say they should have had a more senior cleaner shadow them for a while?

Or do you expect them to arrive with the basic skills they claimed to have during the interview?

If I hire someone trained as a professional driver to drive my car and they head at 120mph down the wrong side of the freeway totaling the car and causing untold damage to other cars, you bet your ass they would be fired.

Even that however is not a good analogy, because I'm supposed to have a guy in charge (CTO) of ensuring drivers can actually drive. I'm not going to speculate further on the size of the company because it's not relevant. The CTO is mentioned as firing the guy, and the CTO should be fired right along with him. The guy in charge of backups should be fired also (also mentioned in the story).

I can relate

[Anonymous Coward]

I can relate to this. While it wasn't my first day, I was working as an admin when my experience is by far more on the dev side. I ran database scripts against the production system, and was later told I was doing it wrong.

Fortunately, they did have backups for most of it, but I singlehandedly took down the production system for a day. Multi-billion dollar system. Despite attempts to clarify what I had done wrong, I could never get instructions on how to do the task "right". I was told how to avoid doing the same thing again, and that sufficed, but was still a PITA manual hack.

First the exec responsible, then maybe the CEO

[uncqual]

If the scenario is as described (which, frankly, seems somewhat unbelievable - why would production DB access credentials be floating around in a document available to a junior dev?), there is at least one senior executive who needs to be fired immediately. Perhaps more than one such executive needs to go as there are at least TWO serious and obvious problems - (1) the DB access credentials being published where a junior dev could even find them and (2) not having a tested and working restoration/return to service plan. If different executives are responsible for these two epic fails, they should all go. Perhaps the CEO should also go (although, that may also be the owner of the company -- in which case of course they would be unlikely to fire themselves!) as she has very bad judgement in hiring and does not exercise appropriate oversight.

Damn

[DontBeAMoran]

That's the second time today this happened! Won't somebody think of the databases?!

Re:How the fuck-The Cloud.

[Aristos Mazer]

Well... maybe. There's this other story on Slashdot today... https://tech.slashdot.org/stor...

The CTO should definitely go.

[Afty0r]

I actually read the article. There are many reasons why the CTO is incompetent, but one is that THE USERNAME AND PASSWORD FOR THE PRODUCTION DATABASE WERE IN PLAINTEXT IN A PIECE OF WIDELY DISTRIBUTED DOCUMENTATION. That alone is utterly crazy. For the sake of making this post more positive, here's a few things the company could/should have done here: 1] Don't have credentials in documentation, they should be in a password vault. 2] PCs on the developer network should have ZERO access to a live environment, ever, even if they have the correct credentials 3] Have working backups, test them regularly. Have backups online and offline in multiple locations 4] Have a release manager or similar code-review or sense check *all* changes which could be deployed to live

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

I've done a couple DevOps roles myself, both for startups, one small, one ~300 people.

There are security controls, logs etc. But DevOps has the power to create and destroy production and the tools to monitor all relevant metrics. Everything necessary for troubleshooting is at their disposal, and in practice that can mean shell access.

Seriously, if you have a different method, post a link to something. We all get blindsided by industry developments, if there's a different way, I'd like to know about it.

e.g., "DevOps should include production support" https://devops.com/supporting-production-applications-devops-way/

NO QA ?

[OppMan29]

I guess thats the price to pay for not having QA dept test code in lower

Dissimilar situation

[trewornan]

I won't state the company but a similar accidental deletion of a huge number of records from a database occurred at a company I was working for. This was a DBA who'd been working there for a considerable time, unfortunately when he realized what he'd done he edited the logs to try and hide who was responsible. Naturally he got fired but (at least from the gossip I heard) the principal reason for firing him was the log business - whether he would have been fired for the original mistake was much discussed but nobody really seemed to know the answer. Incidentally - it took nearly 12 hours working completely through the night to figure out exactly what had happened and restore from backups, although there was still some data loss even though we had pretty good back up provisions.

If your IT systems can't recover.....

[modmans2ndcoming]

If your IT systems can't recover from catastrophic failures like this then you suck as a CTO If your development practices don't involve integration testing then you suck as a CTO If you allow a new employee with no experience to commit changes to production then you suck as a CTO

Re: Old News...

[pikine]

About finding the right person to fire, I suspect that this is an unfortunate but complex choreograph of events such that no single person is at fault, and everyone is doing their jobs.

Junior programmer did the right thing about unit testing: wipe the database between tests to ensure consistent result no matter the order the test cases are run. Senior programmer did the right thing and wrote detailed documentation. The management needed a paper trail for production credential, and the documentation is probably the only SOX compliant way the company has, so the management did the right thing and stored production credential in a SOX compliant way. Production backup is not working but someone is actually working on getting it to work, and they are having trouble with replication and consistency; they want data integrity assurances rather than having backups that could have silent data corruptions.

Everyone is doing their jobs in good faith, and probably doing it well. I think the disaster could have been prevented if someone looked beyond their job responsibility and noticed potential problems, but such busybody could readily cause ire in a hierarchical organization. I think the best thing a company can do is to encourage people to be nosy and speak up. The CEO has to make this part of their corporate culture.

Re:How is it reckless?

[lucm]

Maybe. Or maybe the company has a "live in the fast lane" culture where all resources are focused on growth, and the "security" policy is to tell new hires to follow instruction and not mess with production until they know what they're doing. If you think that kind of company doesn't exist, you need to beef up on your Silicon Valley history.

But really all I mean is that firing executives is not a good response to disasters.

"First day"

[slipped_bit]

"EXPLAIN YOUR ACTIONS!"
"Um... It's my first day?"
"HAHAHAHA, OK, carry on then."

A backup isn't a backup

[Shatahn]

A backup isn't a backup unless you've practised restoring from it. Giving a newcomer privilege to production services straight away is also silly. Is this another one of those hoax "Ansible deleted all my user's websites" stories?

Unix and databases have mechanisms for access cntl

[Douglas Goodall]

In the old days, large unix machines had serious system administrators and a general mix of users and developers. The root password was closely held and users were given access on the basis of their need. There is no doubt that fully utilizing user group and world security at the os level, and granted accesses to data in databases, is a way by which users and developers can be given appropriate access. Back at the Western Bancorp, we had a production mainframe and a test mainframe. Quite an expensive scenario to manage the creation and migration of new code releases through QA levels. These days an extra test machine costs significantly less, sometimes the cost is negligible. In this circumstance, the company has bigger trouble than just a lost database. I would think this company needed a database manager whose responsibilities included assurance that the data will not be lost. I can only hope this was not a publicly traded company.

Re: Old News...

[Quirkz]

Is posting the obligatory like doing the needful?

Quacks Like a Hoax

[StikyPad]

This story has all the hallmarks of a hoax. It's semi-plausible on its surface, but how many companies would have leaked production credentials in a config document AND had production machines accessible from a dev network AND had backups without any integrity testing, let alone multiple redundant backups AND had zero incidental copies? That's violating a slew of helpful best practices but none of the potentially harmful ones, like locally mirroring production data for development use. This sounds more like someone's worst fears of their first day presented as fact, but the details don't add up. It involves a strange blend of strict adherence to some policies with rampant noncompliance to others, and incompetence by a lot (nearly all) of the existing employees, managers, and executives. When things fail big, it's because of a series of events in corner cases, not

Re:Quacks Like a Hoax

[StikyPad]

...hitting submit before you finish composing your post. :)

Throw under the buss

[dougg76]

I have seen this in much larger shops. They build their systems in a non robust way because they are cheap and just throw people under the buss when it goes bad. Companies try to implement the cool side of continuous dev without paying the dev ops price. It is just the same old same old, lets demand the good parts of a paradigm, but ignore the requirements to make such a system viable ex: continuous development, agile etc.

Having IT work on brittle systems just offsets stress to the devs / techs and makes their jobs suck. I once brought this up to a manager and he said something along the lines "Those guys who work on skyscrapers don't work with harnesses..." they do, and have been for a long long time. It is not the f'ing 1930s.

Re: Old News...

[RockDoctor]

I'm not sure we'd have used those titles, but I'm pretty sure someone is getting demoted

NEW HIRE : "Hi, I'm a new hire. Where do I go?"

RECEPTIONIST : "Hi, I'm the Vice President in Charge of Avoiding a Repeat of The Disaster of May 2017. Follow me, and do what I say, and not what I do!"

NEW HIRE : "I just bet there's a war story behind that job title."

Re:How is it reckless?

[david_thornley]

So, what do you do with the CTO that doesn't involve firing? All other appropriate responses I can think of are illegal.

What happened here is that a junior dev wiped the production database and there was no usable backup. That indicates gross incompetence in more than one area. If the CTO was even slightly competent, this would not have happened.

Fishy

[DarthVain]

Not only exactly what you said, but the fact that they have no backups (or they were not working) is also pretty scathing. I've worked with some experienced developers that know our systems for 15 years, and they never ever have any access to production! Never mind they should have DEV and TEST instances as well, probably logging depending on what they are using...

Makes me wonder if their IT was that inept, or if there was some serious financial or other more fundamental technical issues that they needed to go away, and this guy is simply the scapegoat... LIke the CTO was embezzling from the system and needed a reason for all evidence to disappear. Or someone senior with real access screwed up very badly, and again this guy is getting the blame. Seems pretty unreasonable and unrealistic that something like this could even happen. Anyway it all sounds very fishy to me, I'd bet there is something else going on.

Re:Why Was He Mucking With It In The First Place?

[david_thornley]

To be precise, I need read access to the prod database to do my job. I don't WANT write access.

Re: Old News...

[david_thornley]

If everyone is doing the job they're supposed to do, and a disaster like this happens, fire the person who assigned the job roles.

The Developer is Clearly at Fault

[lagunastarman]

Clearly, the Developer is at fault. He should never have accepted to work at such a screwed up company; a dumb CTO (who blame shifts), poor security, no offsite backups, bad HR, terrible policies. ... When you start your first job you are expected to know EVERYTHING about what you are getting into on the first day of your first job ... yeah, right! My serous advice is for this guy to get a lawyer and sue the hell out of the company!!! I know IT Directors that have transferred from one career to another and not been granted admin access for 6 months - even after extensive background checks. In this case, the one most innocent is the brand new, new hire.

Re:Why Was He Mucking With It In The First Place?

[Excelcia]

The interwebz is so lucky to have you, who are so wize in the wayz of the webz.

Re:Why Was He Mucking With It In The First Place?

[lucm]

It's one of those persistent management misconceptions: let's have small identical lego blocks that can be swapped around and that all have the same needs. Then they wonder why productivity is so low.

I had this discussion with a PM recently. They were complaining about a project that kept missing deadlines and that had low quality output. I said: you need a rockstar like such or such developer. The PM said; oh no we don't want to have different skill sets across the team, it's too difficult to reassign tasks.

The problem with PM is the same as with HR. People who have jobs with no complex expertise requirement tend to think all jobs are easy and that it's all a matter of man-hour or budget.

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

DevOps have/should have nothing to do with production.
They usually are the "admins" of the test environments, hence the name: a mixture between operators and developers.
Most DevOps are actually having 2 or 3 roles: developer, tester, admin. If the organization is advanced, they might "prepare" Docker templates etc. for production, but they don't work on production servers. If that was the case you called them admin or operator.

Proxcimate cause, root cause

[eric_harris_76]

The proximate cause was the new hire who screwed up on his first day.

But the root cause was a lack of control over their shop. This company was a disaster waiting to happen. The newbie was guilty of walking too fast past a house of cards.

As others pointed out, it should not have been possible for him to do that.

What are developers -- novices or old hands -- doing with that kind of access to production data? A company big enough to have someone called a CTO ought to have separation of authority, so those sorts of things can't happen.

He was a rookie who made a mistake with serious consequences. The alleged professionals already on staff made it possible for one mistake to perhaps destroy the company.

I'm guessing it's not a publicly traded company subject to Sarbanes-Oxley.

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

The smaller shop I worked in had a very hands-on CTO who would manage testing and test automation, in the larger shop we had a QE/QA department which would manage testing and test automation.

In both places, there was no room for pure operations or admins. They're not needed when the sprint ends, tests are signed off and DevOps deploys the code into production. Admins aren't qualified to troubleshoot issues when they're caused by automation errors in the DevOps toolchain.

Some legacy operations work would fall into the hands of DevOps, but part of their job would be to automate it away.

Nuts and bolts infrastructure, racking servers and difficult-to-automate stuff like networking and firewalls are an exception, but the DevOps model works better for cloud providers where all of that can be expressed as code.

The Dev part of DevOps didn't mean that they were developers of the application codebase, but that they were using developer tools and methods to create infrastructure as code. We tried some of the DevOps sitting in on developer scrums and feeding back on code, or Developers sitting in on DevOps scrums it made people feel good and respect eachother's roles a bit more, but very little actual code came out of it.

In what you describe, it sounds like DevOps was contributing automation components to a more traditional Operations team.

In what I've experienced, DevOps would write the puppet manifests and organize the infrastructure such that when executed, it would produce running machines with dev, test, staging or production code and data, complete with monitoring, appropriate backups and tools for measuring performance and health.

Sounds like a Scapegoat...

[goolac]

Too many issues here to miss. 1) New employee with access to LIVE data. 2) Copy/Pasted code... (implies company provided the code...) 3) Conveniently mentioned backups not working 4) Mention of Legal action. This poor sap was hired to be a scapegoat. They probably had issues meeting production goals or some sort of milestone and it was a convenient happening that a new tech guy broke the whole damn thing. Unfortunately it is hard to prove who knew what but. In my opinion the CTO should be the one who legal action should be taken against. The company should protect the new hire and isolate him from the CTO and immediate supervisors to investigate the job/code/work he was doing and who told him to do what. Also they should investigate who gave him access to LIVE data and Admin level controls of that data. Although I suspect that the this was a crime against the shareholders/investors and the whole management team should be held accountable. Just my $.02

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

Good post, I try to keep it in mind it one asks me again what DevOps are actually doing.

My DevOps were restricted to providing infrastructure to the developers and testers, not for production.

Sounds familiar...

[iq145]

https://tech.slashdot.org/stor...

Re: Old News...

[tom.apostolou]

As a database expert I have to say, no backup means there is no database, no security and access control (developers should never have access to production) no database, no high availability solution means your database is not critical. So yes everyone else except the junior developer should be fired, if anyone should be fired.

Re: Old News...

[Zontar The Mindless]

Good to see you posting again. And thanks for posting a cogent response to my feeble attempt at a funny.

Re: Old News...

[KGIII]

You did make me giggle but my verbosity is an innate trait. It's what I do. I did have some sorta personal issues, including a new love interest, but I am alive. I had a sibling pass away, pretty much at the same time someone decided to ruin my motorcycle by smashing into it while I was going the opposite direction. I don't know if you saw the pics, but it was kinda ugly. I survived but I must have thought I was Superman. I don't recall it, but I pretty much tried to punch said car with both hands. The car won.

The idiot that pulled into my lane, to pass someone else, wasn't prosecuted. I am okay with that. Putting them in jail would help nothing. Insurance covered it all and I am actually looking at a new bike. I love my BMWs but a friend has Sportster for sale.

If I can figure out how to get pics of the x-rays, I'll share them. It was just my hands and my right wrist that were screwed.

And then a sibling died. And that is kinda where I went off the rails. I can't say I didn't have a good time, but it was not really something I needed to do. At least I didn't get any new tattoos.