Developer Accidentally Deletes Production Database On Their First Day On The Job (qz.com)

An anonymous reader quotes Quartz: "How screwed am I?" asked a recent user on Reddit, before sharing a mortifying story. On the first day as a junior software developer at a first salaried job out of college, his or her copy-and-paste error inadvertently erased all data from the company's production database. Posting under the heartbreaking handle cscareerthrowaway567, the user wrote, "The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that I 'completely fucked everything up.'"
The company's backups weren't working, according to the post, so the company is in big trouble now. Though Qz adds that "the court of public opinion is on the new guy's side. In a poll on the tech site the Register, less than 1% of 5,400 respondents thought the new developer should be fired. Forty-five percent thought the CTO should go."

Comments

How the fuck

[Gojira Shipi-Taro]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding. The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Re:How the fuck

[Anonymous Coward]

The entire CXX staff level should be let go. Why a person fresh off the street had permission to even make a mistake of such magnitude is beyond me.

Re:How the fuck

[Anonymous Coward]

Additionally if the backups aren't working, that's something that could have saved the company - if the CTO had made sure that they worked properly.

I'd probably think twice before hiring the new guy as well over this, although at the same time I'd probably limit access to just what he needed to do his job. I'd also make sure I have proper backups.

But I'd avoid the CTO at all costs, at least as a CTO. (OR C- anything if this is how he operates.)

Re:How the fuck

[Anonymous Coward]

easy. it is entirely FICTION. it's reddit. this same shit's been done there before, and it shall be done there again.

congratulations, you've been bamboozled, along with most everybody else.

Re: How the fuck

[Anonymous Coward]

Completely agree, he never should have been given those sort of permissions and should have been checked by several different groups (sysadmin, infosec, lead devs, etc.)

Re:How the fuck

[Anonymous Coward]

It's in the article. He wasn't directly given full access. The company's getting started guide happened to use production login information in the examples.

Pretty dysfunctional, yes.

Story may be real or fictional.

Re:How the fuck

[blindseer]

How the fuck does a new hire have that kind of access?

Having worked at some small businesses before it seems to me to be pretty common. The article said the business had a couple hundred people at most and 40+ developers. Quite likely the people there were there for a long time and they hired a handful of people since they now had a need for some help and finally some money to pay them. This is how things were done for years and not remembering what can happen if a newbie fucks up once in a while they thought nothing about handing over the documentation they used themselves, and never really having to test the documentation against a newbie before they didn't consider the consequences of what might happen if the doc was followed to the letter and/or with a minor mistype like inputting the production servers instead of the developers servers.

The lack of a proper backup is inexcusable. I learned that in college after having lost the only copy to a semester project a week before the end of the semester. After that anything I deemed "important" I kept a backup where it was not likely to be deleted accidentally.

I worked for a larger organization doing some development and I was never given access to the production servers. Once I could prove that I had something working I would hand my files over to a manager and he'd copy the files over himself. This was always a bit of a tense moment because, due to budget constraints for the project, I was using MacOS/Apache/MySQL/PHP for testing while the production servers were Windows/IIS/MSSQL/PHP. It seems everything was close enough because I didn't hear anyone complain about broken code. If something didn't work then it would have been very time consuming to roll back and also very difficult to debug.

This also makes me think a bit on why developers, even newbies, would have access to the production servers. If something broke between the transition between the development system and production then they'd want it fixed right away. This would be easier to do if the developers had access to the production servers. This would be avoided if the production and development systems were identical but like in that project I mentioned it might be that budget constraints prevented them from doing so.

Re:How the fuck

[davester666]

HR is responsible for what? Hiring someone who makes a mistake? Geez, might as well not have a HR department at all...

But the CTO and other people in charge of IT should be fired or demoted for failing to handle the most basic issue, how to handle the production data being lost. This isn't the only way, perhaps a significant electrical spike takes the data out, or a hacker breaks in and deletes or corrupts it all.

Once you actually have a "production" setup, you MUST have a backup (well, backups over time) that must be periodically checked and tested. Even just having a very expensive backup setup, but never testing that you can actually get back everything you need from it, is worthless.

Re:How the fuck

[lucm]

The entire CXX staff level should be let go.

First, "CXX" level is not "staff". Second, firing the entire senior management team because of this incident would be completely reckless. I understand the outrage but that's not how companies work in real life. The last thing a company in that situation needs is more instability.

The proper way to address this is to stabilize the situation, then make sure the problem cannot occur again. And this typically doesn't simply mean firing people, because odds are that there are cultural or organizational factors that made this situation possible (crazy deadlines, shoestring budgets, etc.), and those would probably lead the replacement to making the same kind of mistakes down the road.

What is needed is new processes and controls. You start with a simple governance framework (like COBIT maybe) where each part of the IT ecosystem is linked to a specific business leader, then you let each of those leaders make sure that their area of responsibility is well managed from a risk perspective. That's how you make the company more resilient, not by firing people who maybe were not empowered to make the right decisions in the first place.

Re:How the fuck

[Anonymous Coward]

Let go of what? If the DB is gone with no backup, THERE IS NO COMPANY. It will probably take them a little while for this to sink in.

Re:How the fuck

[ma1wrbu5tr]

The trainer, the CTO, the Managed Services Director, and whoever let that doc with the PROD database URL out into the training materials. Those are the people I would hold responsible.

Re:How the fuck

[sexconker]

Additionally if the backups aren't working, that's something that could have saved the company - if the CTO had made sure that they worked properly.

I'd probably think twice before hiring the new guy as well over this, although at the same time I'd probably limit access to just what he needed to do his job. I'd also make sure I have proper backups.

But I'd avoid the CTO at all costs, at least as a CTO. (OR C- anything if this is how he operates.)

I reconfigured our backups recently. It's a huge pain in the ass because I had to wait for the end of the month, move everything to other storage, recreate new jobs from scratch, wait for them to run, copy to a second location, and perform automated validation checks in both locations, then do it all again (making sure version chaining worked), then actually go in and open up every single new backup, in both locations, to ensure they all worked, the correct passwords were used, etc.

But I know they fucking work.

Re:How the fuck

[Anonymous Coward]

The lack of a proper backup is inexcusable. I learned that in college after having lost the only copy to a semester project a week before the end of the semester.

So what happened? Did you manage to re-do the project in time? Must have been especially hard, with all the other things you had to do near the end of the semester.

Re: How the fuck

[Anonymous Coward]

No person in the company should be able to do that without peer review. Backups should work and be tested. Clearly he's not the one to blame !

Re:How the fuck

[dbIII]

What is needed is new processes and controls

Yes, but I think what people are getting at here is that current policies appear to be so fucked up that the team who implemented them may be unlikely to implement new processes and controls that are any better than they have already done.
I've seen that sort of stuff in a few places, (notably government owned corporations - worst of both worlds) where management have been chosen for reasons other than ability.

As shown by this incident that sort of mismanagement is often self-correcting, but it's a shame that so many others have to go down with the ship when the place goes under.

Re: How the fuck

[Anonymous Coward]

So he was given a getting started guide that contained full access info, maybe without even stating that it was prod and risky to play with. It's even worse than being explicitely given full access. For all he knew, he was in a playground (that's what most companies use to train their new hire)

Re:How the fuck

[dbIII]

The article said the business had a couple hundred people at most and 40+ developers.

Not exactly a small business deluded sig guy.

Re: How the fuck

[Anonymous Coward]

Agree, a new hire should not have access to the production db.

Re: How the fuck

[Tomahawk]

Agreed. Has they said it was a month in then maybe it could be believed. But onlyâ day one you're only getting down around, here's your PC, here's your password, Joe will show you around the product later and explain what we do, the toilets are that way...

On day one, there'd be no production access. Nor access to source code. And if he did have that of the was access, and changes would be scrutinised.
So, yeah, it's a fake.

An interesting hypothetical scenario, though -- if it did happen, given the info available, then who's to blame?

The new dev? No: first day, fresh out of college, mistakes should be expected. He shouldn't have had that level of access, and his immediate superiors should have been keeping a close eye.

His immediate superiors? Possibly. How did they let someone new make such a big error? Why did they allow him to do anything at all?

The DBAs? No working backups on a production database? No transaction logs that could be rolled back? No DR solution in place? Basic stuff here that were all missed. So definitely some blame sits her.

CTO? Hard to say. Certainly policies should be in place to ensure this shouldn't happen, so why did it under their watch? Were the staff too overworked that they didn't have time to get the basics right?

To properly sign any blame, more information is needed. We don't have all the facts. To many questions remain unanswered.

There is a general human flaw that this story highlights, though, which is that the majority will sign blame based on too few facts. This comes up time and time again, both here and elsewhere. Take, for example, "Making A Murderer" -- how many petitioned to have Stephen released based on 10 hours of a TV show that really only showed one side of the story? There were more than 200 hours' of testimony to try to show all of the available facts for the jury to make a decision.

This is a similar case -- one reddit post describing things from the point of view of developer who was on his first day ever working. There's tonnes of missing information here.

Still, and interesting scenario to ponder as a thought experiment.

Re:How the fuck

[Anonymous Coward]

Not uncommon in small business. Early in my career I got a job to that started out as hardware design, which was fine. Then they decided my next job was "make some simple modifications to some software". My training was in electronic engineering, not software, which was just a hobby for me at the time It was written in Java - which I had never used - it controlled an entire building's lighting, a/c etc, and there was no testing, just upload and hope. It introduced me to a lot of new stuff, which was cool, but it's a crazy way to run a business.

Re: How the fuck

[h33t l4x0r]

I can guarantee there was no DBA. A copy+paste error that can't be rolled back *or* restored from backup means that nobody is on the ball over there.

Re: How the fuck

[Anonymous Coward]

Many fucked up things here, 1- dev work on production database (instead of test env) 2- extensive db permissions for a new guy 3- no disaster recovery drills (you realize that your backups do not work only when it's too late). The company should be grateful to the new guy, he made them realize that their IT department is shit.

Re:How the fuck

[Anonymous Coward]

notably government owned corporations - worst of both worlds

Thanks! That explains it!

Re: How the fuck

[Anonymous Coward]

Hospitals in Norway managed to outsource IT operations to India and provide full root-access in production for training purposes. Of course all CEO-driven BS, but their intermediate "IT provider company" (whatever that is supposed to be) had to take all the blame.

So it can be worse.

Re:How the fuck

[Calydor]

If it wasn't a pain in the ass it wouldn't be called work.

Re: How the fuck

[ColaMan]

I read the thread over on reddit and basically he turned up, was given his dev machine, and then given the instructions for creating the dev environment + db's & etc on his machine.

Then after a while he was like, "OK, let's see, step 17: Enter these commands to create your dev DB."

So he copied and pasted the commands from the document, like he'd been doing for the last 16 steps.

Unfortunately, the commands had the production db's connection in them. And they also had queries like, "drop database mainDB; create database mainDB; create table etc etc"

Whoopsie!

Re:How the fuck

[Anonymous Coward]

Don't be an arsehole with the "CXX" not being "Staff". If you work for the company, you are simply staff / employees. You know exactly what he meant.

Re:How the fuck

[tlhIngan]

Well, depending on the size of the company, it could be possible. Maybe not if you're a 1000 person enterprise, but if you're a 10-100 person SME, it's definitely possible, especially a startup.

However, the lack of backups is more damning - it means the entire company is one mistake away from losing it all. I don't care if it's a new guy - it could be the rockstar developer making a typo and deleting the entire database. It could be anyone.

Even though I work for a company of less than 100 people, no one other than the IT guy has full permissions. Even me as the backup IT guy I don't have full permissions. I've been offered, and I always refuse it - if I don't see a need to have it, I don't want it. (It basically evolved down to what I should do - and the vast majority of what is needed involves users, so access to the domain controller isn't required (about the only thing I'd need it was for resetting locked out accounts, but it resets after 5 minutes anyways, so my response is "go for a coffee and it'll be fixed").

Even then, I'm pretty sure the only access he has to delete all the files is via the file server console - his user may have read only to everything, and read-write to what he needs, but not full read-write for obvious reasons.

But we have backups, too, and because it's all shared files, we have volume shadow copies enabled, so even if you do delete it all, there's a snapshot of it from a few hours back. How many asses this feature has saved, who knows? It's saved mine several times (not that I'd tell anyone :) ).

And that's why everyone believes the CTO should go - it's not hard to protect, and have backups and ensure they all work. If a newbie could delete the production database, so can anyone else. Perhaps the rockstar developer has an oops with a mouse and deletes it?

Accidents happen, and there is no excuse for not having basic, simple insurance policies (permissions, backups, etc) in place and operational. I say the shareholders should hold the management team to account for this - something so basic should not be possible.

Re:How the fuck

[William Robinson]

What is needed is new processes and controls. You start with a simple governance framework (like COBIT maybe) where each part of the IT ecosystem is linked to a specific business leader, then you let each of those leaders make sure that their area of responsibility is well managed from a risk perspective. That's how you make the company more resilient, not by firing people who maybe were not empowered to make the right decisions in the first place.

Well, that is what they have been paid for so long. If processes and controls are not in place, to avoid this kind of disaster, the leaders have clearly failed their duties. I might agree that entire staff need not be fired, but it is for sure that blame goes to somebody higher up sitting on his ass, instead of the poor kid.

Re:How the fuck

[geekmux]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding.

This has little to do with noob status as an employee, or even technical experience. The real question is why the fuck a developer has access to the Production system. We call the Non-Production environment Development for a fucking reason.

The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

The CTO should get the shitcan for not ensuring backups were working, as well as not implementing proper security policy that prevents developers from fucking around in the Production system without assistance and a documented approval process.

Regarding HR, they're fucking clueless at this technical level, so don't even really know what could be pinned on them. Their involvement started with processing payroll and ended with him signing the HR handbook, which probably doesn't have jack shit to address this particular fuck-up. That would be IT policy and procedure.

Re:How the fuck

[Z00L00K]

Everyone screws up now and then, most often small, sometimes big. There's sometimes a small screw-up where you accidentally reboot the wrong server. Once is just an oops. It's when people screw up frequently and tries to get away with it you should bury them somewhere safe or remove them. If they admit their mistake then it's better to work on salvaging the pieces and patch together the remains or recover from a backup.

Companies that don't have backups that they test - they are toast as it doesn't even have to be a mistake that causes the need for a backup, it can as well be a hardware failure or thunderstorm.

Good Employees Make Mistakes. Great Leaders Allow Them To. From the article: ...As John Wooden once said, “If you’re not making mistakes, then you’re not doing anything.”...

Re: How the fuck

[Anonymous Coward]

If the company goes down because of this CTO can expect to be in direct contact with legal representatives of what is left of the company as they are resp. for policies and they obviously did not follow them or did not have them - not working backups usually means no verification (unless some other hick-up occurred which it may be). Letting greenhorn to delete production DB is rather silly too. Whether greenhorn will have to hire a lawyer is another thing. I would say it is possible if they want to pursue the legal route. But if CEO has any brains he will kick CTO arse if not to fire then to introduce working process at least in backup area.

Re:How the fuck

[Interfacer]

When I started as the new systems engineer in our company, I was hired to become the guy in charge of the process control infrastructure running the entire pharma plant. Keep in mind we were already running operational for tens of millions per year. I was taking over from the consultants who built the place (during a transition that took a year or two iirc). First real day on the job, he created my user account, granted me enterprise domain admin and full application / db admin rights and spoke the immortal words: You're a full admin now. Don't break anything.

Re:How the fuck

[tburkhol]

The proper way to address this is to stabilize the situation, then make sure the problem cannot occur again.

You're expecting rational behavior from a business? Good luck. /s

I'd add that the size of the business probably matters a lot here. If this happened in a shop of 20 people with 3 computer guys, one of whom is "CTO," it's a very different situation than a billion dollar, multi-site company. It's probably smart to send the new guy home in the immediate aftermath - he's not going to be helpful in the recovery and emotions will be very high. After that, bring him back in, sort out what happened, and how to prevent similar failures of process. Dude has the cost of a fuck-up burned into his brain, and he's going to be a lot more careful and deliberate in the future.

Now, if he does it a second time...

Re: How the fuck

[JWW]

How the heck did the copy/paste text include valid credentials to even access the production database?!?

Re: How the fuck

[Anonymous Coward]

I can guarantee you that this is not always the case. I have been at companies to help integrate some technology and in many smaller shops you can easily get access to the production network if you really wanted (often there is no separation at all). If the instructions happen to include the required password then I can easily see it happening without doing anything malicious.

Re:How the fuck

[Anonymous Coward]

In short - because the superuser credentials were in the developer on-boarding document.

Re:How the fuck

[Anonymous Coward]

For you.

Re: How the fuck

[war4peace]

I'll tell you how. (names below are fictional)
Netadmin or Sysadmin team has no content writer, so Prakash is told to create an induction manual for new hires. Prakash hates this assignment (understandably, he's a net admin, not a fucking content writer) so he does the bare minimum: takes screenshots of prod environments and enters credentials for that generic admin account everyone was using (because fuck processes) as example.
New hire comes, proceeds going through the document, and with his lacking attention to details (or because he's overzealous, or been told to beat the best time for new hires of 30 minutes 15 seconds) doesn't pay attention to the step saying "enter YOUR given connection details" and copy/pastes the ones shown as example, which incidentally are absolutely valid and point out to the PROD DB.

Disaster happens.

Moral of the story?

1. Get a proper content writer to create documentation: responsibility falls to management to ask for it and senior management to approve it. Blame falls on whoever didn't ask or didn't approve it.
2. Review the documentation: responsibility falls to line manager Prakash is under. Blame that line manager and hang him (figuratively) from a tall sturdy branch.
3. Publish the document as a controlled document in a knowledge management environment. If such an environment doesn't exist, blame everyone who didn't ask for it or approve it.

The CTO is hardly to blame, it's not his business to handle such processes, that should fall under a line manager or whichever dedicated person was supposed to handle it.

I personally wouldn't have kept a netadmin/sysadmin who can't follow basic instructions or a manager who didn't review the training document. Everyone else is off the hook because they were either not aware of the risk or did what they could with a task that wasn't in their job description.

Re: How the fuck

[Cesare Ferrari]

Agreed, i've worked places where stuff like this is possible.

Remember, if the place was a windows shop, and the DB is, say, SQL Server, then the permissions to do dangerous stuff may just be group membership of the user account in AD, so you could get production read/write access without typing a password, just by connecting to a db server.

I've worked on linux systems where the production access is locked to a given user, but it was common to sudo to this user for certain tasks. If the user in question isn't called 'prod' or something like that, but say, named after the product, it's not clear what permissions, and in fact, damage you can do.

So yes, it's possible, and i've been places where this can happen. Actually, sitting at my desk at work, I can screw the system in many ways without typing a password. I can delete binaries, or promote something broken to production, or screw up network configuration, or well, just endless ways really. The upside is that I can fix stuff quickly when it goes wrong, and each company needs to evaluate the risk vs reward tradeoff to determine what is right for them. When the stuff I produce goes wrong, only the company is harmed, and they have no external users who would be aware. If I were working on some web facing system with millions of users, clearly a different approach is appropriate!

Re:How the fuck

[Anonymous Coward]

I don't see what the big deal is. Just restore the backup.
OK, you will have some loss for the time between the backup and ... oh wait, the backups didn't work.
You know, screwing up the backups is a bit more severe that deleting the database.

Re:How the fuck

[drinkypoo]

The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Yep. When I read this:

"The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss.

My first reaction was yes, yes they will, and the CTO is the one responsible for that kind of stuff, and the one who will get sued if anyone does. And if his boss has any sense, he'll be sacked, too. If you haven't verified that your backups are working, you are worse than useless. You are a cancer that must be cut out before everything goes wrong, as it has here. That company is probably going to die now, and it's going to die the death of the shit CTO.

Re:How the fuck

[Anonymous Coward]

This also makes me think a bit on why developers, even newbies, would have access to the production servers. If something broke between the transition between the development system and production then they'd want it fixed right away.

I'm thinking that ideally you want to roll back to the working production environment and then go back and figure out why the transition didn't go smoothly.
You should also figure out how to change the development environment so that bringing it to production goes without problems in the future.

Re: How the fuck

[Anonymous Coward]

That's a great point.

If the company isn't destroyed by this, nobody should get fired. Instead, it's a learning opportunity for all involved. Get everyone into a room and run a root cause analysis to figure out what happened, what the root causes were, and what measures will address those root causes.

It's dangerous expecting perfection from everyone -- especially front line workers and supervisors. If you're requiring perfection as part of your business model, it's time to change tack.

Re: How the fuck

[rgbatduke]

I mostly agree with your remarks, but I think there is a question of scale. I've done several startups, and in a startup, one person is often the CTO (that would have been me, twice), the primary developer of the core software (ditto), and the system manager for the entire network of computers owned by the company, which on occasion have been pretty much my cluster of computers plus eventually a "company owned" server or two when we had enough capital or cash flow to afford them.

From this level of "my basement plus your basement" startup, there are a set of scaling steps that lead through VC incubators to getting actual VC to hiring a skeleton staff (cheap, quite possibly fresh out of school and paid in part with options or the prospect of options) to making money but not as fast as you burn it to making money (one hopes) faster than one burns it and moving on to fame and fortune and early retirement.

At intermediate steps in this process, IT is not the polished gem that it might be for a fully developed and capitalized and profitable company. Backups might have been set up by the original founders (and done correctly) but all it takes is one hire that necessarily is given serious responsibility with little oversight in a tiny startup who doesn't completely understand backup to make a small change that fucks it up without even realizing it. I've been around a long time, and trust me, it happens, and if you are LUCKY you discover it when some tiny unimportant file is overwritten and you try to restore it and can't.

If this was a startup in the stage just past profitability -- small but important database of actual customers and/or their data, team of maybe three or four IT people still wearing many hats, so the database person was also the systems manager and the primary software developer was also the web manager and the original CTO/developer/general factotum was distracted by the demands of sales and corporate politics with a board made up partly of the VC people who want rapid growth to a liquidity event and so on, this scenario isn't that unlikely. DB person says I'm going nuts adding all these new workstations to sales desks, CTO says let's hire a DB person, CEO/COO says we can't afford a $150-200K position right now (duh!) so CTO says we'lll hire some kid straight from college with DB chops, kid comes in, they show kid the DB and sit him or her down, he/she tries a few tentative SQL commands but has no experience with whatever actual DB they are using and has credentials that are more talk than substance, tries to make a copy of DB to play with without breaking original and fucks it up. In the meantime, they have been backing up to a RAID, and the RAID started throwing errors but the same DB person who left the kid to play and orient themselves had JUST STARTED the rebuild or was trying to fix the problem and rerun the failed backup when...

Life is a comedy of errors like that more often than one might think. Most times they are non-fatal, but every now and then the ENT surgeon slashes the carotid the first day of his/hir first surgery and the patient dies on the table, or the kid working on steel 200 feet above the ground loses their balance on their fist day trying to emulate people who have been doing it for years and walk a narrow beam over nothingness. Sometimes people can get back up on the horse that threw them and sometimes they end up living under an overpass or dead at their own hand.

Humans are highly error prone information processing systems. We deliberately design systems that are critical -- as much as we can -- to have multiple levels of mutual auditing to catch and prevent errors before they occur, but it simply isn't possible to idiot-proof every process, and accidents can happen even to those who are not idiots. Few are the people who have root privileges who have NEVER EVER entered rm *.junk in some directory, but accidentally entered an extra space before the ".junk" (blush, been there, done that, done WORSE than that). If you are the only systems man

Re:How the fuck

[Anonymous Coward]

Agree...plus anyone in charge of backups. The sole point of backups is to recover from such an oops quickly. And if it is as simple as a copy and paste error then it can happen to experienced ops as well.

Re: How the fuck

[TWX]

No person in the company should be able to do that without peer review. Backups should work and be tested. Clearly he's not the one to blame !

I really could see him being fired from the job. If I were a mechanic and I accidentally broke some important active piece of machinery on my first day I would not find it unreasonable to be fired.

Now, that out of the way, there's a whole host of other people that built the conditions that allowed this to happen that should also be fired. That CTO that suggested the Legal department should be involved is among them, and HR should take exception for how he threatened a new-hire for his own mistake.

Re:How the fuck

[Anonymous Coward]

That explains the "anal" section of YouPorn.

Re: How the fuck

[mixed_signal]

"The Staff" or "staff level" certainly does refer to the C level in many tech companies.

Re:How the fuck

[Ol Olsoc]

What is needed is new processes and controls.

What is needed is a backup system that actually works, and is used.

I never trusted our official backup system, having seen it not work on several occasions. So I installed one of my own for the group. Sure enough, around a year later, a group member called in a total panic - she'd written a script that was supposed to perform a find then print the find results. What it managed to do was delete the whole database.

Calls me in a tearful freakout - the IT folks backup didn't work.

Mine did. There were a few things here that needed addressed. She should have been working on a copy of the database, not the main one. Shit does happen, and if she accidentally deletes a copy due to a derpy moment, its no problem at all. Fix the script, and work on a new copy.

Second is don't ever trust a backup that you can't personally verify will work.

Third is don't be afraid to have backups for backups for backups. People called me Mister OCD or Monk (referring to the television show about a detective with serious OCD issues) but after that the little jabs mysteriously went away.

Re: How the fuck

[Anonymous Coward]

You're giving people way too much credit.

At my first job out of college it wasn't day one but it was less than a month before I had access to the backups of all the research data. I'm not sure if I had permission to delete it and I certainly didn't try but I was definitely in a position to cost the company a lot of money. This was a big company with a lot of smart people. My job was solitary, mostly unsupervised and easily f*d up.

Hah.. before that, as an intern, I almost destroyed the company's server. (don't store liquids in the server room) I don't think they had backups.

Re: How the fuck

[mixed_signal]

Sorry, ensuring business critical data is correctly backed up is obviously critical. No excuses pass muster here, as it's a well known disaster waiting to happen from any number of causes.

Re: How the fuck

[mixed_signal]

Exactly. Backups are business critical.

Re: How the fuck

[Anonymous Coward]

If you hire someone that seems like they know what their doing they should know what their doing

Re: How the fuck

[Defakto]

As a manager and supervisor I wouldn't put you out on the floor without a senior to ensure you could handle and to QA the work. The first 6-12 months are hour learning and trust building phase. I expect mistakes during this time and I expect your peers to work with you on new projects to ensure those mistakes don't happen .

Re:How the fuck

[Anonymous Coward]

Exactly. This is fake news.

Re: How the fuck

[Anonymous Coward]

I did a contract job years ago where they had me work on a mission-critical production database. In MS Access.

On the second day they had me go in and delete some tables. I believe I followed their instructions to the letter, and they didn't give me any feedback on it at the time, but that night I got a call from my agency saying, "don't go back tomorrow."

I never did find out what the reason was. Did I delete the wrong data? Was I too slow, or not skilled enough? I certainly hope I didn't screw up their system, but they bear the responsibility to make sure the data and the process are secure. I was "just following orders."

Re: How the fuck

[Anonymous Coward]

Do you know how big this company is? How much room they have in padding their bills to have spare staff to do that sort of training and hand-holding?

If you hire someone to clean your house and they break an expensive vase on their first day, do you say they should have had a more senior cleaner shadow them for a while?

Or do you expect them to arrive with the basic skills they claimed to have during the interview?

Re:How the fuck

[Anonymous Coward]

I'll tell you how: Business "Transformation"

That's the new code word for "fire as many employees as possible to get the stock price up. Outsource everything you can."

End result is no training budgets, all experienced staff have walked out the door, and you're left with a skeleton crew that has minimal experience (because experienced guys cost money), and mistakes WILL happen. It ain't rocket science, but of course CxO's won't ever take the blame for this kind of shit, will they? No, they'll get millions in "compensation" because despite the collosal disasters, the stock price STILL went up because nobody cares about anything that really matters, and all the competitors have shit services because they're doing the same thing.

Re: How the fuck

[Anonymous Coward]

It absolutely is the CTO's fault. The CTO is the one at the top of the technical food chain. He's responsible. That's how leadership works.

Re:How the fuck

[tommeke100]

It just depends how big a "production database" is. We have a relational database that is maybe 100 GB big. Daily Dumps are a bit less than 18 GB and like 4 compressed. It takes maybe 15 minutes to backup which can be done at night (and even during operations really, it's not such an heavy load on the system anyway). Since the data is used often to implement new functionality or check on things, we deploy the data on test servers often so we know it works. The backups are also copied to other locations. If something drastic happens, we don't lose more than a day of work.
Of course, I can imagine big transaction systems where you only have a couple months worth of data and you need some clever process to mount and unmount the older data to make space for the new. The older data must than be kept somewhere safe yet accessible obviously.
Indeed, if the junior developer managed to wipe out the whole DB, and no fail-safe mechanism is in place, it's not really his fault.

Re:How the fuck

[Kjella]

This also makes me think a bit on why developers, even newbies, would have access to the production servers. If something broke between the transition between the development system and production then they'd want it fixed right away. This would be easier to do if the developers had access to the production servers.

Yes, 99.9% of the time that's what happens. Then there's the 0.1% where our most senior "I know what I'm doing" developer was just updating a status (not through a stored procedure, because why add another layer of indirection) and forgot the WHERE clause. To the global log table that every process is required to use. Or the house consultant (he had more system experience than better skills than most our hires so wasn't that) that was going to scratch a database on the test server and rebuild it from a script, who didn't check what server the query window was connected to. Of course these are totally hypothetical examples that never happened *cough*.

Re: How the fuck

[Anne Thwacks]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Hell, even on a one man business, development should not be done on the same machine as production activities.

Even my home machines have off-site table backups - and yes I have tested recovery works - cross platform - on different hardware and OS.

if your data is worth money, you should be prepared to spend money to protect it. If its not worth anything, then just delete it yourself and save the disk space.

Disclaimer: I do not have an MBA.

Re: How the fuck

[tommeke100]

I reckoned this was the only possible scenario. They gave him some code to look at, with DB creation SQL statements and such. And they just had the Production DB in the connection string and user/pwd.
That's basically a huge mistake of the senior people who need to train him. That's the person you need to fire. I've had to train many devs to work with our databases, and I've never given them code to access the production DB ever. Only after maybe months of working on test servers, they get to run their stuff in prodution, and that's after it's been vetted by different people.

Re: How the fuck

[Anne Thwacks]

If you hire someone that seems like they know what their doing they should know what their doing

Are you trying to justify voting Trump?

Re:How the fuck

[JustAnotherOldGuy]

How the fuck does a new hire have that kind of access?

My thought exactly, which is why I'm a little skeptical of this story. Who hands out this level of access on Day 1?

If this super-newbie was given this level of access on his first day, the CTO and HR and whoever was responsible for making backups should all be fired like a cannon.

Re: How the fuck

[JustAnotherOldGuy]

I did a contract job years ago where they had me work on a mission-critical production database. In MS Access.

Well there's your problem. FFS, who does that??

Re:How the fuck

[slappynipsy]

Testing the backups is a huge pain in the ass but in the immortal words of Don Draper, that's what the money's for!

Re: How the fuck

[Anonymous Coward]

No working and tested backup process = CTO takes the full blame. End of story.

Re: How the fuck

[subanark]

I'm a developer for Azure. The service I develop doesn't have a DBA or any test team. Could I screw things up? Yes, quite easily, however since the DBs are on XStore, we can just pick a date and time within the last week and snap the DB back to that state. I don't run a big enough or impactful enough service to warrant this level of support.

Re: How the fuck

[Anonymous Coward]

>> In MS Access.
> Well there's your problem. FFS, who does that??

I could tell you, but then I'd have to kill myself. :)

Re: How the fuck

[whoever57]

The CTO is hardly to blame, it's not his business to handle such processes

But it is his job to ensure appropriate security and backups for the production database.

Re:How the fuck

[Anonymous Coward]

Agreed. This is a failure of policies and procedures for not providing minimal access required by job role.

As well as for allowing developers access to perform changes into production without going through any sort of testing or QA environment and deployment sign off.

Re: How the fuck

[lgw]

If you hire someone to clean your house and they break an expensive vase on their first day, do you say they should have had a more senior cleaner shadow them for a while?

You know that's how it usually works, right?

Re:How the fuck

[Anonymous Coward]

The company deserved it as a "learning lesson" if a new hire can wipe a production system by mistake on their first day. Clearly they need to get some processes in place? I think so.

Company I work for may not be a perfect example of perfect processes, policies and procedures or operations of its production network, but we sure aint that stupid either.

We have a process for gaining access to prod, the new user has to request access and agree to fairly straight forward T&C that they understand this is prod, we have a change control process and you'll follow it to the letter, and there are repercussions if you're careless.

And lastly prod is isolated from the office network, you cant accidentally run something on your workstation that goes off and hoses the prod network because you forgot.

Re: How the fuck

[HornWumpus]

No _competent_ DBA is not the same as no DBA...

Re: How the fuck

[HornWumpus]

They've moved onto MySQL...

Re: How the fuck

[Anonymous Coward]

Get a proper content writer ?

If you have admins that cant write documents properly and care about it then there are probably other they are doing badly, you have the wrong people on your team.

But yes we have a doc team that writes the docs for the customers. Admins can damn well write it them selves and like it.

Re:How the fuck

[140Mandak262Jamuna]

First, "CXX" level is not "staff"

This is military terminology. The "staff" here is the General Staff, officers who form the core of administration of the armed forces. Officers who has achieved "staff" rank get to wear some special markings on their uniforms, like a red band around their caps, or red velvet under their insignia or red markings under the stars on their collar etc.

It is like the word "secretary". The secretary of the board is a senior administrative officer. These terms secretary, staff etc often refer to the low ranking as well as high ranking officers.

Re: How the fuck

[Anonymous Coward]

It was legit.

world came round from other people who worked for that company. who no longer have work.

the company fucked up completely by having the actual real admin/pass listed on the training sheet the noob was given. 99.9999% company fault.

Re: How the fuck

[140Mandak262Jamuna]

Netadmin or Sysadmin team has no content writer, so Prakash is told to create an induction manual for new hires.

What the hell? It is not some user manual for an application sold to customers. It is an internal document of procedures. Admins routinely write such procedure documents every day of the week. Often for their own references, so that they dont miss important tasks. Many admin jobs are done one a week, once a month, once a quarter, or on triggers like off boarding, on boarding, going on leave, coming back from vacation etc. They should document what is done. Often admins will history > this_session ; emacs this_session and add notes to the commands and make that the document.

Re:How the fuck

[Just Some Guy]

The last thing a company in that situation needs is more instability.

It sounds to me like they've already maximized instability, and any change whatsoever could only lessen it.

Re:How the fuck

[bigtiny]

I'm always amazed when something like this happens, a lot of people's first reaction is 'who gets fired'? It's really not a productive attitude to take in a case like this.
I find it problematic that
- companies take an agile 'quick turnaround' approach to development without seeming to understand the risks, until they get bitten. This is an example.
- seems that whoever's managing the dev team should have had a break-in period/mentoring system in place to make sure new hires (especially newbies straight out of college) know their way around before pushing production code.
- whoever's managing the place is REALLY on the hook if the data cannot be recovered! If a manager/company is not willing to provide the personnel and tools necessary to insure backup and restore integrity, then they have only themselves to blame.

Firing this kid, in this situation is stupid. In a way, he did them a favor....now they know what they need to fix.

Re:How the fuck

[theNetImp]

you apparently didn't read the article, where it says the credentials were in the onboarding documentation....

Re: How the fuck

[JustAnotherOldGuy]

They've moved onto MySQL...

And that would be a good move, if they did in fact do that. I know people deride MySQL, but it's light-years ahead of MS Access.

You might not want to run your bank on MySQL, but it's a perfectly suitable database for 98% of the kind of stuff that most companies do on a day-to-day basis. Unless you're sending people to the Moon or doing financial services, MySQL is fine. Tesla, Netflix, Youtube, Spotify, Facebook, Twitter, and the NASA Jet Propulsion Lab all use MySQL on a daily basis and they seem to be doing just fine.

If you don't like MySQL, use Postgresql or SQL Server or IBM DB2. Whatever floats your boat and gets the job done.

If you want to keep your recipe collection in MS Access, great, but for anything more than that, ewwwwwww.

Re: How the fuck

[HornWumpus]

MySQL is in a league with DBase 3+. Routine reindex.

Re:How the fuck

[Anonymous Coward]

It's very simple, if you think about it.

He may be entirely unconnected to the actual incident except as a convenient timing trigger. A scapegoat. A patsy, if you will. First day on the job, the moment someone in the office hears even the smallest "whoops", and everything is then his fault.

Perhaps the production database had already been wiped hours ago, and someone didn't want to get fired but found a FNG was joining in a few minutes. Or perhaps something far less legal's going on.

Re: How the fuck

[lgw]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Welcome to DevOps. It's the fad sweeping the industry. Yes, it's exactly as stupid as you think it is. It's so obviously wrong, so mind-bogglingly stupid that only a manager with an MBA could possibly have thought it up. So of course it's the latest fad.

Re: How the fuck

[lgw]

That's the CIO, BTW. CTOs have nothing to do with internal IT operations, their job is to guid the tech that the company creates, not uses. Well, mostly it's to tell large sales prospects that the company is working on really cool stuff. The CIO runs IT.

Re: How the fuck

[lgw]

It's a real step up from MS Excel, which is shockingly common. The huge tech company I work for keeps all it's pricing stuff in freaking Excel sheets: I blame the mandatory lobotomy that accompanies an MBA.

Re: How the fuck

[Entrope]

The "script" command is awesome for that kind of thing, as it captures intra-program I/O as well as command lines.

Re:How the fuck

[Anonymous Coward]

I reconfigured our backups recently. It's a huge pain in the ass because I had to wait for the end of the month, move everything to other storage, recreate new jobs from scratch, wait for them to run, copy to a second location, and perform automated validation checks in both locations, then do it all again (making sure version chaining worked), then actually go in and open up every single new backup, in both locations, to ensure they all worked, the correct passwords were used, etc.

But I know they fucking work.

As a software engineer, I extend to you my thanks for making sure the backups fucking work.

Re: How the fuck

[Anonymous Coward]

There is a general human flaw that this story highlights, though, which is that the majority will sign blame based on too few facts.

I agree, and I'd add that your post highlights another general human flaw; assuming that "blame" can or should be assigned to one single point. In reality, in this scenario all of those people would be at least partly at fault (arguably not the CTO, depending on whether they had set up adequate processes and oversight that had just not been followed down the line, which gets back to your point about assigning blame without facts), with the better question being how much. The only person I know definitely had some blame would be the new hire... but it's a vanishingly small amount, with a much larger amount being spread among some/all of the staff above and around them.

Re: How the fuck

[war4peace]

The fact that the behavior you're presenting seems normal to you scares me, because it shows how widely accepted this has become.
While I agree that the admin should start writing the draft, their job would end there. Technical people rarely can produce proper documentation, and whether the documentation is internal or customer-facing is irrelevant. Standards should be the same. the mindset you're presenting is similar to that of people writing in bad English online because "fuck it, it's not important, I can write good when I have to".

Writing note for tasks != procedures. Also procedures != processes. Also induction manuals != processes or procedures (albeit they often overlap, still they are not identical).

At the very least, you need a dedicated person (or team, depending on how big the organization is) to verify, review and vet all documentation (except for private notes, comments and such) and store it in a controlled, unique repository. This could be as simple as a collection of printed brochures for small companies, or a wiki, or a more complex Knowledge Management System. The person or team assigned to that bears the ultimate responsibility. The CTO (of a larger organization, of course) has better things to do than review all documentation. Responsibility is then compartmentalized.

I've seen my fair share of horrible documentation over the years. often I have to go and learn what the processes inside a certain group or team are built, in order to understand the data generators (I develop analyses and reports) as well as why that data is generated, how is it generated and what does it mean from a business perspective. It greatly helps me provide correct, helpful and thorough analyses and reports. The problem is that the documentation is most times a mess, usually a collection of disparate word and excel files of various versions circulating through e-mail back and forth like feces in a sewer. Open one and you'll be hit by a jumble of acronyms I'm sure the tech people understand, but I have no idea what they mean.

Yes, from your point of view, that little information bubble you're in, it all makes perfect sense and you're used to it. Introduce a stranger to it and they'll stare at you blankly while you're thinking "how stupid is this guy?"

TL;DR version: Isolated communities develop their own unintelligible language, while isolated dev teams develop their own unintelligible documentation.

Re: How the fuck

[Anonymous Coward]

It's a real step up from MS Excel, which is shockingly common.

Yep, I can't count the number of companies I worked at where people just kept sending workbooks to each other, making edits, and sending them back rather than keeping the current information in a database.

It's not that Access is all that difficult; I think it's that you have to build something before you can start using it. With Excel, they start with a blank sheet and just keep making edits until it's what they want.

Re: How the fuck

[Anonymous Coward]

If the company is destroyed, there is no one to fire...

Re:How the fuck

[Anonymous Coward]

The entire CXX staff level should be let go.

First, "CXX" level is not "staff".

Unless they own the company they most certainly are staff.

Re: How the fuck

[Anonymous Coward]

On day one, there'd be no production access. Nor access to source code. And if he did have that of the was access, and changes would be scrutinised.
So, yeah, it's a fake.

So, you've never worked for stupid people before? I had a supervisor and manager who would give the admin password to work experience students from a CDI-type program on their first day. I expressed concerns about that and was told I was being uncooperative. They went about creating admin ids in each OU, without a password so it would be more convenient.

Re: How the fuck

[Anonymous Coward]

Yes, of course. Middle-managers with some background in PHP who fancy themselves as "backgrounded in software engineering" are all too eager to go in for "DevOps", and frankly, let them. And let their shitty companies get ravaged when they try to build real infrastructure on unconfigured LAMP WordPress sites or whatever.

It's an industry chocked full of halfwit web developers ("front-end engineers") who mutter "muh Docker" every time an issue comes up that demands OG experience.

Fortunately, we'll be the COBOL guys of our generations.

Re:How the fuck

[Anonymous Coward]

you must be CXX staff since you seem to take so much offense to it.

Re:How the fuck

[modmans2ndcoming]

Depending on how large of an enterprise you are dealing with, personally verifying the backup of the database is functioning might not be possible.

Re: How the fuck

[Brockmire]

You either work for the city or you're in a union.

Re:How the fuck

[Anonymous Coward]

The entire CXX staff level should be let go.

That would be going easy on them. To keep this from happening again, you need to really show them you're serious. Shut down the company immediately and demolish the offices and datacenters.

Here's hoping this guy gets another IT job quickly, if he hasn't already. I know ONE thing he won't do again!

Re: How the fuck

[Anonymous Coward]

...development should not be done on the same machine as production activities.

But, nothing tests like production!

Re:How the fuck

[Anonymous Coward]

The first company I worked for had 39,525 employees.

Re: How the fuck

[lgw]

How do you think people get into the job? Some minor skill is required, and must be demonstrated. Usually one of the older women will bring someone new along with her for a little while, until she seems to get the hang of it.

Re: How the fuck

[lgw]

Facebook, Google, Microsoft, and Amazon all do DevOps to some extent (Amazon especially). So everyone else copies them. Damn all management fads.

Re:How the fuck

[lucm]

Maybe. But usually when talking about the enteprise world, C-level executives are called corporate officers, and everyone else is the staff.

Re:How the fuck

[lucm]

It sounds to me like they've already maximized instability, and any change whatsoever could only lessen it.

And you say that based on 2 paragraphs posted by an anonymous junior developer who deleted a production database without knowing it?

Re:How the fuck

[Anonymous Coward]

CTO of a similar sized company (36 engineers of ~80 people) here.

This so could have happened at our shop.

It is impossible to track the accuracy of everything. F**kups happen all the time. People who are supposed to check things, often cheat. They say they checked when they do not. Understand that being in charge of technology means dealing with people, not with the technology.
I can see how this scenario might have happened: DB guy leaving, giving his one or two week notice. He is told to write a doc for his successor once the company manages to hire him. If that doc then ends up using the real production DB info, the reviewer guy might not realize (since the reviewer would not have the real passwords, and hence not recognizing them).
If there even was a reviewer guy, or if he ever read the doc, or maybe he just did not pay attention to the example passwords but skimmed past them.

Re: How the fuck

[Anonymous Coward]

Hactually - one could argue that google are is the OG of devops.

Re:How the fuck

[Anonymous Coward]

Welcome to the real world, snowflake

Re:How the fuck

[Anonymous Coward]

>The last thing a company in that situation needs is more instability.

Ah, but you see, nobody in the forum (besides you) actually gives a shit about this company, or really, any company. It's far easier to have opinions when you don't care about the nonexistent consequences.

But I didn't come here just to say the obvious.

I also want to point out that this company is clearly incompetent for letting a day 1 hire delete everything, because the following things happened:

1) Onboarding process was clearly not defined
2) Not only was there no set of good written instructions made for this position, even if they were, well, they clearly weren't followed (or maybe this was all a cleverly constructed ruse for more media coverage? Who the fucks knows and will actually give a shit 24 hours from now?)
3) Permissions? What permissions?
4) Safeguards? Security checks? Say whaaaaaaaaaa?
5) Backups?
6) Redundancies?
7) What the fuck was this guy doing on the production system on day 1?
8) Do they even have a development system? Why the fuck wasn't it used? Even the goddamn Matrix talked about the Construct, which was their playpen. If a goddamn fucking science fiction movie from 1999 can get onboarding done currently, why the fuck can't a goddamn company that exists to sell a product/service?
9) Who the hell was training this guy? Are they fired/retrained yet?
10) Where the fuck does HR's responsibility for this shit begin?

So sure, you're absolutely correct that this company doesn't *need* more instability, but frankly, it's hard to see how they've gotten this far by anything except sheer luck before, and frankly, as long as anyone in control of making sure that the "existing" security procedures are still there, that shit's gonna continue.

Re: How the fuck

[Anonymous Coward]

>Welcome to DevOps.

Where the rules are made up and the points don't matter.

Re: How the fuck

[Anonymous Coward]

Also, hard coded user pw in the connection string doesn't sound right

Re:How the fuck

[Anonymous Coward]

Yeah, this CTO isn't deserving of his title. Devs of any level shouldn't have test access let alone production, and no one but devs should really rely on a dev environment, either. Separating dev, test and production assets via good security policies eliminates like 99% of all production outages.

Re: How the fuck

[minstrelmike]

Most mechanics have "backups" of all the tools junior mechanics will use. In a normal environment, deleting the production database isn't really that big a deal. About the same as ordering the wrong part for a repair or having the electricity shut down.

Backups are supposed to work. Shit happens. All the time.

The CTO should have had the junior devs working off backups so 1) nothing goes wrong and 2) even better, you know those backups work. Load a new one each day.

Re: How the fuck

[minstrelmike]

I remember the very first SQL stuff in the Oracle 4 manuals. First instruction was how to list out tables. The next one was how to drop them. So the first thing you do is drop the training tables you're supposed to use in the next step of the tutorial.

Whoopsie. That wasn't user error.

Re: How the fuck

[Anonymous Coward]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.

Hell, even on a one man business, development should not be done on the same machine as production activities.

Even my home machines have off-site table backups - and yes I have tested recovery works - cross platform - on different hardware and OS.

if your data is worth money, you should be prepared to spend money to protect it. If its not worth anything, then just delete it yourself and save the disk space.

Disclaimer: I do not have an MBA.

Right on. He should not have access to production database. What kind of workflow did they have set up there?

Re:How the fuck

[minstrelmike]

I'm one of those senior developers. I only delete all the data about once or twice a decade now. Live and learn.

Re:How the fuck

[Anonymous Coward]

How the fuck does a new hire have that kind of access? that's not even enough time for on-boarding. The CTO should definitely get the shitcan, as should anyone in HR involved in that debacle.

Agreed. The new hire is nothing but a scapegoat for bad security and backup/restore processes. Except for the lawsuit threat, the poor guy is better off not working there.

Re: How the fuck

[Anonymous Coward]

Welcome to DevOps. It's the fad sweeping the industry. Yes, it's exactly as stupid as you think it is. It's so obviously wrong, so mind-bogglingly stupid that only a manager with an MBA could possibly have thought it up. So of course it's the latest fad.

Don't blame the devops.

I've been in higher Ed my entire career, I've been a "devop" before there was such a title. Never enough resources, ie running on a shoestring. Developers call me a sysadmin, sysadmins call me a programmer. I lock things down as tight as I can get approved by management. Sometimes tighter if I can get away with it. My manager tells me to give a new employee anything other than very minimum, I would laugh at him.

Working for higher ed, student staff are "cheap labor", You need to watch them like a hawk because you know given the chance, they will mess something up. Eventually the good ones rise to the top and you trust with "some" production tasks.

Someone needs to lose their job, but don't scapegoat the new guy.

Re: How the fuck

[lgw]

They may have started the fad, but they at least turn important stuff over to pure Ops people. Unlike Amazon, where a dev makes a typo in a command line and takes all of S3 down for a day.

Re: How the fuck

[lgw]

There's a real difference between DevOps and "small company, so I wear both hats". The latter is a forced error, the former insanity.

Re:How the fuck

[martinfb]

Also, the IT staff that failed to implement and verify backups needs to go.
PLEASE be sure these idiots are put on the national "IT offender" list!

Re:How the fuck

[sh00z]

Don't be an arsehole with the "CXX" not being "Staff". If you work for the company, you are simply staff / employees. You know exactly what he meant.

Quite possibly not. I work for a medium-tech company that gives all sorts of authority to the IT and Procurement organizations. Unfortunately, all of the accountability is assigned to the Science and Engineering teams. Not having to be responsible for their own f@ck-ups has created a culture where these groups work without oversight, and project managers have to pad schedules about 40% for the inevitable "you wanted a cable--what's wrong with this perfectly good rope I ordered?" moments.

Re: How the fuck

[Anonymous Coward]

I really could see him being fired from the job. If I were a mechanic and I accidentally broke some important active piece of machinery on my first day I would not find it unreasonable to be fired.

This is more like pressing a fictitious "drop all car lifts immediately" button, which is right out in the open without any access control, and looks just like the fictitious "order pizza" button that is right next to it.

Oh, and both are unlabeled.

Do not look into laser with remaining eye.

Well, at least your sig would seem to carry the same logic. That it is the fault of the employee rather than the fault of upper management to properly secure the device.

Re:How the fuck

[mongothesecond]

Someone on the management team is accountable to the board for disaster recovery and other functions including data backup. How is not safeguarding corporate assets and intellectual property not a management level responsibility?

Re: How the fuck

[thesandtiger]

I worked for a start up that handed all new developers the keys to the kingdom on day 1, regardless of experience level.

On day 1 you were expected to deploy code changes to production (minor stuff - usually adding yourself to the about our team page) AND republish the production DB w/backup.

It was every bit as stupid as it sounded, and yes, in the very short time I was there before I got the hell out of dodge, at least 2 or 3 times the site was taken down completely by a day 1 newbie fucking this process up. At least they had backups, though.

Re: How the fuck

[Anonymous Coward]

He was a developer not a systems administrator. There is NO reason why he should even be allowed access to production data, let alone live production data.
Hell, even on a one man business, development should not be done on the same machine as production activities.

Yep ain't really the new guys fault but you see this all the time developers working off of the live data. I seems here that coders are held in such high God like regard when the truth is below writing code developers are dangerous with systems and data security.

Your also right developers should never have the right to run DROP TABLES; on any data base not even the test units. Why is so hard to take a backup copy of the production database and dump it on a test DB server especially in these days of VMs. Lazy? or just plain stupid. This process also tests your backup data. If you backup doesn't import into the test server you better check your backup plan you got problems and problems like in the article of "the backups didn't work" are also found and fixed.

Disclaimer: I do not have an MBA.

The reason you understand this stuff and still have a brain and common sense.

Always wanted to name my kid Jimmy DROP TABLES Smith

Re:How the fuck

[lucm]

Only the CEO is accountable to the board. The CTO is accountable to the CEO. The VP-IT (or whatever) is accountable to the CTO. And down it goes until it gets to the idiot who couldn't follow instructions. Where do you draw the line? The executive who derives priorities based on the overall business strategy? The manager who is supposed to enforce policies? Risk management people who didn't foresee or implement safeguards? Team lead who didn't train or supervise the newb properly?

This is a witch hunt and it won't bring back the lost data. The blame game is never the best way to move forward. Processes and controls, that's how you tame the beast.

Re: How the fuck

[easyTree]

No person in the company should be able to do that without peer review. Backups should work and be tested. Clearly he's not the one to blame !

Perhaps the new-hire was also be responsible for putting their backup strategy in place? :D

Re: How the fuck

[cthulhu11]

Lemme guess -- you consider yourself a developer and look down your nose at "ops"?

What's stupid is segregating "A&E" and "ops". The former fiddle while Rome burns, like they're working on a thesis, pisisng away person-months on deadends, LLD' s that defy the laws of physics, and wheel reinvention. The latter are left to get the service to actually run, so they do their own dev in a vacuum.

What's also stupid is dedicated scrum masters who enforce dogma despite reality, counterproductive stuff like planning poker and droning on about how everyone on the team should be able to do any job.

What works is having a variety of complementary people in each team, providing perspectives and cross-pollination. May of us were doing this years before anyone invented the term DevOps.

Re:How the fuck

[NockPoint]

Where are my Mod points when I need them? CTO should be out the door. +5

Re: How the fuck

[Anonymous Coward]

Technical people rarely can produce proper documentation, and whether the documentation is internal or customer-facing is irrelevant. Standards should be the same. the mindset you're presenting is similar to that of people writing in bad English online because "fuck it, it's not important, I can write good when I have to".

I am technical and I am one of the rare ones who has cared to write decent documentation for companies. I wrote "Faxable solutions" back in the day. I have created guides, with screenshots when it was beneficial.

I made some mistakes early on. I found that I had to edit screenshots to take out the names of production servers, production databases, usernames and anything that could be construed as a password - real or example.

No one taught me about any of this and there was very little review or oversight and I think that is where the real problem is.

I don't think firing this person or that solves the problem. The PEOPLE who made mistakes ALL have something that could be learned and passed on from incidences like these. Firing someone just leads to a bunch of people who were guilty by association remarking about the stupidity of the person who got canned.

Had some superior listened to the guy he probably would have said something like, "I followed the instructions exactly." Then this person could get the person(s) who wrote and/or maintain the documentation together with some Techs and they could work together to figure out what happens when the document is followed "exactly". Problems would likely be found, hopefully corrected, everyone involved benefits, no one gets fired.

I have experienced it both ways. As I noted above, I made mistakes early on - those were handled correctly... but it did not always work out well.

In fact, not documentation related but once I was fired because I could not write a program which took information produced by drafters and turned it into input for machines on the manufacturing floor. I did it three times already so why not a forth?

The CFO spent close to a million dollars on the wrong machine (could only produce half of that we needed it to) so I could only program it to produce half of the parts on that line. Firing me allowed him to cover up his mistake a little longer because there was no else to write the program. (Small company, I was the only programmer/IT/Network tech/phone tech/support tech onsite.)

That was not the best decision for the company. I got another job.

Why Was He Mucking With It In The First Place?

[Frosty Piss]

Well... Maybe a brand new developer shouldn't be mucking about with the Production Database in the first place?

But what about daily backups? Maybe twice daily? Was the the database replicated over load balanced servers? No? Can't have been all that big a web site / app...

Re:Why Was He Mucking With It In The First Place?

[sanosuke001]

He was given a worksheet with information regarding how to clone a test database onto his workstation. He ran a script when printed out the test DB's URL and login information but he just copied the info off the worksheet instead. The info on the worksheet was the production database with an account containing full write access. He overwrote the production database with the test data he was attempting to clone.

Whoever gave a new hire a worksheet with a URL for the production server and login with full access should be the one fired. This new kid showed the company that their new hire setup is completely insecure and that their backup infrastructure doesn't work. It's not this kids fault.

Re:Why Was He Mucking With It In The First Place?

[charlesbakerharris]

Mebbe read the article?

Re:Why Was He Mucking With It In The First Place?

[Anonymous Coward]

Well... Maybe a brand new developer shouldn't be mucking about with the Production Database in the first place?

From the sound of things, the onboarding document contained creds for an account with RW access, and instructed new hires to use that to dupe prod for use as a dev environment. That's hilariously irresponsible.

Re:Why Was He Mucking With It In The First Place?

[Excelcia]

He wasn't mucking about with prod, per se. He was following the instructions he was given for creating a sandbox copy of prod, and those instructions at one point had an example login he was supposed to replace, and that example login was a valid admin login for prod. He used the example login, admitedly by mistake, and ran the scripts that were supposed to scrub the database and give him an empty sandbox copy of the prod db structure to play with. Those scripts, however, because of that bad example login, ended up running on the real prod with actual admin access. Small mistake on his part. Huge mistake on the documentation's part.

Despite the number of catastrophic failures of the system, they fired him. Likely so the CTO could go and claim it was malice or vandalism on the part of the new hire, instead of owning that they had documentation that included admin prod access as their example.

What I find is interesting, is that despite what this company has done to him, he has never once named them. This is an example in loyalty, professionalism and discretion that is rare today. I would hire him in a heartbeat. I want that kind of professionalism.

Re:Why Was He Mucking With It In The First Place?

[lucm]

Having production databases that can be reached from developers workstations is always a bad idea. They were lucky, in a way, that the developer deleted the data and didn't just alter it slightly.

Firewalls, people, they're not just there to block wordpress exploit scanners. Danger can come from insiders, and that includes hostile employees as well as clueless morons.

Re: Why Was He Mucking With It In The First Place?

[Anonymous Coward]

It's not real. That's why. It's just another reddit misinformation thread.

Re:Why Was He Mucking With It In The First Place?

[Kjella]

Having production databases that can be reached from developers workstations is always a bad idea.

Welcome to DevOps ;)

Re:Why Was He Mucking With It In The First Place?

[Darinbob]

Welcome to startups. It's very good experience straight out of school to know just how screwed up startups are and to avoid them for the rest of the career.

Re:Why Was He Mucking With It In The First Place?

[Just Some Guy]

We've had very different experiences. The startups I've been at have been packed with smart, competent, diligent coworkers. I've had much worse luck with large corporations where employment was effective for-life and it was so easy to blameshift that no one ever got fired for anything.

Re:Why Was He Mucking With It In The First Place?

[Anonymous Coward]

Not to mention - the very premise of just copying production database for use in testing is already terrible.

Re:Why Was He Mucking With It In The First Place?

[Shadyman]

Having production databases that can be reached from developers workstations is always a bad idea. They were lucky, in a way, that the developer deleted the data and didn't just alter it slightly.

You'd also have to pray they did not alter it any further.

Re: Why Was He Mucking With It In The First Place?

[Anonymous Coward]

I agree. Instead of being scared this kid should revert back to cto with this.

Re:Why Was He Mucking With It In The First Place?

[lucm]

Not to mention - the very premise of just copying production database for use in testing is already terrible.

In theory, yes. But honestly, I've seen countless situations where there was no realistic alternatives given the available resources, especially if the volume or distribution of data could have an impact on the system. For instance, testing a new address autocomplete feature for an order form; if you only have a few hundred clients, maybe you can generate some decent test data, but if you have millions of customers it can be very difficult to make sure that the feature will work with the real data set. Same goes for specialized reporting (e.g quantitative analysis features in a trading platform).

There are a few interesting data obfuscation tools out there if data privacy is a concern, but they can be quite labor-intensive to set up. Sometimes an ironclad NDA and a copy of real prod data is the only available solution.

Re:Why Was He Mucking With It In The First Place?

[Darinbob]

Yes, but with startups you can have brilliant people, rarely, but almost always there's little coordination as procedures haven't been set up yet. Plus a constant non-ending rush to meet short deadlines, with demos needing to be done often and no time to slow down and take stock of what's going on.

Re:Why Was He Mucking With It In The First Place?

[No Longer an AC]

Having production databases that can be reached from developers workstations is always a bad idea.

I don't disagree but sometimes it's impractical with smaller teams especially if that team consists of only one person and they are also expected to provide support for production systems.

It's just VERY important to make sure you're aware of which system you're on.

The worst I ever did was delete all the inventory in an entire aisle in one of our customer's warehouses. Thank God I didn't delete the entire warehouse. I felt really bad as I called up the warehouse manager and told him about it.

He took it surprisingly well. I was expecting to be harshly chastised but he was very understanding for some reason. Possibly because they do physical inventories all the time and I just gave him a heads up on where his inventory team should focus next. Or maybe it's because I had dealt with him many times before and usually I fixed his problems instead of creating them.

That was about 20 years ago and as best as I can recall I had both production and development windows open at the same time and I adjusted inventory in the production environment instead of the development environment by mistake. I quickly realized what I had done and in a minor panic to correct it I compounded my mistake by more than 10-fold.

Since then I think most places have a change management process which involves more than a developer recklessly altering data in production on the fly without any oversight.

Even at the largest company I've worked for primarily as a developer I had full access to any production system my software ran on and there really wasn't a distinction between developer and sysadmin. Some were stronger in one area than the other, but we all did both.

Eventually they split us up and took away the developer's root privileges from all the machines forcing us to use sudo instead...and we grumbled a bit about that, but we accepted it. It really was the right thing to do.

Re:Why Was He Mucking With It In The First Place?

[lucm]

Eventually they split us up and took away the developer's root privileges from all the machines forcing us to use sudo instead...

sudo bash

Problem solved!

Re:Why Was He Mucking With It In The First Place?

[JaredOfEuropa]

Some people thrive in that kind of environment. Working under pressure on something that really matters to the company, and with fluid roles and a lack of procedures, you have a lot of freedom as well as an opportunity to make a meaningful and unique contribution to the product as well as to the way the company will do things in the future. I know I perform better in such an environment than in the corporate one, with entrenched interests, highly compartmentalised roles, stifling procedures and generally terrible middle management. Sure, there's downsides to each environment as well as advantages. Personally I found a nice middle ground by working on innovation in larger organisations.

Re:Why Was He Mucking With It In The First Place?

[Anonymous Coward]

This.
You hand someone a list of instructions that deletes the database, and then get mad when they delete the database?

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

When DevOps exists, DevOps has responsibility for what's in prod.

The guys staffed, trained and assigned to cover emergencies 24x7x365 and answering that call at 4:59pm on Friday should be the only guys who can mess with production code.

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

"Sometimes an ironclad NDA and a copy of real prod data is the only available solution."

When you're getting executives to sign off on risk and only handing the data to specific named trusted developers, then it's not "just copying production databases into testing". It's a calculated risk by the executives, as informed by their staff.

Most of the time, when a developer complains about not having access to prod, the request disappears when I tell them to get their boss to go to their boss and pitch the case. They usually find a scrubbed set of data or another method to reproduce.

Shoulder surfing operations (with operations hands on keyboard), often solves things too, but everyone hates it. It has a huge benefit though in that operations needs to know what the developer is thinking as much as the developer needs to know whats' going on in production.

Re: Why Was He Mucking With It In The First Place?

[mixed_signal]

Wrong conclusion. Rather, avoid startups staffed with inexperienced people.

Re: Why Was He Mucking With It In The First Place?

[mixed_signal]

Or at 2am Sunday...

Re:Why Was He Mucking With It In The First Place?

[Anonymous Coward]

> has done to him, he has never once named them. This is an example in loyalty, professionalism and discretion that is rare today

That's what I would do too, when I was posting a fake story to reddit.

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

What has that to do with DevOps?

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

I worked as a DevOp quite often recent years.
As a DevOps I never had access to prod or preprod.
You must have a wiered idea what a DevOp is actually doing.

Re:Why Was He Mucking With It In The First Place?

[Darinbob]

Lack of procedure is a major problem. People who want that may not really be experienced, or as much as they think they are... Lack of organization is a major flaw...

Re:Why Was He Mucking With It In The First Place?

[sootman]

> The info on the worksheet was the production database
> with an account containing full write access.

Then whoever wrote the docs should be fired. You should NEVER write docs with actual credentials. Not only should they be fake, they should be DESCRIPTIVE and fake, like YOUR_NAME_HERE and YOUR_PASSWORD.

Re:Why Was He Mucking With It In The First Place?

[Just Some Guy]

But too much procedure is also a major problem. I've been in shops where I had to hold a serious of meetings to use open source library A to accomplish a task because an unrelated group already started using open source library B several years ago (even though no one uses B anymore because A is clearly better in every way). "It's important that we standardize everything!" was the logic given. We were in a very competitive industry and competing with shops that weren't spending their days in endless meetings to "build consensus" on every single little detail of every task. Sure, you can't allow total chaos, but it's probably OK for the engineering department to start using Flask even though the marketing department also has some Django deployments.

At one place, I got in several arguments with a manager who insisted I should be using an IDE (when I was already using Emacs). Him: "We'd be more efficient if everyone was using a similar work environment." Me, finally: "OK, I'll be happy to help them all upgrade to Emacs. No, I'm not switching. Stop asking."

Too little procedure is bad. I think it's less bad than having too much, especially when most of it is cargo cult stuff that's only done because "that's the way we do things here" even though no one remembers why.

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

I've done a couple DevOps roles myself, both for startups, one small, one ~300 people.

There are security controls, logs etc. But DevOps has the power to create and destroy production and the tools to monitor all relevant metrics. Everything necessary for troubleshooting is at their disposal, and in practice that can mean shell access.

Seriously, if you have a different method, post a link to something. We all get blindsided by industry developments, if there's a different way, I'd like to know about it.

e.g., "DevOps should include production support" https://devops.com/supporting-production-applications-devops-way/

Re:Why Was He Mucking With It In The First Place?

[david_thornley]

To be precise, I need read access to the prod database to do my job. I don't WANT write access.

Re:Why Was He Mucking With It In The First Place?

[Excelcia]

The interwebz is so lucky to have you, who are so wize in the wayz of the webz.

Re:Why Was He Mucking With It In The First Place?

[lucm]

It's one of those persistent management misconceptions: let's have small identical lego blocks that can be swapped around and that all have the same needs. Then they wonder why productivity is so low.

I had this discussion with a PM recently. They were complaining about a project that kept missing deadlines and that had low quality output. I said: you need a rockstar like such or such developer. The PM said; oh no we don't want to have different skill sets across the team, it's too difficult to reassign tasks.

The problem with PM is the same as with HR. People who have jobs with no complex expertise requirement tend to think all jobs are easy and that it's all a matter of man-hour or budget.

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

DevOps have/should have nothing to do with production.
They usually are the "admins" of the test environments, hence the name: a mixture between operators and developers.
Most DevOps are actually having 2 or 3 roles: developer, tester, admin. If the organization is advanced, they might "prepare" Docker templates etc. for production, but they don't work on production servers. If that was the case you called them admin or operator.

Re:Why Was He Mucking With It In The First Place?

[Dr. Evil]

The smaller shop I worked in had a very hands-on CTO who would manage testing and test automation, in the larger shop we had a QE/QA department which would manage testing and test automation.

In both places, there was no room for pure operations or admins. They're not needed when the sprint ends, tests are signed off and DevOps deploys the code into production. Admins aren't qualified to troubleshoot issues when they're caused by automation errors in the DevOps toolchain.

Some legacy operations work would fall into the hands of DevOps, but part of their job would be to automate it away.

Nuts and bolts infrastructure, racking servers and difficult-to-automate stuff like networking and firewalls are an exception, but the DevOps model works better for cloud providers where all of that can be expressed as code.

The Dev part of DevOps didn't mean that they were developers of the application codebase, but that they were using developer tools and methods to create infrastructure as code. We tried some of the DevOps sitting in on developer scrums and feeding back on code, or Developers sitting in on DevOps scrums it made people feel good and respect eachother's roles a bit more, but very little actual code came out of it.

In what you describe, it sounds like DevOps was contributing automation components to a more traditional Operations team.

In what I've experienced, DevOps would write the puppet manifests and organize the infrastructure such that when executed, it would produce running machines with dev, test, staging or production code and data, complete with monitoring, appropriate backups and tools for measuring performance and health.

Re:Why Was He Mucking With It In The First Place?

[angel'o'sphere]

Good post, I try to keep it in mind it one asks me again what DevOps are actually doing.

My DevOps were restricted to providing infrastructure to the developers and testers, not for production.

"Their backups weren't working."

[Chas]

Okay, the guy fucked up ROYALLY.

It happens. And he SHOULD get in a bit of trouble for it. That's how you learn "don't do that". I don't think they deserve to lose their job though.

The CTO and all the people in charge of the backups need to be on the street YESTERDAY though. That the dev COULD do something like this is a major fuckup on their part. They simply didn't have their production system locked down properly.

The fact that their backup system was non-functional is double-plus unforgiveable. The dev is merely the highlight for their massive cluster-fuck of a setup.

Re:"Their backups weren't working."

[Just Some Guy]

Okay, the guy fucked up ROYALLY.

I don't think he did. I actually RTFA this time, and the guy was following the onboarding directions he was given. Where it went south was that he copied-and-pasted the wrong database credentials. He was supposed to use the username and password that a command had spit out, but he instead used the ones from the onboarding docs.

I'll pause for a moment to let that sink in.

Some jackass had put actual prod root creds in the onboarding docs, then gave them to a new graduate fresh on his first day of his first job, then walked away while he onboarded himself without supervision.

This poor kid did absolutely nothing wrong except misreading some instructions. The engineering team responsible for the chain of events that led to this colossal fuck are completely and wholly to blame.

Re:"Their backups weren't working."

[lucm]

The fact that their backup system was non-functional is double-plus unforgiveable.

In my experience, continuous SAN replication is often to blame for a poor backup strategy. It creates the illusion of security - yes, your DR site is synchronized with production within seconds or milliseconds, but guess what, mistakes are also replicated.

Replication -> floods, fire and similar disasters
Backups -> oops my bad

Both are needed.

Re:"Their backups weren't working."

[Anonymous Coward]

The entire company should be fired and cease to exist. You should also be fired, chazz.

Re:"Their backups weren't working."

[ooloorie]

The CTO and all the people in charge of the backups need to be on the street YESTERDAY though.

I think the company is hopelessly lost. It's not only the CTO that screwed up but the CEO who hired him. At that point, you run out of people to fire, and the company just goes out of business.

And he SHOULD get in a bit of trouble for it.

He should thank his lucky stars that he found out so quickly what a poorly run company had hired him.

Re:"Their backups weren't working."

[Chas]

He's a new guy. And the onus is on him for attention to detail.

However, he's a new guy. Fuckups are to be expected.

I've had fuckups at new jobs myself. This is how we LEARN.

And yes, in that environment he was set up to fail.

Re:"Their backups weren't working."

[Chas]

The entire company should be fired and cease to exist. You should also be fired, chazz.

Thanks! Now I can go home and catch up on my sleep! Later!

*Burnout noises*

Re:"Their backups weren't working."

[Anonymous Coward]

Yea.... NO!

In working in a highly integrated environment that I do.... dozen different vendor apps, maybe 2 dozen different databases spread across 3 different database servers.... There is positively NO WAY this guy would have had access to ANY production system, beyond being logged onto the Domain to check his email day 1.

Maybe by the end of the week, he'd be able to have read-only access to the databases, but even if his job was a dba, or programmer, NO ONE gets THAT level of access to a production server, much less production databases.

This is an environment that didn't have safeguards in place for their production operations. This falls on whoever designed the instruction document he referenced, and everyone up the chain that allowed it to happen.

My guess? This is a one stop software shop, that is resting on its laurels getting by on 'just enough' IT, to do what it does. In other words, there TIME WAS UP, and they got bit in the ass, and deservedly so.

Re:"Their backups weren't working."

[Anonymous Coward]

The entire company deserves to go out of business. I realize that there are some innocent people working there, not in tech or management, and that really sucks but that's kind of the point. Mistakes like this cost big time: customers, employees, families, investors. But the investors aren't innocent either because they invested in people who have no idea how to run a company properly.

After RTFA-ing I do have a bit of sympathy for the employee, assuming his account is accurate (for all we know the 'some jackass' specifically told him which creds to use - which would not exonerate the company by any means but would implicate the hire to a slightly greater extent), but I have a hard time letting him off of the hook completely. Yes the kid made one mistake while the company made several. No backups, no disaster recovery plan, inconsistent docs / procedures, giving a new hire the ability to make that kind of mistake ... the whole ship deserves to burn ... but the new hire failed to notice the inconsistency, surely must have seen not only two different possible usernames but different ip addresses or host names as well (he said he was issued a laptop for a development environment, it could not have been the production server - so it stands to reason he had to point the install script at a specific host).

There's plenty of blame to pass around. The whole ship should go down and if the company name is ever made public customers should be aware that - if the company does manage to stay afloat - it's best to avoid them. The new hire can come out on top by learning some lessons: when you're new, ask for help and clarification if you're at all unsure about something or are given mixed messages. Never assume that management has provided you with a safety net, especially if the company is young and small. Lots of start-ups are run incompetently and incompetent managers survive by throwing their subordinates under the bus. Learn to spot them. In your future interviews, discuss the lessons you learned but focus on owning up to your own mistake(s), leave out the part about incompetent managers. In fact, if you relay every single fact but focus on your own inability to spot the inconsistent creds, your new prospective employer should spot for themselves how incompetent that company was AND your ability to own up and learn lessons and it will look very good.

Re:"Their backups weren't working."

[Anonymous Coward]

The important lesson here is: developers are fucking stupid outside of coding. Don't let a developer touch anything. They are fucking morons.

Re:"Their backups weren't working."

[Just Some Guy]

You raise a lot of perfectly valid points that I'd expect the storyteller to have caught and reacted to. That is, if it weren't their first job out of school.

I'm senior, but I've taken jobs at places where I was given a few pages of setup instructions that could've been a shell script, and I had to work through the steps one at a time. I didn't know how their environments were arranged, or what systems called "cat-kicker" were supposed to do, or why on Earth their git repos were carved up like they were, but I slogged through and got stuff going. I have enough experience to fill in the missing pieces and tell when things don't feel right.

If you or I saw "Step 3: run sudo rm -rf /", we'd stop and ask someone. But if it was literally our first day at work ever, and we didn't have the experience to know that just because someone started before you doesn't mean they know what they're doing, it might not even register. You or I see "echo 'drop table users;' | psql --host prod", we're finding a coworker to see what's up. New guy sees "echo ... | psql --host rds-32494032" instead of "--host rds-1005828322", he might not think anything of it as he plows ahead. That no one looked over his shoulder to make sure he didn't actually use the dang ol' prod creds they gave him is entirely, 100% on them.

Re:"Their backups weren't working."

[Just Some Guy]

But his mistake was easy to make and should have resulted in an "access denied" error message. If you give a five year old a hammer and say "don't whack shit with this", and they whack shit with it, you're the one done goofed.

Re:"Their backups weren't working."

[Anonymous Coward]

Agreed. The kid shouldn't be fired. Everyone in charge of on-boarding new hires and general production and backup management should be fired. This is so mind numbingly fucking stupid I have a hard time believing it's an actual real story. Sort of like reading about the Trump presidency.

They put root credentials to the production system in a new hire doc? Who the hell does that? And on top of that no one was checking the backups?

Honestly, it sounds like it was a GOOD thing that the kid was fired. With a tech department that fucked up you wouldn't want to work there anyway.

Re:"Their backups weren't working."

[Just Some Guy]

That's the modern version of "we have RAID backup!"

Re:"Their backups weren't working."

[Maxo-Texas]

A new guy may not be capable of distinguishing between details yet.

I've been hired before because a developer deleted a production library. My onboarding test consisted of asking me how I would perform the task. lol.

Re:"Their backups weren't working."

[rossz]

Yep. I have a hot standby for our production database server. Everything done on the primary is almost instantly duplicated to the standby. That would include fucking shit up, e.g. drop table foo; Which is why I make a weekly backup to disk and keep the entire WAL history until the next full backup. The backup and WALs are kept on a filer, which is also replicated to another filer.

Re:"Their backups weren't working."

[AmiMoJo]

How could someone in this position recover from it?

Suing the company for giving you in a potentially career ending position through no fault of your own might generate a one-off lump sum, but will be expensive and risky and likely make other companies not want to employ you in future.

Doing nothing and hoping they don't sue might be the best thing. Just keep quiet, never mention that you ever worked there in your job history and try to move on. If it brings the company down or they do sue and your name becomes public though...

Re:"Their backups weren't working."

[squiggleslash]

The engineering team responsible for the chain of events that led to this colossal fuck are completely and wholly to blame.

It's so over the top it's hard for me to believe any of it. If the CTO who fired him was described as a "Fat woman stuffing her face with a donut", then we'd know for sure it was just another Reddit story, but alas, no obvious signs beyond it being close to impossible to believe a department could be that stupid.

Re:"Their backups weren't working."

[Anonymous Coward]

It doesn't appears as if they were that stupid, not all of them at least.
The only two major screwups we know is the guy responsible for the backups that failed at checking that they worked and the CTO who didn't bother getting all the facts right before throwing blame and firing people.
Then there are two minor screwups. The new guy who didn't follow the instructions perfectly (But that is expected from a new guy.) and the guy who copy-pasted the production server credentials into the documentation without changing them.
The latter could be considered a major screwup but would have led to no problems if the instructions were followed and only been a minor inconvenience if the backups worked.

Of course, it could be the case that the CTO is responsible for the backups and was the one putting together the documentation. In that case we only know about one really stupid person on that department.

Re:"Their backups weren't working."

[Anonymous Coward]

Having recently gone through the setup process to get access to our QA automation Git repo, I can easily believe this. Our documentation... well, there's several sets of it, different people give you links to different versions, the versions are contradictory, the versions are _self-contradictory_, the versions sporadically use generic github addresses instead of ours, etc.

This kind of things seems to be a specialty failure mode of the Harvard MBA management style. Everyone's an interchangeable cog, right? So you can slot anyone in anywhere, give them deadlines that have no accounting for ramp-up time or experience, and drop the result in the laps of fresh college hires and expect everything to go perfectly. Hah.

So, yeah, I can believe this story, and I can also believe that the newbie peon will suffer all the blame instead of any of the management chain involved in setting up this SNAFU.

Honestly, my experience in tech companies makes me wonder how any company that isn't an engineering lead startup gets anything out the door without it randomly bursting into flames.

Re:"Their backups weren't working."

[Anonymous Coward]

If you give a five year old a hammer and say "don't whack shit with this", and they whack shit with it, you're the one done goofed.

From what I gather, that story didn't even include the bolded part. I.e. they did not tell him "careful with this login, it is production/live login".

Re:"Their backups weren't working."

[HornWumpus]

You just never mention 'one day' jobs, for whatever reason they happen.

The better question: How will the CTO recover from this?

Re:"Their backups weren't working."

[Anonymous Coward]

A haiku for something I did during my first month on the job at IBM. This was a similar situation as the story describes -- following copy and paste directions, except for this one step where you were supposed to substitute something into similary-formatted directions.

the directions said
log in as root, then to run
r m user dollar USER

On that day, I learned AIX will do anything you tell it without prompting for confirmation.

Re:"Their backups weren't working."

[Anonymous Coward]

More like they gave a five year old a hammer, told him to whack a bunch of shit, and then told him don't whack this, and the kid missed the "don't."

Why?

[Anonymous Coward]

Apparently Slashdot's content is now week-old forum posts from other social media websites. So the question is.. why should I come to this social media website any more?

Re:Why?

[Anonymous Coward]

*Ding*ding*ding*

Re:Why?

[__aaclcg7560]

That's funny. When I started reading Slashdot back in 1999 or so, it was called a website. Someone probably slapped on the "social media" label a few years ago to get some VC money.

Re:Why?

[Anonymous Coward]

You do realize that 99.999999999999% of "social media" is week-old posts...so why come to any "social media" sites at all? Why not cut out the middle man and experience reality directly? You know? Like humanity used to do....?

Re:Why?

[AHuxley]

Freedom of speech and freedom after speech.

My recording school instructor/head...

[Anonymous Coward]

...supposedly erased an entire tape's worth of recordings (the only copy) at the Nashville studio he interned at.

Bullshit

[Anonymous Coward]

I have a feeling this is a dup of the bullshit story /. tried to feed us a couple of years ago, which after some investigation turned out to be 100% pure made-up bullshit. (No, I don't feel like searching /. history, but I'm sure it will turn up later in the comments.)

Anyhow, as we all commented in the previous story: If this was possible, then the company deserves to go under.

Re:Bullshit

[Anonymous Coward]

A blank variable was passed to script which ended up doing rm -rf /{missing variable}, IIRC. That was the story anyhow.

Re:Bullshit

[Anonymous Coward]

Yeah, that sounds right.

But didn't he later admit that he didn't actually do it? (And that he just realized it was possible.)

Re: Bullshit

[Anonymous Coward]

This actually happened at work about 7 years ago. The webmin module we were using to delete accounts would execute rm -rF /home/{username}. Some support guy managed to screw something in the webmin UI and the username ended up empty in that script, and all the user folders in the network drive were deleted.

We had working backups, but it took a couple of hours to get everything back up again.

Re:Bullshit

[_merlin]

There was a script in Steam for Linux that did just that and wiped out some home directories when the variable managed to be empty. I didn't hear about it happening to anyone on their first day of work.

Re: Bullshit

[Anonymous Coward]

I passed in .. as a directory in an argument to a script to create my account in the first day, creating my own account as a privileged user in someone elses shell as a walkthrough. One of the chmod commands assigned ownership of all the users home directory on the NAS to the new account.

Permissions and script were both fixed.

bullshit click bait ...

[Hugh Jorgen]

just like the "supposed" rm -rf / last year ... And if anyone is given access on their first day to production data, eh the CTO should be whacked first.

Re:bullshit click bait ...

[lucm]

just like the "supposed" rm -rf / last year ...

Something similar happened at Columbia Internet a while ago...

http://ars.userfriendly.org/ca...

Re:bullshit click bait ...

[__aaclcg7560]

In an IT department a long, long time ago...

No worries

[Anonymous Coward]

It would seem that the junior programmer is in line for a big payout, you see firing him, using profanity and implying legal action is not permitted. The poor guy doesn't even have to hire a lawyer in some states (like California) that have a Department of Industrial Relations (Labor Board) which he's _required_ to use instead of a lawyer (unless they eventually grant him permission to pursue with a lawyer). The way the Law will view it is that the company is totally responsible for the consequences of giving him too much responsibility.

Now you can argue the Law back and forth for all the good it will do you, this is nothing novel and this has happened many many times before.

Re:No worries

[HornWumpus]

CA is an 'at will' state. He's fired and that's all there is to it, unless they talked about him in public (slandered or defamed him).

Nothing new

[Tourney3p0]

Pretty much any comment that might be posted in this thread was already posted in the original Reddit thread over a week ago. Nothing insightful or interesting will come from this being posted here now.

Re:Nothing new

[Anonymous Coward]

On the contrary, I love seeing what kind of an idiots people low ordinal /. identifiers still are.
Never signed up for /. myself, and never will.

Re:Nothing new

[Anonymous Coward]

You mean people like you who can't write a proper English sentence?

Simple question

[Anonymous Coward]

How does this developer's mistake affect anyone except the developer and the business? Why should I care that some developer make a severe mistake like that? This is not newsworthy, and nobody should care that this occurred. I suspect I'll be modded down for saying this, but surely there's something else more important to discuss than this. The moderators can't wait to censor my post, but it needs to be addressed. And yes, by definition, it is censorship. My position will be silenced because users don't want to address this important question. Those who do answer will certainly reply with ad hominem logical fallacies and personal attacks. But being modded down to -1 and attacked simply proves that I'm right and nobody has any real substance with which to dispute my point. Can anyone explain why I should care about this developer or how the mistake affects anyone else? I'm certain the answer is no, but I'll ask anyway.

"Fire you? We just spent $2M educating you"

[Anonymous Coward]

http://the-happy-manager.com/articles/characteristic-of-leadership/

A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, “I suppose after that set of mistakes you will want to fire me.” Watson was said to have replied, “Not at all, young man, we have just spent a couple of million dollars educating you.”

S/He should sue...

[YVRGeek]

for wrongful dismissal. The fact that this F'd company's CTO left their *entire operation* vulnerable to such a simple and possible blunder means he should be fired, not the new hire. Someone tell that asshat, it's 2017... there is absolutely NO excuse for what happened being even remotely possible. And the fact that the new hire was following *verbatim* instructions in the setup doc and caused the issue is beyond laughable... I would fire the CTO for extreme incompetence. Whatta douchebag.

Backups? Different passwords?

[Anonymous Coward]

Sure it is dumb but don't they have backups? Don't they have different passwords for dev/qual/prod? Probably many more things that could and should have been in place to prevent it... Sounds like the CTO should also be fired to me.

If backups are not working and this is known...

[nomad63]

If a company looks at a non-working backups as a minor inconvenience, I think the CTO's ass should be on the firing line before this poor guy's. Yes, wat he did is inexcusable and in some cases, firing might be justifiable (as in, a junior developer on his first day in his/her job doing what on a production database case) but if someone assigned him/her to perform anything on the company-critical data, that person should be the one getting fired, not this guy's. I am a 20+ year experienced sysadmin and not too long time ago, when I started at a new position, I was not able to touch any system other than few development machines for 2+ months of my start date and I know/knew my shit. This company management shows their incompetency in more ways than one. Yet they are making this person the scapegoat. Good riddance to them, as their days are numbered.

Re:If backups are not working and this is known...

[Njorthbiatr]

Yeah, they were doomed to fail without backups.

What if their server failed irreparably? What if some code went rogue and overwrote it? What if the server burns down because someone somewhere in the building left the stove on.

The CTO should be fired for total incompetency. You can have read-write access to database servers without having access to schema changes on the database. Personally, even though one of my creds actually gives me this access to some of our production databases, I never do it myself and always ask our admin to do it for me. This should be the default policy. Not even senior devs should be allowed to make schema changes on production databases without having limited credentials/sysadmin oversight.

Re:If backups are not working and this is known...

[thegarbz]

Yes, wat he did is inexcusable

Following the onboarding process on a sheet of paper that was given with him that some numbnuts decided should have the address and administrator credentials to the production database on it, on his first day, unsupervised?

What this guy did is 100% excusable. The CTO and whoever created the onboarding process should see themselves out.

I mean fuck I was on a visitors badge fully escorted for a whole week at my first job. That's right, they wouldn't even let me simply walk around the building unsupervised to say nothing of getting mission critical credentials on my first day on an instruction document that could potentially seriously fuck up said mission critical system.

Allowing it to happen is wrong to begin with

[sentiblue]

I've been in production management for 15 years... The first 5 years as a Sr. datacenter engineer. The next 7 years as a staff engineer and the past few years as a cloud architect. One thing that has NEVER changed is that developers are NOT allowed to touch production... and I mean not allowed to even log into any host at all...They don't even have a damn clue where the machines physically sit.

Allowing a junior developer fresh out of college to log into production with privilege that makes even a minor change is already in violation of industry protocols. I wouldn't blame the guy too hard. He's fresh out of college remember? The first person to blame is operations management who doesn't have a clear separation of duties protocol. Even if ops management allows him to log in, the next person to blame is the lead datacenter engineer who allows his account to issue intrusive commands.

If company leadership asks me to handle this fall out... I will fire those two guys first. Then the Jr. developer will get only a warning but that would be his only warning.

Re:Allowing it to happen is wrong to begin with

[Just Some Guy]

So much this. It's a major PITA requiring permission from a director for me to get access to a machine that can access a production database. I am perfectly fine with this arrangement!

Re:Allowing it to happen is wrong to begin with

[rossz]

One thing that has NEVER changed is that developers are NOT allowed to touch production... and I mean not allowed to even log into any host at all...They don't even have a damn clue where the machines physically sit.

In my office, only one developer has limited production access, the senior guy. He's the only developer who can do code releases and he has RW access to the DB, but he can't mess with the system's configuration. If he's out of the office, I have to do the code releases as the senior system administrator.

Some Company Information Please

[mykepredko]

I'm surprised that the firm has not been named - while I would think that any company that had this happen to them would want to keep this confidential, I would think that somebody would talk about it separately. I suspect that the "company" is some podunk startup in which the CTO is the CEO, CFO, head of development and probably the HR head and they've just hired a developer without thinking about access restrictions (or verifying that backups are actually happening).

Some more information would help clarify these questions and maybe better explain how such a situation could happen.

Re:Some Company Information Please

[ryen]

The author in the reddit thread (in a buried comment) describes it as a 100+ company with at least 40 devs. Pretty sad

Re:Some Company Information Please

[wvmarle]

Sound like your run-of-the-mill startup that has "the great next thing" and in the process got more investor money to burn than is good for them, so starts hiring more people than they can properly manage.

Re:Some Company Information Please

[CaptnCrud]

100 strong doesn't mean much, if its a sales heavy parasite company.

Probably every one of those devs are contractors with less than 6 months at the company....I've seen revolving door companies like that before, they are pure meat grinders. The pure essence of evil, they have no incentive to put good procedures in place or make sure backups are working because they have no incentive of existing beyond milking their current situation dry.

Re:Some Company Information Please

[Anonymous Coward]

Its probably the Trump organisation.

Re: Old News...

[Anonymous Coward]

I for one come Slashdort for the old news you insensitive clod.

Re:Old News...

[Anonymous Coward]

Reads more like an Onion article, or just an anonymous apocryphal tale

Double plus ++ good

[Anonymous Coward]

Typically HR types think because someone that has a job-title or degree, will actually be able to perform at the same level as someone with 10+ experience.
If you let in someone off the street and gives him the keys to the kingdom, you are responsible for the consequences.
Backups are not backups unless they are tested by restoring, until then its just a good intention.
Software developers can work on a copy of the production software, which was the case here, but the original DB was not protected by a read-only bit for the newbie.
The CTO was responsible for this cock up, he should be replaced, the rookie would likely never make the same mistake again, and should stay on. As the story went, their system was a house of cards held together with bailing wire and duck-tape.

Sounds like a poorly run IT system

[quietwalker]

Seriously, a day 1 dev has direct production access? Hell, any dev has direct production access? No QA, no release management, no integration or functional test suite if they're doing some sort of continuous deployment?

It's a pain in the ass, but if they've got any sort of actual real database, they'll have had a real database admin, running it with archive logs they can use to restore their data? ... plus their backups are gone?

What sort of fly-by-night operation is this?

Re:Sounds like a poorly run IT system

[dbIII]

Hell, any dev has direct production access

I agree with that - different mindset. Devs want to get stuff done ASAP and usually don't seem to get the concept of multiuser systems. Even very experienced devs do shit like reboot servers in working hours leaving dozens of staff twiddling their thumbs and unable to work unless months of effort has been put into changing their attitude to production.

Re:Sounds like a poorly run IT system

[Anonymous Coward]

Good devs do.

Most/some devs are shit, most/some SAs are shit, etc.

We need to have medals to give out.

[clovis]

I say if he succeeded in putting that company out of business, then he should get a medal for sacrificing himself to destroy the company.

My belief is when he saw on his first day, the badly written docs they handed him, with a printed (!) account/password having RW access, he instinctively threw himself on that grenade by destroying their production database. Only the most cowardly IT worker would have done otherwise.

Thank you, selfless IT worker from saving us from the horror of whatever product they were trying to produce.

I accidentally the whole database

[rpresser]

now I'm too full

1% new user, 99% company

[Anonymous Coward]

The new user has 1% of the blame, the company 99% of the blame. If they weren't backing up their data and protecting it from easy destruction they need to be responsible. I have no problem with them firing the new guy out of incompetence, but legal action is ridiculous and pointless, as any judge will throw it out immediately. Who ever controls the database and CTO should be kicked out immediately for allowing such sophomoric conditions to exist.

Sounds like he got "Patsied"

[Anonymous Coward]

Seriously....... I betcha whats really going on here is the CTO or someone else important made a copy of that data first. Then they could leave and start their own competing company with a whole database full of potential customers. But they gotta kill the original company first of course by killing their copy of the database. And of course they'd need a patsy for that. A gullible newhire fresh out of college takes the fall........that certainly sounds plausible to me.

Who left the rookie alone with the launch code?

[Anonymous Coward]

Any business that leaves a first-day-on-the job rookie unsupervised with that kind of access deserves to be shut down immediately, as an issue of public safety. Don't even think of what their data security must have looked like. O, the good old days, of taking the laptops with copies of company data home with you.

be happy

[ooloorie]

You don't want to work at a company where the backups don't work and where a new hire can accidentally delete all their data. Don't beg to stay, instead be happy that you found out quickly how incompetent that company actually is.

Re:be happy

[hyades1]

Wish I had a mod point for you, my friend. That is exactly, 100% correct. Rookies make mistakes...sometimes even stupid mistakes. It happens.

If a rookie can wreck a company this badly, it's hardcore proof that the problem is a long, long way up the food chain.

THAT is where heads should roll.

Re:be happy

[Anonymous Coward]

This exactly, if it literally happened on day one just walk the fuck away from this dumpster fire of a company and look for another job. Do not put this on your resume. If the company tries to publicly name and shame you in any fashion, immediately consult a lawyer.

Re:be happy

[Anonymous Coward]

This here....

Also there is plenty of 'blame' to go around. A third perspective is they are ALL to blame. Including 'the new guy'. If your company is one command away from shutting you down you have a big problem. Yeah a fuck up like that will get you fired. Sorry. But you also found out quickly that company was not long for this world anyway at that rate. All of that infrastructure and procedure takes care and understanding time and money. Some rando startup is not going to do that. They think they are going to change the world. They are now learning the hard way (as we all do). That an untested backup and recovery plan is probably the one that does not work

Fuckups of the CTO

[Anonymous Coward]

1) Why didn't the new hire get set up automatically? (sanity issue, get everybody the same environment so the sysadmins can know the environment everybody works with).
2) Why was the new hire given account information that had write access to the production database? (critical security vulnerability: social engeneering)
3) Why were the backups not working? (critical stability issue)
4) Where's the backup production server? (critical stability issue)

That company deserves to fold.

I did it too

[Anonymous Coward]

It wasn't my really first day but sort of it was. I was sent to their new unit out of town and they had a DEC machine which I had never used. I created lots of temp file and decided to delete all files from my account. I accidentally gave command to delete root dir (frankly don't remember what command I gave, but it was equivalent of deleting root dir) and then left the terminal. When I came back 1 hour later, the command was still going on. I called another guy and was told that I destroyed the mainframe hard disk which had all scientific data which was never backed up. All scientific data usually gets analyzed over 6-8 months and people take printout at the end. So nearly 20 scientist lost all of their 6-8 months of experimental data! I blamed whole thing on them for giving me account with admin privilege.

Back in the day....

[GerryGilmore]

...we were a small company making the transition from a proprietary (TI/990 - DX/10 OS) system to a Prime UNIX system and I was the guy A) learning UNIX and B) writing shell scripts to replicate basic DX/10 stuff. One of my first scripts was to delete a user. Beyond the basic password deleting stuff, I thought it would be cool to delete the user's home directory also. You know, delete user "gerry" and also delete user's home dir "/usr/gerry". Guess what my brilliant script writing failed to anticipate? Yep, someone entering an empty user name. So, the script - running under SUID perms proceeded to delete "/usr/*", including "/usr/data". Oops! Fortunately, my backup script was better written and the customer only lost a few hors' worth of work, but....Whew! That's also how we learn!

Re:Back in the day....

[rvw]

> You know, delete user "gerry" and also delete user's home dir "/usr/gerry".

Who puts a user home folder in /usr/? Isn't that bad practise in the first place? The usr directory is for system stuff.

Re:Back in the day....

[david_thornley]

That's what the /usr directory is for now. Back when I started using Unix, we had /usr/home rather than /home.

Re:Back in the day....

[Anonymous Coward]

He said back in the day... and guess what? Back in the day, the user home dirs were in /usr/

Ya no shit

[Sycraft-fu]

When we bring someone on, they do NOT get root/admin to critical servers their first day. They have to be off probation first, which is 6 months where I work. Even then, credentials for things are not on a document. That is just asking for them to get lost or stolen. They are given on a wallet sized card, written specially for that person, and they are instructed to keep them safe until memorized.

The reason is, of course, to prevent fuckups, as well as to make sure we trust them fully. The idea of giving someone full access to critical stuff on day one is stupid. Shit it sometimes takes more than a day for them to get access to e-mail and all that just because of all the other things they need to do.

This is 100% on the company. Have working backups, CHECK YOUR BACKUPS, and don't give a new hire a sheet with access to your critical data.

How's the budget now?

[Anonymous Coward]

Don't have the budget to hire experienced professionals? This is what happens.

Re:How's the budget now?

[Calydor]

If you think hiring a professional is expensive, just wait until you hire an amateur.

My opinion

[Anonymous Coward]

The CTO should be blackballed from the industry. He should be doing something closer to his skill level, like ditch digging. At any rate, he should be kept far, far away from anything that might affect other people.

Re:My opinion

[lucm]

This is what the anonymous poster actually said:

The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that i "completely fucked everything up".

It's not clear that the CTO was threatening legal action against the developer. For all we know, it could simply mean that the company would have to issue some kind of statement to shareholders or to customers based on some contractual commitment.

There's nothing in the entire story that clearly indicates that the CTO or HR acted improperly. Is it normal to have production credentials in documentation handed out to junior developers? Of course not. But the developer was supposed to configure a local instance using the output of a specific configuration tool, but opted instead to copy-paste from somewhere else a connection string that must have been obviously not a local instance. Then the developer, after being fired, left with his company-issued laptop, and started sending messages on the company chatroom. Would you want that person on your team?

This was inevitable

[gweihir]

I mean, come on:

- No working backup
- Excessive access for the new person
- CTO is incompetent and cannot admit to mistake

This is an accident waiting to happen. And the new person has zero responsibility for it. Might be better off to be out of that fucked up company though.

Can you spell Patsy?

[Anonymous Coward]

Sounds like a put-up job to me. Give the new hire root privs on the first day? Right. No backups? Right.

Get the kid out of there. Obviously the jinx is strong with this one. Keep him and your company will die horribly in short order.

Second, find who benefits from this disaster and who had the power to create it. Fire them and start the lawsuits. You are dead otherwise.

Lucky New Guy

[Anonymous Coward]

The New guy is actually in a pretty reasonable position OTHER THAN being unemployed.
He was only at the company for such a short period of time that nobody is going to ask questions about a gap in his employment record because of being fired.
In the distant future he gets to have a really powerful story to tell his colleagues. Once he is a Senior tech somewhere.

Back Assward

[Tablizer]

Corporate cheapskatism fucking them in ass: newbies handle key data and no backup system in place.

Sue/fire the CEO, not the grad.

Shit happens

[Camel Pilot]

Hey I had a security guy come into our data room to do a drive inventory and while waiting for me his curiosity got the best of him and he popped open a drive on 70 TB raid.

Lousy Management/Leads

[oldgraybeard]

The CTO and all the managers/leads that created the opportunity for this to happen are the true failures. And to think, they are all still there, so that place is still really messed up.

Seems to me this is not a single point of failure, it is multiple points of failure acting in unison. There was a lot of passing the buck and ducking all responsibility going on. Not to mention several individuals not doing their jobs.

A final thought, I sure would not want to work in a corrosive, petty and vindictive environment like that. I know it does not help now, but drive on. Not working there may turn out to be a, not totally bad thing for his sanity and future in the long run.

The real test is not how well one handles success, but in how one responds to failure.

Reasonable Response

[Anonymous Coward]

I would respond in the same way if someone deleted all my porns.

FAKE

[Anonymous Coward]

Horror and pity story is fake

Re: He needs to go to prison for that

[Anonymous Coward]

But these days he'll still get a participation trophy.

et tu, RAID

[lucm]

RAID (especially those with parity) can be terrifying. Just think of it: you have a group of disks probably acquired at the same time and probably coming from the same vendor (or even same production batch) serving the same workload in the same environment. That implies a fairly similar MTTF for all the disks.

Then one of the disks fail; this causes the other disks in the array to first handle a higher load, then to be brutally impacted by the rebuild process. That's like playing Russian roulette with a gattling gun.

Re:et tu, RAID

[Carewolf]

RAID (especially those with parity) can be terrifying. Just think of it: you have a group of disks probably acquired at the same time and probably coming from the same vendor (or even same production batch) serving the same workload in the same environment. That implies a fairly similar MTTF for all the disks.

Then one of the disks fail; this causes the other disks in the array to first handle a higher load, then to be brutally impacted by the rebuild process. That's like playing Russian roulette with a gattling gun.

Yeah. I had a RAID-5 at home. When one disk starting failing I didn't notice because the system kept running, and being a home setup it didn't actually have any lights or warning unless I manually opened the RAID manager and checked. I noticed when a second disk failed this time completely for all sectors, during the recovery two more disks started failing.

Re: et tu, RAID

[DocInverter]

I had a RAID 6 and then the NAS enclosure failed. Luckily I also had a backup.

Re:et tu, RAID

[ColaMan]

About 15 years ago I was looking after an old Compaq ProLiant server that had a 6 disk SCSI raid array in some configuration I cn't recall.

Oh, a drive just failed with an "Exceeded power on hours" error? Well, that's ok, there's a hot spare in the array, no problem.

Next day, two other disks went offline, because they were all powered up at the same time when they were new, weren't they? And they were perfectly usable, it's just that the array controller noticed that their smart attributes had exceeded a threshold that equaled "failure", powered down the disk, and that was that.

It wasn't a very pleasant week after that.

Re:et tu, RAID

[angel'o'sphere]

You have a point, but that has nothing to do with MTTF, note M means 'mean'.
In other words: if one of the disk fails 15 days after purchase, and the last one after 30 years, the mean can still easily be an acceptable 15years.

So, what you want to imlly is that they come from the same batch, and might all have the same flaw or faulty component and hence might fail in short time of each other. So yes, that again leads to 'same MTTF' but that is not what MTTF means or describes.

Re:et tu, RAID

[Asgard]

Was that a default configuration of the controller?

Re:et tu, RAID

[lucm]

The MTTF is provided by most disk vendors, it's not a calculation one makes by hand. The point here is that if a disk shits itself way before it should, odds are that disks from a same production batch running in the same conditions are likely to follow.

Re:et tu, RAID

[fateblossom]

I'm runnign a RAID 6 at home.
5 disk from one batch and 1 from a nother (Thats how I got them)
I have set up my NAS to E-Mail me with any warnings.
So when one disk failed. (One of the 5 from same batch) I order a new disk, and shutdown my NAS when I got home. And the next day I got my new disk. So I replaced it. And startet the NAS up and it used a LONG time to do the recovery.

So.
1) Alway set up your NAS/Server to send your warnings.

2) Just because the disks are from the same batch do not mean that they will fail at the same time. (The other 4 from that batch is still running)

And the reason I shutdown my NAS even when I could lose one more disk. Was it was my Home NAS. I could live without it for a day. And just in case more disk was about to fail. I could still lose one doing recovery without losing date. So just me being extra carefull.

Fake. Made up story.

[McBooCZech]

full stop.

Re:Fake. Made up story.

[Anonymous Coward]

It makes me sad that so much effort was put into this story on Slashdot. This is a typical reddit story designed to feed the masses with those things that align with their world view.

Test backups by actually restoring

[Maxo-Texas]

We backed up daily, weekly, and then monthly.

Fortunately decided to restore an older copy of a file to work with and found despite the IBM software saying the backup was occurring and was good- it wasn't.

None of the backups were any good. IBM had to come out and figure out what was going on and after that we also retired a file from backup every single day. We had previously used to disk to disk backup but went to tape when the data got too big.

If you do backups- you need to actually restore a file and confirm the file isn't empty.

Re:Test backups by actually restoring

[stoatwblr]

I can tell you a story like that about a NEAX61M telephone exchange with 30,000 subscribers on the other end.

Which was writing crap to backups.
For at least 2 years before it was discovered.
It was discovered when the system was restarted and came up braindead.
It had been restarted after loading in y2k software to prevent any crashes and stepping back to the old software wasn't an option.

Mobile phones weren't a big thing in 1999

People get upset when they can't make phone calls. They get even more upset if they have no dialtone and 911 isn't working.

It took over a day for a 2.5 year old backup to be located and restored - but the problem with phone switches is that people move house regularly, so thousands of phone lines didn't have the number they were supposed to have.

People get even more upset that they can't _receive_ phone calls than when they dial Fredd and end up talking to a complete stranger on Fredd's number and someone dialling Bill ends up talking to Fredd, etc.

It took more than 6 weeks to replay every single transaction into the switch to bring it up to date - and even after that had supposedly been done properly there were still hundreds of errors because on every played back transaction worked.

We don't care. We don't have to. We're the PHONE company (Since the market opened up to dialtone competition in 2011, this company has dropped from 100% of the enduser market to something less than 20%. I wonder why?)

Managing to simultaneously piss off everyone in an entire city takes some doing.

Multiple levels of fail

[rossz]

Lot's of blame to spread around.

The code used in production should have been reviewed by someone before execution in production. No exceptions. Especially because it's a new guy on his first day. The code should have been run in a staging environment first. How long was it known that the backup system was broken? This mistake was obviously not the newbies fault.

If my production DB backup was hosed, I would be dropping just about everything else to get it healthy again. A deleted database would mean some lost time since loading up the last backup and doing a PITR from WALs isn't fast, but at least I know I can get production back up no matter what someone does to the active DB.

How the fuck: DevOps.

[Anonymous Coward]

If something didn't work then it would have been very time consuming to roll back and also very difficult to debug.

This also makes me think a bit on why developers, even newbies, would have access to the production servers. If something broke between the transition between the development system and production then they'd want it fixed right away. This would be easier to do if the developers had access to the production servers. This would be avoided if the production and development systems were identical but like in that project I mentioned it might be that budget constraints prevented them from doing so.

Now you have some idea why containers (http://www.drdobbs.com/architecture-and-design/containers-for-development/240168801?pgno=1) are such a big deal. One can do development (https://azure.microsoft.com/en-us/blog/containers-docker-windows-and-trends/) without touching the production servers, but it does allow for a more seamless transaction when needed.

Why Was He Mucking With (G)It In The First Place?

[Anonymous Coward]

Interesting. I 'cloned' a Git repository and nothing bad happened, and with proper permissions no commit would have been destructive.

As Lu Tze said in Thief of Time

[Anonymous Coward]

disaster planning always leaves out something. The disaster. It's when you don't check if your backups work that this mistake becomes such a huge fuckup. And that wasn't this new guy's fault. Sure it's a huge cockup, but less of a one than the system that doesn't have any way to verify the backups worked. Any of them asked to leave and never return?

Fire the CIO and CISO

[Opportunist]

If a junior developer can FUBAR the company, these are the two that fucked up royally. Very obviously processes are crap and gross negligence is running rampart.

Fire these two bozos. Out of a cannon.

Ah, the memories

[shanen]

Actually reminds me of two fellows I worked for briefly. Both of them were actually too small to have a separate CTO. In the first case I decided to leave as soon as I figured out the legal liability their main customer had incurred due to pirated software. There were actually two packages, and one of them was a database. The second case was a total shoestring operation and one of the first things I discovered was that their so-called daily backup processes were not actually backing up anything.

Now that reminds me of a third case, where they actually did have a CTO, in title if not in competence. While they did fire him, it was too late to save the company... It was one of the few things they'd done right while I was working there, but I decided to leave when one of the actually competent managers was fired, and they disappeared not long after that. (That was actually my last stint in a startup, and after that I just went back to safety in a big company for the rest of my career...)

Re:Ah, the memories

[sheramil]

... The second case was a total shoestring operation and one of the first things I discovered was that their so-called daily backup processes were not actually backing up anything.

Ah, that moment... months into it, when you decide to check the backup media and find it's empty. Heh heh heh.

Re:Ah, the memories

[HornWumpus]

It's actually pretty simple to make the last step of a backup be: Restore to a particular development database server.

And it's handy for devs to have near current data to test against, as long as it's not the only dev database.

Re:Ah, the memories

[shanen]

I feel obliged to note that one of my highest priorities before messing with anything was to figure out the state of the backup system, even though the boss was screaming about other emergencies. It still took me a couple of days to figure out what was going on, or more precisely not going on as regards backups. Various parts of the system had been done by various people with various degrees of competence, all of whom left for various reasons.

Re:Ah, the memories

[minstrelmike]

I wrote a cassette tape backup system for pcs a long time ago and was also on the help desk. I was talking to one of the users about something else and they just mentioned that the new backup procedure (which handled multiple kinds of units) was really nice and fast. It finished in less than 2 seconds!?!

If the tape unit was not found at all, not plugged in, then the software failed with a bright Success! message. I did not think to test not plugging in the unit.

Always test your disaster recovery plan!

[No Longer an AC]

At least know that your backups work.

In the mid-90s I had just started working at some place and the guy I was replacing (he left for a better opportunity) was showing me how everything was set up and as a demonstration he deleted his own account. I guess he felt he didn't need it anymore, but then he says there might be some useful stuff in there and tells me it would be a good exercise for me to learn how to restore from backup.

Not a big deal, it was actually documented and I had done that before at a previous job...but every single tape was bad.

When I told him, his reaction was that I must be doing it wrong, but he was similarly unsuccessful. I think they had been using the same tapes ever since they bought their system and they were all kept in a shoebox in the same small office with all the servers.

After 6 months I left for a better opportunity too.

How is it reckless?

[Anonymous Coward]

The company can work for months without any Cxx level staff. Maybe years. It will just be less efficient (if, and this IS an assumption not justified by any data, they are doing some sort of positive job). But without the workers they won't last a week.

So please explain how losing all the C* staff is reckless.

Re:How is it reckless?

[lucm]

The C-level executives are the people in charge of executing the company strategy as established by the board (who are representing the owners). If you remove them, you basically have a headless chicken. If you replace them all at once, then you have months of instability as they have to rebuild their teams and puts their systems in place.

That's why elections are staggered in the USA. Could you imagine the mess if everyone was elected at the same time?

Re: How is it reckless?

[Anonymous Coward]

Um, the elections in the USA are all at the same time. Local, city, state, federal, all on the same day. The four year presidential cycle sees 200% of congressman and 67% of the senate cycled too...

You fail at civics.

Re: How is it reckless?

[lucm]

I don't fail at civics. You fail at understanding what "at the same time" means.

Here's a hint.

https://en.wikipedia.org/wiki/...

Re:How is it reckless?

[Anonymous Coward]

USA elections are not the only ones in the world. Most other democratic countries do replace both the entire lower house (those who actual make the law) and the executives at the same time.

Re:How is it reckless?

[someone1234]

But this company had a very erroneous fault control strategy and/or execution. The CTO was definitely not on the ball.

Re: How is it reckless?

[Anonymous Coward]

Hahaha.. Holy shit... People actually believe this?

I suppose soldiers would win a war without any officers or leadership too? I mean, they're the ones pulling the triggers... Who needs a plan?

Re:How is it reckless?

[lucm]

Maybe. Or maybe the company has a "live in the fast lane" culture where all resources are focused on growth, and the "security" policy is to tell new hires to follow instruction and not mess with production until they know what they're doing. If you think that kind of company doesn't exist, you need to beef up on your Silicon Valley history.

But really all I mean is that firing executives is not a good response to disasters.

Re:How is it reckless?

[david_thornley]

So, what do you do with the CTO that doesn't involve firing? All other appropriate responses I can think of are illegal.

What happened here is that a junior dev wiped the production database and there was no usable backup. That indicates gross incompetence in more than one area. If the CTO was even slightly competent, this would not have happened.

Re:How is it reckless?

[Anonymous Coward]

That's why elections are staggered in the USA. Could you imagine the mess if everyone was elected at the same time?

I cannot even imagine the mess it is in now and I have reality for a template! =D

It's the optimal strategy

[Anonymous Coward]

It's part of "throw them out in deep water and let them sink"-strategy that proves the most cost-effective.

Re: It's the optimal strategy

[Anonymous Coward]

In the long run? No, it isn't.

Re:Backups? Different passwords?

[Anonymous Coward]

What are the chances that their "backups" were simply a replication of the database? An empty database replicates very quickly.

Hehe

[Anonymous Coward]

One of people in my team has a fancy degree from a fancy university and is totally arrogant and looks down on everyone. He did something very similar to this. What an idiot.

It goes to show that just because someone went to college and has a degree, it doesn't mean they are not stupid. Also, anyone can make a mistake. Also, don't be an asshole.

Reckless behavior

[Anonymous Coward]

If someone deleted all the data and you want it back simply rollback database to any point prior to incident. No need to panic or even restore from backups.

I suspect: Fake News

[Anonymous Coward]

It sounds too easy for anyone to do what (s)he did. I do not believe it for a second.

I have seen so many unbacked up production DBs

[EmperorOfCanada]

It is scary how few companies backup their crap. I would be willing to bet that less than 30% of billion dollar companies (that really depend on their computers for day to day operations) could be back in operation in under 24 hours if they lost all their servers at once.

I wouldn't be surprised to find that a good percentage would be very screwed in the long term.

This is not only their data but the system as a whole. When I am consulting at most companies that are retiring servers I often suggest that they box them and put them in a safe place as a super emergency backup. The idea being that those servers are probably close to what they need to struggle along. This is after I have usually laid out a solid backup and disaster recovery plan. Seeing that most companies won't do a good backup then having some 2 year out of date servers might be better than nothing.

I did this too

[riverat1]

I can relate to this. I wiped out the production database for our ERP system when I was trying to create a copy of it. Fortunately I had good backups and was able to restore the DB with minimal loses but it took all day (this was back in 1997 and the computer wasn't particularly fast, a SparcServer 1000 with 2 CPUs and 500 MB of RAM). In my case I wasn't fired and retired from the job last year after 31 years.

Re:I did this too

[Anonymous Coward]

As a new sysadmin (Modified Mainframer), I deleted the old sysadmins account on the first day. No probs.

Next week, when the system rebooted, JES2 would not start, because the 'owner' no longer existed, and nor would RACF (just say AD). Yeah, IBM later fixed that one.

Boss though it was hilarious and the onepack backup recovery sorted that out fast.
Now this is relevant today, because nobody checks the backup tapes/oops indexes are offsite, and there are LOTS of userids that can read them. The lesson is to have an emergency admin account in the safe - and tested - because you dont want it locked out because it is older than 30 days- or too many attempts - and there is no-one else.

Best one I saw is backups finishing in under an hour. On inspection only 10% was being backed up, as the backup did not have permission on the other 90%, as well as not seeing 2 years worth of new disks added.

In conclusion, all the recovery testing I have seen- has ALWAYS failed, but management Cxx always said PASSED. If the story was true, then the Cxx has to be fired is his signature is on the recovery test..

Re: Old News...

[KGIII]

Retired boss, here. If a junior dev can push to prod AND can delete data, it isn't their fault. Okay... I am still going to try to salvage my hire and see if they fit in QA, or see if the debs will stop fucking with him. Seriously, not his fault. Nobody should be able to push to prod, withou someone signing off. Ask me how I fucking know. I'm not even a programmer. I just paid a lot of you and shut the hell up and listened.

So, if I am wrong then they were wrong. You don't push code to prod without someone signing off. The person signing off has ultimate authority. You sure as fuck don't let a junior do it, without oversight. Not now, not ever.

If this happened in my shop, some titles would have been changed.

People make errors - thats why you have backups.

[Anonymous Coward]

I had been on my first job over a year , every morning I religiously went to the office safe and retrevied the two backup tapes to use on that day, I would check the log and if no errors eject the previous tapes and insert the new ones. One of the tapes would go into the office safe and the second one I would take to the security office off site.

That day I put the tape in without looking at it and inserted in the card inlay with it . Meaning it would not eject Down time on that machine meant an entire factory production line was sat idle . So that was an expensive mistake I had to fess up.

Wrong person fired

[Anonymous Coward]

They should fire who ever gave the new guy full rights to the production database. That's where the real fuck up was.

So many thing are wrong here

[Anonymous Coward]

There are so many things wrong here:
1) the backups aren't there
2) this also means the backups aren't tested. Your backup strategy only works after you verify (on a regular basis) that you can restore
3) there is no good staging nor testing nor development environment - why would he need access to production otherwise
4) there is no proper process to move data and software from development into staging and testing
5) the CTO does not take responsibility for his own created mess

If I was the CEO, this CTO was certainly not there.
He might have hired the wrong junior, or not, but that's not what's wrong with this company!
Sadly, I still see many clueless CTOs in real life.
And clueless CEO's.
One often wonders how it's possible to world still works ;)

We have this happen all the time!

[Anonymous Coward]

Due to stupid fucking touchpad server consoles! How the fuck thought it was a good idea to remove the track balls and put in click sensitive touch pads with drag and drop support? We frequently have people move executables and whole directory structures to the wrong places.
(No, we can't use VNC or remote desktop since its a "secure" facility. No, we are not allowed to modify the hardware or replace it since this is a "certified" configuration. Yes, we have written multiple memos and reports. Yes, our management hierarchy are mostly retards.)

PS. Not this severe and we have backups.

Not His Fault

[StormReaver]

This kid will have no problem getting a new job. What happened was not his fault in any way, shape, or form. The fault lies squarely with the CTO (in no particular order):

1) He allowed a new hire to have superuser, unsupervised access to the production database without an ounce of training.
2) He allowed an unvetted new-hire script to contain actual superuser credentials that could be used to wipe out the production database with a simple copy and paste error.
3) He allowed a new hire to run that script unsupervised.
4) He set a new hire loose on their production database with no mentor looking over his shoulder.
5) He did not require the backups to be verified and/or tested to make sure they work.

I'm sure there are more (feel free to expand on it), but those are the ones I can think of off the top of my head.

Re:Not His Fault

[StormReaver]

By the way, I did something similar on my second day on the job (no, I didn't get fired). I'll skip the details, but yes the management fucked up and had a hard time coming to grips with their fuckup. Fortunately, circumstances in my case were slightly different; but it was close enough to this story that I don't blame the new hire one bit.

Systemic issue resolved....

[neurosine]

Although obviously the new guy fucked up...unless his job was to be solely responsible for the repository...he probably shouldn't be fired. He's pointed out a systemic issue with the system...and punishing him for it isn't reasonable. I don't know all of the details...but on first glance...the people responsible are looking anywhere else for the blame. Typical.

Re:Old News...

[K. S. Kyosuke]

Be more considerate; /. has just recovered its deleted database. It did take some time, obviously.

Seems pretty obvious...

[DaveAtWorkAnnoyingly]

They gave an employee a procedure that deleted their database. The only mistake the employee made was following the procedure and not using his own credentials, instead using the credentials in the procedure. That was a standard human error, no big deal. Humans make mistakes, it's all of our's jobs to reduce the probability of this, with good quality procedures, questioning attitudes etc. To provide an employee with a procedure that contains credentials to the production database is ridiculous, this was going to happen at some point with a procedure as poor as that. To then blame that employee immediately and fire him is knee jerk and immature. To be fair, the guy is probably better off not working for that company anyway.

Not having backups etc, well, that's the companies fault. All the employee has done is to uncover a poor working culture within that company, and expose a bad CTO. This is not the failing of the employee, it's clearly a failing of the company. They deserve what they got.

The person fired can make many positives from this. This is an excellent case study about business culture, procedures, strong catastrophe planning and testing, threat and error management, how to treat fellow humans. Your experience in this can be valuable to other companies, and the good ones will recognise this. Use this to your advantage, let that company fail, it was going to happen at some point, you were simply unlucky...

To give a personal experience, I work in a nuclear power station, in the control room. One shift I made a mistake, I rushed a job. I followed my procedure to the letter, I didn't make any mistakes, but because I chose to complete a certain activity early before an additional check was performed I broke our operating rules (the law effectively). I owned up to it as soon as I was conscious of it, and the investigation started. My authorisation was pulled and I went for drugs testing etc.

The first thing that occurred once management found out was that I received a phone call. It was the Operations Manager calling for me, and he called me to thank me for raising this report and owning up to it. He understood that without this honest reporting, the problem couldn't get fixed, and would happen again in the future. It has probably already occurred in the past too. My authorisation was recovered that same shift, and I carried on. There was no detriment to my career (quite the opposite infact) and life goes on, but now with better procedures...

Industry Standard Best Practices Ignored

[zifn4b]

- Insufficient IT resources (inadequate or no backups)
- Hiring developers to do everything in the company and giving them broad access to everything

These are two industry standard best practices that many companies ignore and they have no one to blame but themselves.

backups

[AndyKron]

Backups weren't working? Who's fault was that? That's who needs a talking to.

I'll believe this...

[Evil Kerek]

This post smells a bit. I'd like to some proof it actually occurred like this or that it occurred at all. The fact we've heard nothing indicates it was a small company - this isn't the sort of thing one keeps quiet. The lack of a functioning backup tends to also point at a small company.

The description is such a slam dunk in favor of the new guy and making out the CTO to be such an idiot...I dunno....I'm just skeptical at this point.

The rancor coming from what amounts to utter hearsay is...well actually, no, it's not surprising these days. Any sort of proof for people to pile on stopped being a requirement long ago - it won't surprise me if this is someone doing a social media experiment.

My 0.02
EK

This sucks so bad for that kid it's not even funny

[tomxor]

Fuck you incompetent business, this will go down badly for you in the media: Anyone hiring a junior straight out of college knows not to throw the crown jewels of the company into their lap to play with, and if you don't have working backups then you're double to blame...

Kid: Don't worry, notice how every single comment here will put the blame squarely on your terrible employer. It's just luck getting a first good employer, you will find more. Remember the hill climbing algorithm... your marble just rolled straight into the deepest pit, it's not your fault it's just bad luck, you've got a lot of random rolling to do, keep your head up and you will eventually find some nice reasonable people that appreciate you.

Re:Agree

[Anonymous Coward]

I have seen this "Heads are going to roll!" mentality several times from people in their teens and 20s. It is an immature 'King Henry VIII' mentality. They think that if someone causes a single fuse to blow, that power should be cut from the whole building. This is why most companies are wise and never let people younger than 30 get into management jobs. Those companies that do always suffer harm from the youngster's decisions.

AAAAAAAAAAHAHAHAHAHAHAHAHA!! Lol

[Anonymous Coward]

excuse me while i gather myself... HAHAHAHAHAHAHAHA... HAHAHAHA. this made my day! Thank you fine sir!

Re: Old News...

[wisdom_brewing]

Cannot believe the obligatory has not been posted...

https://xkcd.com/327/

Re: Old News...

[Highdude702]

I dont think people like it when you take away their job title and replace it with 'fuckwit' and 'moronic fuck who leaves permissions open'

Re: Old News...

[volodymyrbiryuk]

Exactly, this is why the people who are in higher positions are payed more. What else could justify the higher salary if not the greater responsability. Because usually they don't do low level stuff, the actual work that drives a company.

Re: Old News...

[Anonymous Coward]

If you were his/her boss then you should dig a big deep hole for yourself, jump in and ask the junior very very politely to put the dirt back into the hole. Really. I would be very very ashamed of myself as a CTO if a junior dev would have access to a production database. I'm a very old dev and I still categorically refuse to do ANYTHING on production.

Perhaps it will make more sense this way

[dbIII]

It's not about good or bad, it's priorities and a lack of understanding of what the implications of disturbing a production environment are. They are not going to get that understanding by magic, they only get by looking at how a system is used over time, and if that's not related to what they are supposed to be doing for a living it's hard to put in the time.
Like the above problem it is a management issue. If you want a dev to act like a sysadmin you have to schedule time for it to happen.

Re: Old News...

[Anonymous Coward]

This story is precisely the argument for functioning backups and some sort of seniority process that doesn't put new staff in the wheelhouse steering the ship on their first month, let alone their first day. For that matter, permissions, or single switch software bombs, or... the list goes on and on. Pretty much everyone is to blame except the person telling the story. While I can imagine they might not keep him around for the simple reason he reminds them of their own ineptitude, he's a junior, and juniors make mistakes.

no backup? i've no sympathy at all..

[zr]

none at all..

Wow

[Anonymous Coward]

When I got hired at my job as a Unix, not quite admin, but kinda like jr admin, we didn't even get root until months after we started. My question is, why was there not a back up? Before I touched anything, as root (or sudo), I would make a copy of the file so I could roll back any changes I made that might have gone afoul.

The guy probably screwed up, but a good infrastructure should be in place to revert those changes. Sounds like they're making this guy a fall guy for their own incompetence.

Re: Old News...

[thegreatbob]

This is more like they had a spurious cable attached between some random door below deck, and some critical lever in the control room... The point people are trying to make is: this guy is being given the shaft as a direct result of the fallout from idiots putting privileged/critical information into widely accessible documentation.

Re: Old News...

[Anonymous Coward]

A junior should at least know the commands that make this happen. So nerd cred is lost. Facilitating this level of screw up is IT and management.

The company's backups weren't working,

[QuietLagoon]

When the root cause of a problem presents itself so clearly, it is difficult to understand why the new person was fired. The person in charge of doing the backups should be the one who was fired.

Let me guess

[nospam007]

The company is named 'British Airways'?

Have we not learned anything in the last 40 years?

[Anonymous Coward]

In my 35 years at an IT shop we had three major issues - but none as bad as this

1. IBM had two interactive environments - CMS and TSO. In CMS when you catalogued and uncatalogued files you were acting in your own environment.
          When you uncatalogued a file (DSN) in TSO no job (even production) could fine the file.
          We were just glad the new guy did not delete the file. We did nave backups. Very soon after the production files were password protected.

2, At the Data Center, there was a large red button that would shut off the power to the computers and disk drives in case of file. Unfortunately someone leaned against it. Shortly there after it was put behind a cage to shield it.

3. When disk drives were very expensive (10X the annual cost of a programmer) IBM came out with device that move files to tape and reloaded files from tape to disk on demand. Worked well until the SINGLE read write head went bad, The tapes were in a special physical format (like a jute box) and could not be read, No production jobs that had one file in the subsystem could run. IBM flew the files out NY, loaded then on disk drives, and installed the drives in the data center,
Needless to say the subsystem was soon removed and IBM never sold a lot of them, even after they shipped with two read write heads.

These all happened in the early 80's

.

   

Everyone calm down

[jon3k]

Let's all relax, this is obviously fakey. Slashdot is now citing reddit posts from self-proclaimed throwaway accounts as news? Yeesh.

Process, process, process..

[malkavian]

The company is responsible for process.. They should be continually having it as a background task (even in the back of their minds) of "how do we keep the wheels on this thing".
Just as important for a startup as it is for a big company (maybe moreso).
Starting point of everything is backups, backups and more backups! They're your lifeblood if you're a company using data for doing real things (hint, they all do).
If a backup isn't tested, it's not a backup. If you've not given your systems team time and resources to test backups, then the problem lies with a lack of understanding (or listening) further up the management chain.
Once you know you can recover, you know you can use your data safely... Then it's the process for letting someone loose in your systems...
If they're not authorised for prod, then they don't get the credentials for prod. And credentials should be stored in a nice encrypted file on a VERY well backed up area. Credentials should NEVER be in plain, or embedded in unencrypted documents.. That''s so much a 'no' that it fails the basic test of "do you understand what the word security means?".
Rough analysis of the situation is that there's a marked problem with process, and large problem with security and information governance, and an insane problem with backup processing. An accident waiting to happen..
The developer made a mistake.. Yep, a big telling off for not engaging the brain and being careful with the credentials is well in order. You need to treat systems carefully.. But us old grizzled systems farts got our caution from somewhere, and it's called experience.. And experience involves coming up through the ranks and seeing stuff go wrong. Even stuff that's almost guaranteed never to go wrong. I'm all for having young blood in a place.. They do bring new approaches, and new ideas.. But us old farts have seen many of the approaches before and can say why they're broken and/or dangerous.. It's just that little gap of new idea where we thing "Hey, that's actually new.." that means things move forward... But with us old farts around, it happens in a controlled and safe manner..
So, I'd say this guy is actually MORE valuable than he was before he made the screwup.. I can be fairly sure that they'll be MUCH less likely to be cavalier in the future, and put an awful lot of thought into safety of systems..
From the way it's set up, I'd actually say they're better off out of that environment anyway with the CTO having that lack of understanding of the root cause of the problem and/or management also not understanding/standing up for the dev and explaining root cause.
Alas, I'm not hiring at the mo, otherwise I'd have a serious chat with them and pick them up.. People getting through extremely bad screwups AND openly admitting they screwed up is a good sign of a person you want on your side.. Screwups happen.. It's the people that HIDE them that are dangerous..

Re:Process, process, process..

[angel'o'sphere]

In the environments I work, developers can not even connect to any prod and most of the time not to any preprod system either.
They only can access dev and test environments, and often they have no access to the test filesystem, hence admins have to set up test dbs, test systems and basic test data and developers or their software can only access the running DB or other software via REST etc.

Likewise admins that actually can work on PROD have usually no access to test or dev. Lets call them 'operators', old school :)

Toddler with a gun?

[Theovon]

The security measures we put in place in IT systems aren’t just about keeping out malicious users. They’re also about granting access only to those who understand how to use them properly. You don’t give guns and knives to toddlers, and people who do are the ones responsible for the carnage.

That being said, I recall one time that a friend wanted to back up his hard drive before upgrading his OS or something. He had lots of music and photos and stuff, so that made sense. So he asked our sysadmin to help, and the sysadmin used some disk imaging software to do it, to copy from one drive to the other. Unfortunately, he got the drives backwards.

Doesn't everyone know the basics?

[Anonymous Coward]

PROD / STAGE / DEV / QA - these are environments that keep you safe, even if you work in all of them

Irony

[Shoten]

You know, a few weeks ago there was a post on reviewing code before pushing it to production, and some people seemed appalled by the idea that another person...who would invariably be a coder, lest they be unable to understand what they were looking at...would be in the process flow for committing a change. I pointed out that this was actually an industry best practice in enterprise organizations, and an inherent part of any SDLC. In any large environment, people with access to development environments do not have the rights to push changes to production. This concept was not well-received.

And now, I see an awful lot of people saying that this problem above was the CTO's fault, for giving a developer sufficient access to change the production environment. I agree with that point...but it's amusing to me how a lot of us seem to live in a fantasy world where we must have access to everything, but when we screw up it's someone else's fault for giving it to us. We have to choose one of the other; either we are justified in having full autonomy and accept the consequences of our actions...for good and for bad...or we accept safeguards that protect both us and the organization we support.

Telling

[Anonymous Coward]

Wading through these comments, its pretty obvious how many posters here do not have a clue about what their talking about.

Whether its a small shop or large one, what the junior employee did should be "impossible". Following basic operational practices, not just the backup should be recoverable, but production databases should have been physically separate from development/testing. That way, even if a developer needed to access a database, they wouldn't have the administrator privileges to override the "person" responsible for the production data.

Its pretty obvious that the person responsible for implementing the computer operations for the company does not know how to do their job. (That usually is CTO or COO.) The only way its not 100% their responsibility is if management senior to them overrode what was supposed to be proper operating practices to prevent this from ever happening. Its still the CTO/CIO's responsibility to prevent management from laying the seeds for its destruction. Usually, you eventually polish your resume and look for a new employer with more competent management.

Yes, whomever was responsible for system/backup administration also needs to be fired, but usually the boss of them is the COO, and he's the one truly responsible for not making sure backups were recoverable (which is only accomplished by testing). Perhaps for a small company its become impossible to hire competent employees, but this is what top management is responsible for.

So sales people, developers, and other non-operations people, stop sticking in your two cents. Its truly ignorant noise. For the few senior people here, get lost. You're not supposed to be wasting your time on Slashdot. For the grunts that feel the need for validation, get lost as well. If the company is capable of firing you for merely making a mistake they're responsible for, you're working for morons (and you're a moron).

Sounds like management is an idiot

[Anonymous Coward]

While deleting the data is a big screwup, it sounds like the bigger screwup is the CTO/management which first off gave such significant access to a new employee who's skills were not yet tested. On top of that they didn't appear to have proper backups. This office was probably a few years from a virus or hard drive failure taking out the data anyway. I don't think the firing itself is all that unreasonable, but blaming everything on the new hire is also pretty disingenuous.

Re: Old News...

[Applehu Akbar]

That newbie going the wrong way on a copy operation could even more easily have been a disk error or a ransomware infection. The problems was not having backups. I'm with the Register readers on this one.

I fail to see how they have the legal ground to do

[waspleg]

anything. You would have to prove malice. This was clearly an accident and something they should have been able to recover from.

Firing? Whatever, I'm assuming this is America we're talking about where I can fire you because I don't like your fucking ankle socks and you have 0 recourse.

The root cause

[XB-70]

Forget the kid, forget the CIO etc. The root cause here is that setting up an easy backup system is a right, royal pain in the ass.

Backup protocols should be integral to every database vendor's product. Db creation should not be allowed until a backup system is in place FIRST.

Secondly, Information Systems Audit and Control Association, ISACA standards (or similar) should apply and vendors should work to integrate those standards into their database creation protocols.

In the end, your database represents your organization's livelihood. Why would you not protect it as such?

Re:The root cause

[Wolfrider]

> Backup protocols should be integral to every database vendor's product. Db creation should not be allowed until a backup system is in place FIRST.

--I find what you say to be fascinating, and would like to subscribe to your newsletter. :b

--Veeam bare-metal backup does a pretty good job of this, there's a prompt to generate a recovery ISO before setting up/performing the 1st backup.

It would be stupid to fire the kid for this

[Anonymous Coward]

But not unexpected given the quality of the folks that set up the situation.

He did what is expected of a new hire fresh out of school.
Followed instructions pretty closely.

The difference between 'pretty closely' and 'exactly' should not cause harm on the first day.
That's on the folks who gave him the wrong credentials for the exercise.
(Also on them is not having a backup that could survive a bad command being typed.)

Assuming the kid is smart and can learn, he got a REALY good lesson in the importance of knowing what you are doing before doing it.
If they fire him after this they will loose a person with this training.
Folks trained in this appear to be rare in this organization.

Answering yes

[Anonymous Coward]

Answering yes to a question you don't know the answer too is a sure sign of a fuck up

Learning From Others

[Anonymous Coward]

Guys, THIS, and crypto locker, and wanna cry, and fat-finger (human brain virus that causes people to hit return way to fast) is WHY you pay for ENTERPRISE STORAGE that takes snapshots that are efficient, and fast.

There is a difference between logical protection (snapshots on the same array) and physical protection (backup to different devices).

While we are at it, WHERE is the DR copy? If a sprinkler went off in the datacenter or drive/array data loss occurred, where would the production data be?

WHO has money for a CIO/CTO type and can't afford proper Enterprise Storage?

I work in the Enterprise Storage industry. Use a modern enterprise storage array - take snapshots, replicate to another array, take snapshots there, too. Take backups from the snapshots (so as to not affect production). Use a backup SW that support array off-loading so you can make your backup windows (instead of all the host side copying).

Duh. There is a reason these products exist.

Millennials Are Nihilists Throw Them Away

[Anonymous Coward]

Just cause.

Should burn them all.

Call them "Zuckers" for their patron saint.

Jajajajajajajajaja

Re: Old News...

[Anonymous Coward]

The DevOps fanboys would provide a much more mixed response.

I've seen testing turned over to the customer in the name of DevOps: protecting operations is "deprecated"

Re: Old News...

[Zontar The Mindless]

-10, Gratuitous XKCD Link That Is Not Actually Relevant To Story

Re: Old News...

[Zontar The Mindless]

"Great news, everyone! You're all fired!"

Re:Old News...

[ganjadude]

well, we only got the backups back online today!!!!

Everybody has admin rights

[PPH]

p.And this is what happens. How about dev, test and production accounts or environments? And a QA process in place to make sure code from the devs works properly before promoting it? And scripts prepared (and tested) for every maintenance and operation function. So nobody has to log on as admin and fat-finger anything on the command line.

I'd hire there kid tomorrow.

[Anonymous Coward]

He has learned a very important lesson and will pay more attn to detail than any other fresh out of college jr dev on the job market.

Recent grad given full production access

[DukeLinux]

What even marginally competent IT manager would give an inexperienced person the ability to modify or delete production objects? I feel sorry for this guy, but his management is completely at fault here. We know the story. Nothing will happen to them and one or more will likely get promoted.

Re: Old News...

[KGIII]

I gotta stop trying to type on a tablet.

Anyhow, I'd not want to fire anyone, without more information at hand. Firing people has large costs associated with it.

How to describe this?

As stated, I am retired. I've been retired for over ten years. Back in my day, we trained people and paid for them to continue their education. Don't laugh, that really used to happen.

So, I'd probably try to salvage this as a teaching moment. I'd also be changing someone else to the lead position. If I were to fire anyone, it might be the person who enabled this to happen and probably wouldn't be the junior.

Our shop was kinda laid back. It is unlikely that I'd fire anyone, but someone is going to catch a whole ration of shit. If this is a habit, then someone is getting fired, but if this is a habit then it speaks to a larger problem.

I'd start with the junior dev and work my way up. I'd keep digging until I found the problem in the process. I'd then take time to listen and find out what we need to do to make sure it never happens again. There's gonna be a meeting. Hell, there are going to be several meetings. Chances are, this is going to have a bunch of little teaching moments.

Junior devs, autocorrect hates that word and thinks it should be debs, sure as hell shouldn't be able to push to production. Even if they could, those backups damned well better be timely and functional. The IT staff, I guess you call them ops today, were great at their jobs, thankfully. For reasons, we had a pretty robust backup methodology. I am still flummoxed when I hear stories about not backing up properly.

I haven't done this for a while, so it is time to do it again.

I used a computer because I had to. I actually kinda hated them, for the longest time. I am not a programmer, but I programmed because I had a task that needed to get done. I'm not an admin, but I did that job because it also needed to get done.

Then, I was able to hire capable people. I was able to hire competent developers and IT staff. It was those people who did much of the heavy lifting. I learned a lot from them. I learned what best practices were and why they were that way.

So, as I said, I've not taken the time to do so lately. Here's a tip of the hat, and a nod, to those folks in the admin side and in the developer side. Here's a tip of the hat to the database admins and to the jerks who secured my network. Here is a tip of the hat to the programmers and to the QA. Here's a tip of the hat to those who spent long hours beside me, enabling me and teaching me.

April Fools Day!

[Anonymous Coward]

Programmers don't have access to the database(MySql, Oracle, etc).
CTO should of had a policy where things are tested first before deployment in a real world production environment.

So, Did this guy say he was a programmer or a Database Admin?

Where I used to work, they had backups of the server and data.

lol

[Anonymous Coward]

I almost did the same thing last week :/

Re: Old News...

[KGIII]

I owned the company. We were also pretty relaxed. I'm not sure we'd have used those titles, but I'm pretty sure someone is getting demoted - and it's probably not going to be the junior developer. Hell, I might have taken Mr. Junior Dev out to eat, or given them a small bonus - for having discovered the glaring hole in our process that enabled them to do this and for demonstrating that our backups were worse than useless.

Not his fault????

[Anonymous Coward]

BULLSHIT.

How about failed to follow written instructions?

He even writes, "After running the command i was supposed to copy the database url/password/username outputted by the command and configure my dev environment to point to that database. Unfortunately instead of copying the values outputted by the tool, i instead for whatever reason used the values the document had."

So little attention paid to detail, what else will he fuck up in the future? I'm with the company on this one.....his ass is grass....no need for legal to be involved but damn right he's fired.

Don't Worry

[cstacy]

Don't worry about this guy, he got a job at Verelox the next day!

Bad analogy

[s.petry]

If you hire someone to clean your house and they break an expensive vase on their first day, do you say they should have had a more senior cleaner shadow them for a while?

Or do you expect them to arrive with the basic skills they claimed to have during the interview?

If I hire someone trained as a professional driver to drive my car and they head at 120mph down the wrong side of the freeway totaling the car and causing untold damage to other cars, you bet your ass they would be fired.

Even that however is not a good analogy, because I'm supposed to have a guy in charge (CTO) of ensuring drivers can actually drive. I'm not going to speculate further on the size of the company because it's not relevant. The CTO is mentioned as firing the guy, and the CTO should be fired right along with him. The guy in charge of backups should be fired also (also mentioned in the story).

How the fuck-The Cloud.

[Anonymous Coward]

A good reason for one-man businesses to use the Cloud. Let someone else worry about backups.

Re:How the fuck-The Cloud.

[Aristos Mazer]

Well... maybe. There's this other story on Slashdot today... https://tech.slashdot.org/stor...

I can relate

[Anonymous Coward]

I can relate to this. While it wasn't my first day, I was working as an admin when my experience is by far more on the dev side. I ran database scripts against the production system, and was later told I was doing it wrong.

Fortunately, they did have backups for most of it, but I singlehandedly took down the production system for a day. Multi-billion dollar system. Despite attempts to clarify what I had done wrong, I could never get instructions on how to do the task "right". I was told how to avoid doing the same thing again, and that sufficed, but was still a PITA manual hack.

First the exec responsible, then maybe the CEO

[uncqual]

If the scenario is as described (which, frankly, seems somewhat unbelievable - why would production DB access credentials be floating around in a document available to a junior dev?), there is at least one senior executive who needs to be fired immediately. Perhaps more than one such executive needs to go as there are at least TWO serious and obvious problems - (1) the DB access credentials being published where a junior dev could even find them and (2) not having a tested and working restoration/return to service plan. If different executives are responsible for these two epic fails, they should all go. Perhaps the CEO should also go (although, that may also be the owner of the company -- in which case of course they would be unlikely to fire themselves!) as she has very bad judgement in hiring and does not exercise appropriate oversight.

Damn

[DontBeAMoran]

That's the second time today this happened! Won't somebody think of the databases?!

The CTO should definitely go.

[Afty0r]

I actually read the article. There are many reasons why the CTO is incompetent, but one is that THE USERNAME AND PASSWORD FOR THE PRODUCTION DATABASE WERE IN PLAINTEXT IN A PIECE OF WIDELY DISTRIBUTED DOCUMENTATION. That alone is utterly crazy. For the sake of making this post more positive, here's a few things the company could/should have done here: 1] Don't have credentials in documentation, they should be in a password vault. 2] PCs on the developer network should have ZERO access to a live environment, ever, even if they have the correct credentials 3] Have working backups, test them regularly. Have backups online and offline in multiple locations 4] Have a release manager or similar code-review or sense check *all* changes which could be deployed to live

Real world

[Anonymous Coward]

In the real world the dev will be fired because the person that gave the ok should be fired. They will be trying to cover their asses by getting rid of the dev and trying to explain it away. It won't work unless their are more dumb asses up the higher chain of command.

NO QA ?

[OppMan29]

I guess thats the price to pay for not having QA dept test code in lower

Dissimilar situation

[trewornan]

I won't state the company but a similar accidental deletion of a huge number of records from a database occurred at a company I was working for. This was a DBA who'd been working there for a considerable time, unfortunately when he realized what he'd done he edited the logs to try and hide who was responsible. Naturally he got fired but (at least from the gossip I heard) the principal reason for firing him was the log business - whether he would have been fired for the original mistake was much discussed but nobody really seemed to know the answer. Incidentally - it took nearly 12 hours working completely through the night to figure out exactly what had happened and restore from backups, although there was still some data loss even though we had pretty good back up provisions.

If your IT systems can't recover.....

[modmans2ndcoming]

If your IT systems can't recover from catastrophic failures like this then you suck as a CTO If your development practices don't involve integration testing then you suck as a CTO If you allow a new employee with no experience to commit changes to production then you suck as a CTO

Count yourself lucky

[Anonymous Coward]

If the company was running without a working backup on their production database, then it was just a matter of time before they would have a similar problem with payroll or some other "critical" component of their business.

You have a question to ask your next potential employer. "When was your last successful full backup of your production databases?"

Shit Happens

[Anonymous Coward]

I did this on my second month on the job or so. Well, I slipped on the keyboard after partially typing a command in the shell.

The rest of the team sure was mad when they found out I had deleted everything on the system drive. They messed around all day long before they'd let me back on the system. One command and I recovered everything. The rest of them were all analysts, programmers and managers. They had no clue and were about to restore from week old tapes. I had spent the morning rading the The Vax VMS manual.

The moral of the story is that shit happens and sometimes it pays to Read The F**king Manual.

Re: Old News...

[pikine]

About finding the right person to fire, I suspect that this is an unfortunate but complex choreograph of events such that no single person is at fault, and everyone is doing their jobs.

Junior programmer did the right thing about unit testing: wipe the database between tests to ensure consistent result no matter the order the test cases are run. Senior programmer did the right thing and wrote detailed documentation. The management needed a paper trail for production credential, and the documentation is probably the only SOX compliant way the company has, so the management did the right thing and stored production credential in a SOX compliant way. Production backup is not working but someone is actually working on getting it to work, and they are having trouble with replication and consistency; they want data integrity assurances rather than having backups that could have silent data corruptions.

Everyone is doing their jobs in good faith, and probably doing it well. I think the disaster could have been prevented if someone looked beyond their job responsibility and noticed potential problems, but such busybody could readily cause ire in a hierarchical organization. I think the best thing a company can do is to encourage people to be nosy and speak up. The CEO has to make this part of their corporate culture.

Re: Old News...

[Anonymous Coward]

This. I'm the senior engineer at work, I've worked there 18 years, I don't have production access, I don't want production access.

Giving that sort of access to a new hire, particularly a recent graduate is beyond stupid.

Dissimilar deletion.

[Anonymous Coward]

Wouldn't it make sense that before any deletion the date would be copied somewhere safe for 24 hours? After that then delete the copy. Be a lot less regrets.

Real Problem

[Anonymous Coward]

The real problem is lack of backups. The question then is - who knew? Who should have been fixing it?

Perhaps CTO is pissed because (if he's like the pointy-haired boss types I had the fun of working under) he ignored the gravity of the situation until someone showed him why he should not have. If someone hid the bad backup situation from him, they should be biting the big one.

What if, instead of a cut-and-paste, it was a server crash, or a full-on server room disaster. It's the CTO's job to plan for these in any organization big enough that it has a guy with title CTO.

FTFY

[Anonymous Coward]

Developer Accidentally Deletes Production Database On Their Last Day On The Job.

FTFY.

Same thing happened to me

[Anonymous Coward]

Yes, the CTO should go, but when your on the bottom rung you're the easy scapegoat. He fucked his company - you're just the mechanism, but data loss is a risk he clearly didn't prepare for. My error was rebooting a financial system because of a one letter typo. Yes, day 3 of new job and this firm had me bouncing mission critical systems without passwords or authentication.

"First day"

[slipped_bit]

"EXPLAIN YOUR ACTIONS!"
"Um... It's my first day?"
"HAHAHAHA, OK, carry on then."

A backup isn't a backup

[Shatahn]

A backup isn't a backup unless you've practised restoring from it. Giving a newcomer privilege to production services straight away is also silly. Is this another one of those hoax "Ansible deleted all my user's websites" stories?

I smell clickbait B.S.

[Anonymous Coward]

However, if it is true -- as written -- then this kid should lawyer up and sue the company. The execs will be forced to settle for low six figures just to keep their own incompetence out of the legal record.

Unix and databases have mechanisms for access cntl

[Douglas Goodall]

In the old days, large unix machines had serious system administrators and a general mix of users and developers. The root password was closely held and users were given access on the basis of their need. There is no doubt that fully utilizing user group and world security at the os level, and granted accesses to data in databases, is a way by which users and developers can be given appropriate access. Back at the Western Bancorp, we had a production mainframe and a test mainframe. Quite an expensive scenario to manage the creation and migration of new code releases through QA levels. These days an extra test machine costs significantly less, sometimes the cost is negligible. In this circumstance, the company has bigger trouble than just a lost database. I would think this company needed a database manager whose responsibilities included assurance that the data will not be lost. I can only hope this was not a publicly traded company.

By Neruos

[Anonymous Coward]

Not all companies have micro segregation of duties.

1. If this person is a developer and they are working/fixing/supporting the production system, then he should have access. the question isn't, Why does they have access, that isn't the issue.

2. why did they delete, this is a joint accountablity problem.
A. as a new hire or a junior, there should have been a senior shadowing, they should have had a Change Management and SDLC process in place.
B. as a new hire from collect, his education has let him down, this is basic tech foundations. (before you make a change, backup), that is the gospel, done.

So, while I do blame the company for not having good processes/practices, but I also blame the employee.

It's an expensive lesson to learn and since both are to blame, both should feel the burn of the lesson. This is how we learn, science.

scapegoat...

[Anonymous Coward]

What might legal be telling that CTO now? Court action by the company will surely bring scrutiny. Management, investors and customers might be concerned by how the company was defended and prepared for such a simple mistake. I wonder what an employment lawyer might make of such a firing?

Not the fault of management in any way, of course.

[Anonymous Coward]

We fired the kid. Problem solved.

Re: Old News...

[Quirkz]

Is posting the obligatory like doing the needful?

Quacks Like a Hoax

[StikyPad]

This story has all the hallmarks of a hoax. It's semi-plausible on its surface, but how many companies would have leaked production credentials in a config document AND had production machines accessible from a dev network AND had backups without any integrity testing, let alone multiple redundant backups AND had zero incidental copies? That's violating a slew of helpful best practices but none of the potentially harmful ones, like locally mirroring production data for development use. This sounds more like someone's worst fears of their first day presented as fact, but the details don't add up. It involves a strange blend of strict adherence to some policies with rampant noncompliance to others, and incompetence by a lot (nearly all) of the existing employees, managers, and executives. When things fail big, it's because of a series of events in corner cases, not

Re:Quacks Like a Hoax

[StikyPad]

...hitting submit before you finish composing your post. :)

Throw under the buss

[dougg76]

I have seen this in much larger shops. They build their systems in a non robust way because they are cheap and just throw people under the buss when it goes bad. Companies try to implement the cool side of continuous dev without paying the dev ops price. It is just the same old same old, lets demand the good parts of a paradigm, but ignore the requirements to make such a system viable ex: continuous development, agile etc.

Having IT work on brittle systems just offsets stress to the devs / techs and makes their jobs suck. I once brought this up to a manager and he said something along the lines "Those guys who work on skyscrapers don't work with harnesses..." they do, and have been for a long long time. It is not the f'ing 1930s.

Re: Old News...

[Anonymous Coward]

Fuck the new guy. If this is what he did in one day, imagine what he would do in a week

Re: Old News...

[RockDoctor]

I'm not sure we'd have used those titles, but I'm pretty sure someone is getting demoted

NEW HIRE : "Hi, I'm a new hire. Where do I go?"

RECEPTIONIST : "Hi, I'm the Vice President in Charge of Avoiding a Repeat of The Disaster of May 2017. Follow me, and do what I say, and not what I do!"

NEW HIRE : "I just bet there's a war story behind that job title."

Fishy

[DarthVain]

Not only exactly what you said, but the fact that they have no backups (or they were not working) is also pretty scathing. I've worked with some experienced developers that know our systems for 15 years, and they never ever have any access to production! Never mind they should have DEV and TEST instances as well, probably logging depending on what they are using...

Makes me wonder if their IT was that inept, or if there was some serious financial or other more fundamental technical issues that they needed to go away, and this guy is simply the scapegoat... LIke the CTO was embezzling from the system and needed a reason for all evidence to disappear. Or someone senior with real access screwed up very badly, and again this guy is getting the blame. Seems pretty unreasonable and unrealistic that something like this could even happen. Anyway it all sounds very fishy to me, I'd bet there is something else going on.

Re: Old News...

[david_thornley]

If everyone is doing the job they're supposed to do, and a disaster like this happens, fire the person who assigned the job roles.

The Developer is Clearly at Fault

[lagunastarman]

Clearly, the Developer is at fault. He should never have accepted to work at such a screwed up company; a dumb CTO (who blame shifts), poor security, no offsite backups, bad HR, terrible policies. ... When you start your first job you are expected to know EVERYTHING about what you are getting into on the first day of your first job ... yeah, right! My serous advice is for this guy to get a lawyer and sue the hell out of the company!!! I know IT Directors that have transferred from one career to another and not been granted admin access for 6 months - even after extensive background checks. In this case, the one most innocent is the brand new, new hire.

How to handle this as an EVIL Mastermind

[Anonymous Coward]

This new guy was just way too nice and just too new in to the work place. Here's how I think he should have handled this:
1. Tell the CTO: "No you won't fire me. In fact, you're going to give me a raise..."
2. One little typo causes a massive data loss? Come on... Briefly explain to CTO that you know it was his incompetent IT infrastructure which was really to blame for this ticking time bomb. In fact, it would be career ending for the CTO if this story got leaked to the world along with the name of said CTO... What a shame that would be.
3. Now explain, that "you don't want to be greedy, just reasonable. How much is your career worth to you?" Don't laugh evilly after you say this. It's not professional.

Re: Old News...

[Anonymous Coward]

Yeah, help point out tons of other fuck ups at the company. If backups aren't running you've got to treat that like the production environment being down. Its like having a simple RAID array running with a failed drive, you HAVE to get it fixed before another one goes down. And if the senior level staff thought it was OK to run in a degraded state for a few weeks or months, then that is on them when the next drive fails - or in this case when the production environment goes down

Proxcimate cause, root cause

[eric_harris_76]

The proximate cause was the new hire who screwed up on his first day.

But the root cause was a lack of control over their shop. This company was a disaster waiting to happen. The newbie was guilty of walking too fast past a house of cards.

As others pointed out, it should not have been possible for him to do that.

What are developers -- novices or old hands -- doing with that kind of access to production data? A company big enough to have someone called a CTO ought to have separation of authority, so those sorts of things can't happen.

He was a rookie who made a mistake with serious consequences. The alleged professionals already on staff made it possible for one mistake to perhaps destroy the company.

I'm guessing it's not a publicly traded company subject to Sarbanes-Oxley.

what was wrong

[Anonymous Coward]

Hello.
I'd like to suggest one way this could happen - the production database was once a test database of the project and when it got OK, probably under the pressure to hurry from the big bosses it was assigned production database - and remained so with no changes to the initial technical documentation.
Also I'd fire the junior anyway - for incapability to read and follow technical documentation properly. This is a crucial must have skill.

Sounds like a Scapegoat...

[goolac]

Too many issues here to miss. 1) New employee with access to LIVE data. 2) Copy/Pasted code... (implies company provided the code...) 3) Conveniently mentioned backups not working 4) Mention of Legal action. This poor sap was hired to be a scapegoat. They probably had issues meeting production goals or some sort of milestone and it was a convenient happening that a new tech guy broke the whole damn thing. Unfortunately it is hard to prove who knew what but. In my opinion the CTO should be the one who legal action should be taken against. The company should protect the new hire and isolate him from the CTO and immediate supervisors to investigate the job/code/work he was doing and who told him to do what. Also they should investigate who gave him access to LIVE data and Admin level controls of that data. Although I suspect that the this was a crime against the shareholders/investors and the whole management team should be held accountable. Just my $.02

Sounds familiar...

[iq145]

https://tech.slashdot.org/stor...

Re: Old News...

[tom.apostolou]

As a database expert I have to say, no backup means there is no database, no security and access control (developers should never have access to production) no database, no high availability solution means your database is not critical. So yes everyone else except the junior developer should be fired, if anyone should be fired.

Who should get fired

[Anonymous Coward]

1) The person how gave permission on the database
2) The person who was incharge of backups
3) The Executive who created an environment, where an idot can take out the company on his first day.

Re: Old News...

[Zontar The Mindless]

Good to see you posting again. And thanks for posting a cogent response to my feeble attempt at a funny.

Re: Old News...

[KGIII]

You did make me giggle but my verbosity is an innate trait. It's what I do. I did have some sorta personal issues, including a new love interest, but I am alive. I had a sibling pass away, pretty much at the same time someone decided to ruin my motorcycle by smashing into it while I was going the opposite direction. I don't know if you saw the pics, but it was kinda ugly. I survived but I must have thought I was Superman. I don't recall it, but I pretty much tried to punch said car with both hands. The car won.

The idiot that pulled into my lane, to pass someone else, wasn't prosecuted. I am okay with that. Putting them in jail would help nothing. Insurance covered it all and I am actually looking at a new bike. I love my BMWs but a friend has Sportster for sale.

If I can figure out how to get pics of the x-rays, I'll share them. It was just my hands and my right wrist that were screwed.

And then a sibling died. And that is kinda where I went off the rails. I can't say I didn't have a good time, but it was not really something I needed to do. At least I didn't get any new tattoos.