Slashdot Mirror
When Sysadmins Go Bad
An anonymous reader writes "Here is a story about what can happen when you think you're being oh so clever. This sysadmin planted so-called logic bombs on the systems he was responsible for and then quit. He also tried to game the stock market, buying put options on his former company, hoping to cash in when the disaster he engineered struck. Who can companies trust if they're afraid that this kind of thing can happen? How can they prevent it?"
Something similar happened to my Dad's business about 15 years ago. Back then, they just trusted the employees. For some reason I can't recall, they decided to fire the sysadmin that was running their billing systems and gave him a months notice. During that month, they let him take time off from work to interview at other places and were generally pretty nice about the whole thing.
A couple weeks after he left, the system started crashing and losing data. Apparently he used a rather well-known bomb because the company they used for support was able to dial in and found it rather quickly. He was charged, arrested, tried, and found guilty. It was a big deal because the state (South Carolina) had just passed some really though computer crime laws at the time, and the Attorney General wanted a "test case" for the law.
My Dad and his partner's requested that the guy not get any jail time since he had a wife and some kids, but he got major probation and a huge fine (something like $60,000, which was a lot back then). Plus he now has a felony charge on his record. Last I heard, he had gotten out of the computer biz and was working in a family business.
Anyway, the short lesson is: if you're a company firing someone with privileges, pay them the two weeks or whatever but don't let them back on site. And if you're the guy getting sacked, don't try to get revenge through sabotage; it's just not worth it.
As an aside: every place I've worked had a policy that whenever someone was fired they were led to their desk with a cardboard box, then escorted out of the building that very moment.
... pull a stupid crime and spend the rest of your life in a state-funded institution.
Sheesh exactly, so, what happened here.
1: The sys-admin had enough access to the systems that he could change the configuration and clean up and prevent the changes from being detected.
2:
The company didn't have proper procedures inplace to stop 1 happening.
Examples of good procedures could be.
*Systems provide automated roll back.
*Changes can only be applied through a script that is run by xyz and required GOD access (say knowlage of a password that changes daily)
*System should be configured to audit any changes that take place
*A review process, where by any changes are reviewed by another member of staff
etc.......
the sysadmin was bad the company was useless, I'm not supprised he quit and tried to take the company down.
thank God the internet isn't a human right.
Trust in God; Everybody else pays cash
Who can you trust? -- Nobody. As our master said:
Machievelli, The Prince Ch 17.The answer to the question is no one, not even your mother. If you are not secure against being hacked by an insider, you are not secure. And that means everybody, Newspapers are full of headlines about CEO's ripping off their companies. Stories about long-time trusted employees who embezzle a few hundred thousand dollars are so common that they usually wind up on page 7 of the Metro section.
Could be worse - I've had accounts deleted BEFORE I was let go. In fact, thats how I found out I was terminated - my login no longer worked.
Ethics aside, I have to admire this guys balls!
I'll put my ethics back on and fix the sendmail f'up I made this morning now :-)
Help children born unable to swallow - www.tofs.org.uk
Forget the sysadmins hosing the company, how many friggin execs run the thing into the ground looking to pad their stock options, then leave?
At a big EDA firm I worked at the sysadmin got into big trouble (I think he was fooling around on his old lady and was trying to run away with some other chick). He decided to hose the backups by placing a small magnet on the read/write head (IIRC). Then he did real backups, which he hid in the drop-down ceiling. His stupidity led him to try to blackmail the company (gold coins). The episode ended badly--high speed chase, crash, prison. Now that I think about it, yeah, a Fox mini-series!
doug
Gets my vote. I saw this blow up at my current workplace when a former IT drone's account was deleted (not suspended) as soon as she left the building, without anyone realising it was used as the service account for many things, including the backup server. It took many hours to track down all the things it was used for and to furnish them with saner accounts. I think this probably counts as an accidental logic bomb.
The really sad part of this is tale that it took over a fortnight for anyone to notice in the first place. Weep.
(I'm not part of the local IT department, so I'm blameless with respect to this particular fuck-up. I commit enough fuck-ups of my own without claiming responsibility for anyone else's!)
"Are you being weird, or sarcastic?" said Emma. I said I didn't know because I get the two feelings mixed up.
20 years is sick! I think that it should should be a maximum of 2 months, there was no voilence involved.
My take on it would simply be that your employer did not pay enough attention to your activities abd subsequently due to their mismanagement you would not be at fault. Comments?
--Chag
without anyone realising it was used as the service account for many things, including the backup server
This absolutely screams of bad process design and the blame must go to inept management.
Some suggestions I'd pass along (having learned the hard way the first time, as well have having played on both tech and manager side of the fence):
- use role accounts/contacts, not personal ones: Domain registration, administrative accounts on servers, contact email addresses for company stuff, etc. should all point to a generic role contact or account. It's easy to map these to the appropriate individual accounts, but avoids the hell of deleting accounts when someone leaves. I've had to personally intervene with countless companies that have had their Internet domains registered in an employee's name (individual, not role) and experienced all sorts of nonsense when the employee left.
- require documentation (and if you're a tech, provide it and maintain even if you're not asked): Too many tech folks act as if knowing and not sharing process information, passwords, etc. is job security. It's not - it only ensures that when you go, they'll get rid of you like ripping off a bandaid, rather than offer obligatory goodies (severance, consulting contracts, etc.). I've been an advisor to many of these episodes where some tech had attained too much system control and refused to share it. The slightest demand for special treatment from these techs usually creates a knee-jerk reaction, but in the end, the tech always loses (so what if he downs the company's server for a few days - he just ensures bad references will spread and he'll be unemployable at any real job). Share your information! Document your password. Give copies to your boss. Being open like this creates trust and you'll be rewarded by knowing more things not usually shared, or in the even of a downturn, you'll probably get favorable treatment or even be retained (because they can trust you).
*scoove*
I just want to say that this happens in the real world. I've done it myself, not just to have some stick behind the door towards the management, but to retaliate to any kind of attack (internal or external). Anyone messes with any of the administrative accounts and the entire network of servers starts to protect itself against destruction. Going from re-creating shadow accounts to suspending accounts from which the attack was made. So if some hacker gets in my network and gains some kind of admin access (most likely on Windows boxes) and deletes my personal account (or let's say the backup account), the system shuts of the machine that was hacked and re-creates the accounts. The system works with daemons & services communicating to eachother, watching both unix & windows machine in a multi-national IT environment.
But to come back to the topic: If some fuck-up deletes my account before I do a proper handover of the system (system which is unknown to the management) to the new admin: BOFH.
Here in Venezuela, when the Oil strike begun some sysadmins blocked and placed logic bombs in the critical computers. It is costing the country an average of US$ 15 million a day. The computers that control the fuel-load process in the tankers where so sabotaged that any try to get the system up would end up spilling fuel on every "island" (the place where the fuel truck loads). The only way to stop the spill would be to activate the emergency system in the plant. Gladly (it's already very known worldwide) the goverment set up a "hackers team" to take over all the sabotaged industry computers. Most of them are running Solaris or Windows NT 4, so it wasn't too hard to break all the systems. If you calculate: US$ 15 Millions * 16 days = 240 Million US$ ... and most of it is because the admins who sabotaged the critical computers.
1985: A travel company with several offices (local big group) had only one sysadmin for their computerized booking system. He was this nasty guy who was related to one of the founders, and no one wanted to fire the guy because only he knew how to run the damn things. Not that he did a good job. He was lazy, rude, and demanding. Well, one day, new management got sick of him, and tried to get an "assistant" for him (read "learn his job so we can fire him"). Sysadmin was wise to that, and basically they went through several employees in as few months. Finally, they decided to fire the guy, and hire a contractor to replace the systems. The firing was ugly, they ex-admin had to get dragged out by the police in the end. Days later, the whole system went down. Guess what? No backups. No one knew how it ran, and years of data was lost, chaos among their customers ensued, and six months later the company went out of business.
1996: Our company bought out a competetor. They guy in charge of the call center was the only one we didn't lay off right after the merger because he was the only one who knew what went where, and he used this knowledge to leverage his job security. He was impossible to work with, never did anything on time, never answered his pages, and did just enough work not get fired, but it was really, really hard to get him to do anything else. Finally, we gathered a team of experts (our staff plus vendors) to go as a group, figure out what he was doing, then fire him. His response? He deleted all the call center tables, databases, and destroyed all paperwork... then quit. We had him arrested, but he posted bail, and we never found him again. It took half a month to get everything working right, which meant we had to tell 300 call center employees they couldn't come to work or get paid until we called them back. Boy, was that a clusterfuck.
I saw this button once, "Now that I have changed the master password for the database, it is time to discuss my salary." Heh.
1997: The head of our HR department was fired due to some political bullshit. Standard procedure was to take an ex-employee's computer, wipe it, and give it back to the tech department. Guess what we lost because no one thought about it? All employee records for the department. Backup was on a single floppy that wouldn't load, and she hadn't done backup since the first of the year anyway. We had to have every employee resubmit 1099s and W4s, plus tell us honestly what vacation and sick they already took.
1999: Same company, same situation, but this time it was the guy who kept the entire tech department hardware inventory records. It took a year to recount what we had, and re-enter serial numbers and license keys into a new database. The stupid thing was, this guy made regular backups on the network drive... which was on a server they wiped by accident. Doh!
2001: After a round of layoffs, one of our more brilliant and inspired programmers had "expiration dates" on all his compiled software. He wrote most of the tools we still use today. Months after he was laid off, all of them stopped working on September 17th, 2001 at 12:00 midnight. The only way we got saved was that no one wiped his original desktop box (which had the source code on it, which is how we found out about the "expiration date"). So we recompiled without the date, and everything worked again. Due to WHEN it happened, our whole company thought we'd been attacked by terrorists (the clever generic error only said there was a "network failure") until the truth was revealed. Later we found 9/17 was his birthday, and it was just coincidence it happened so close to 9/11; the layoffs were in March, and they were unexpected and sudden. I doubt this guy had Al-Queda (sp?) connections, so he must have been planning this "job security" (as the comment in the code labeled it) way in advance.
What if the employee is a good guy? What if they have discovered one or more security flaws in the company's systems(s)? Flaws that range from minor (Joe Random customer being able to format a sales terminal) to intermediate (changing employee paychecks or discounting merchandise) to major (stealing the entire payroll account)?
The question: How does the employee tell the company without getting in trouble? After all,the employee did gain... improper... access to the systems to find out this information. obviously, the employee is good or they would have taken advantage of this opportunity, but the company may not see it that way.
So, how can the employee (or anyone, for that matter) handle this?
You must be inexperienced. I've set up systems where no one had root access. You set up sudo (or one of its commercial clones) to give specific people permission to do specific things, then you write a script to change the root password to a very random string and send it to a real printer. As soon as the printer delivers the goods (in the presence of one of more officers), it is folded and placed in an envelope (which everyone signs on the seal) and locked away. Any emergency big enough to require the password needs to be brought to the possessing officer's attention anyway, and anyone can look at the envelope to make sure that it hasn't been tampered with.
Nothing for 6-digit uids?
The truth about proceedures is they are in place to reduce the likelihood of a screwup, to reduce the damage, and increase the chances of detection.
They are never 100%
Happy Fun Ball is for external use only.
This story is about a large company my previous employer did work for. Of course I won't say the company's name, but it's often used as a verb, and their products are probably in your office.
:)
:)
We were hired to write software to show our customer's customer how our customer was doing. It kept track of when shipments went out, things like that. It was replacing an earlier attempt from the sole sysadmin at that location.
Now I must mention that the entire network was 5 years old. Everything was purchased at one time, when the location opened, and nothing had been bought since.
Anyhow, the admin gives us a Compaq P75 workstation with 24MB and NT Workstation to use as our production web/database server. Significantly below our requirements.
He refuses to give us access to their current data to convert/test. Etc, etc. The Manager then gives him the ultimatum to comply or quit, so he walks out. No one there knows any passwords, no network diagrams, not even what boxes do what.
So I had to own every device on their network to give them control again. While writing the software we were there to do originally. Lots of 80 hour weeks, and my previous employer is a bunch of bastards so I was not well paid for it. But to this day, the customer location is still in business, and I have a terrific reference on my resume from them.
A company I previously worked for treated me like absolute crap. Eventually they threw me out and I before they threw me out they let me go clean up my desktop. I copied a "logic bomb" that I had studied out of interests sake onto the firewall and then left. This one required a specific IP/request to set it off, but I never did it, because after I had calmed down it was just too childish and irresponsible. They had been scared however, that I would do something like that and deleted all my accounts, thereby shooting themselves in the foot when they needed to work on the webserver sometime later, I heard from a former coworker. For all I know that bomb is still there today.
I second that motion. Money is only one means of rewarding/compensating your staff. Respect is another one, and one which often is ignored.
I once did a gig as a conslutant for $COMPANY. When the $PHB who hired me introduced me to the SysAdmin, the $SA was visibly displeased. I suspect that
- $PHB had failed to mentioned to $SA that this hire was taking place
- the $SA didn't have a say in the hiring process (he certainly didn't interview me)
- the $PHB may not have mentioned to the $SA that $PROJECT was taking place.
So, when $PHB mentioned to $SA that he needed to set me up with a computer and network account, $SA gave me the list of all of the admin passwords on all of their servers and said I could set up my computer and account myself. $SA quit within a week after I was hired.Needless to say, that was an interesting experience.
And you've never fought a fire.
As a volunteer department, it takes us between two and ten minutes to get to the scene. When we get there, we have to appraise the situation, even before parking apparatus. (What good is an engine if powerlines detach from a home and fall on it?)
We don't make split-second decisions. If you rush, you make mistakes. Even if the mistakes seem minor, people can die. Including you.
You follow every procedure you're taught.
Right down to feeling doors with the back of your hand before opening them. If you forget, you're going to get hit with a backdraft.
Forget to wear latex gloves before treating a bloody accident victim? You better hope they're not HIV positive.
Did you remember to put the spanners back in their mounts? (A spanner is a firefighter's wrench.) If not, how are the people running the engine going to know where to get the spanners to tighten the leaky coupling between the hose and the engine itself?
Did you remember to turn the coupling between that 200psi hose in the right direction, to tighten it? No? I wouldn't want to be in your shoes when it whips around like a possesed snake. (For reference, a 2 1/2" uncapped hoseline expels enough force to accelerate a 50' charged section of hose at 12 m/s^2.)
The bottom line is, you don't come up with a solution to the problem halfway through, you need to spend some time coming up with a plan. For large public locations, like a Best Buy or a Sears, the fire department responsible for the area will usually work out a plan ahead of time for handling anticipatable situations.
What's this Submit thingy do?
In the real world your company should have code documentation standards. Unfortunately most standards seem to focus on compiled code (C,C++) and not php, perl, bash or configuration scripts.
In any case, typically sysadmins work unpaid overtime to meet unrealistic delivery schedules set by marketing or management.
Is it better to have a working system or unfinished well documented code?
Supervisors should set a good example. Peer code reviews and team projects lead to better documentation.
Beware of the lone wolf and loose canon.
For any large production system all changes are reviewed before moving from test to production, but if you have a bad sysadmin they simple won't put their change into the review queue. The only way to spot this sort of thing is to have auditing (built into the OS is best) that the security department reviews.
It is very hard to catch a smart, bad, sysadmin. They have to have the keys to do their job.
What about when you have been working for years with minimal documentation. Suddenly upper management wants you to document everything. Not too suspicious until you consider the amount of layoffs that has been happening recently. On the other hand new equipment is being implemented and there is more time during this slow economy.
So if "The writing is on the wall", do you take your time? Do you procrastinate? What quality do you provide? How much do you let your documentation interfere with your job hunting?
My boss was given this dilemma, right after setting up a W2K cluster. I think he followed the procrastination route. It seems management realized he is still worth what they pay him so they are not bothering about the documentation anymore.
Life moves pretty fast; if you don't stop and look around once in a while, you could miss it. -FB
We got back after one day, and had more than 20 (!) messages on our answering machine. The entire line was shut down because the software was not seeing any new orders. My boss had been going around, saying, "Well, he's finally left. I knew he would do something like that. We're screwed."
Turns out some fool had modified a record without using the proper indexes (ancient FoxPro for DOS). Because the indexes were no longer synchronized, the software's "do while order == opened" loop hit a closed record that was indexed as open, and exited prematurely.
I went in, fixed it in five minutes, and left. They were bankrupt within 4 months, and I was thankfully on to a new employer (that didn't trust employees any further, but that's another story).
You quitting proves that the karma kap worked. The most annoying of the whores shut up. --CmdrTaco
I'm a UNIX sysadmin and Oracle DBA. I've always had root (and sys, for Oracle) on all systems I manage. I've done this for years and have never compromised any data or any system. And I don't think I'm an anomoly. As the admin, I'm very proud of the work I do and the efficacy of the systems I'm responsible for. Employers have extended a trust to me and I wouldn't dream of violating it. No amount of money would be worth the loss of self-worth.
At my last job, I had unfettered access (at work and at home) to thousands of customer's credit card info. It was not even a temptation for me (it was a source of concern that the info might be compromised by others, and I brought that to management's attention on a number of occasions). When the company started layoffs and morale plummetted, I left, but on extremely good terms. The level of trust between us was so high that I was asked to keep my secured access to the system in my home for several months in return for a consulting retainer.
When we were getting new PC's, they let us spec what we wanted. The PC dept prohibited us from ordering the PC's with CDRW's because they were afraid that we would use them to steal company data or code. My boss chuckled when I pointed out that it would be safer and more convenient for me to download said data or code via the company provided ISDN to my house. I just bought a CDRW myself and installed it. Either the PC guys never figured it out or they were afraid to mess we me. Doesn't matter much now, as they are all unemployed anyway.
Hearing about this kind of abuse really pisses me off, it puts us Sysadmins that are legit in a serious bind, and we are less trusted.
The Sys Admins need to form some kind of honor system/group, that puts a code of ethics in place that group members need to follow, If they are suspected of malicious intent during a screening process or on the job, they are banned from the group and can never work in the IT industry again, that's how serious these types of actions should be taken.
Then employers could at lest be assured that we tried to screen out as many plp as possible that are shady.
Anyway just my 2cents.