Slashdot Mirror
When Sysadmins Go Bad
An anonymous reader writes "Here is a story about what can happen when you think you're being oh so clever. This sysadmin planted so-called logic bombs on the systems he was responsible for and then quit. He also tried to game the stock market, buying put options on his former company, hoping to cash in when the disaster he engineered struck. Who can companies trust if they're afraid that this kind of thing can happen? How can they prevent it?"
Obviously, in the sake of security, you should NEVER provide system administrators with dangerous tools such as root passwords!
Seriously though, security is a very delicate matter which is entirely built on trust.
Ways to improve security is to limit access to only what you actually need to use. In the case of system administrators and the like, it's not quite as easy as they obviously need a high level of access.
One solution would be to have third party audits of the systems, perhaps with read-only access in order to prevent tampering, but even then you need to trust the integrity and skill of the auditors.
Another thing to remember is to have a solid disaster recovery plan, but that's only good AFTER something happens and the person designing and implementing this plan will likely be the person that has the most access.
There's no universal answer to this problem. If I knew of one, I'd be rich as heck from selling it to companies.
Many years ago one of our staff left at the end of the summer. Our boss said "Thank you very much for working for us ... [pause as the door closed, then turned to a coworker] ... delete his account."
Slashdot monitor for your Mozilla sidebar or Active Desktop.
Have two sysadmins, who work in different areas, and who a la "missle key firing system" both have to approve additions to important code bases.
Obviously, you could get two bad apples and have the same thing happen, but odds are slim.
Problem is, it tough to find ONE good admin, much less two, esp. with tough times for business... having to dole out twice the budget to protect yourself "just in case". Then again, it would double the job market =)
OR mabye CVS everything, and look through all changes an employee made after they quit... then again, the clever get around this, etc.....
*sigh* People just suck sometimes.
Department of Homeland Security: Removing the rights real patriots fought and died for since 2001
When you have reasonable salaries, reasonable work hours, and no one that runs everything.
First of all you'd have less disgruntled employees.
Second, you'd have less disgruntled employees.
Third, you wouldn't need to trust anyone 100%. Most egos of sysadmins wouldn't let them let someone else compromise their system. If you have 2 or more admins 100% responsible for the integrity of a system, and each performing checks on each other, you would reduce the occurences of these types of attacks.
I was disappointed to find that this was an article, and not a new show on Fox.
It's better to burn out than to fade away
> Who can companies trust if they're afraid that
> this kind of thing can happen?
Nobody.
> How can they prevent it?
They can't.
Employee misbehavior spans an entire spectrum of seriousness, from stealing paper clips to embezzling billions. You can't prevent a determined and dishonest sysadmin from sabotaging a system any more than you can prevent an accountant from diverting funds or an after-hours custodian from taking things off peoples' desks.
There is no panacea, technological or otherwise.
Preventing employee misbehavior has several parallels with Copy Protection. No affordable and practical scheme is bulletproof if the person is determined enough, so the best method is to remove the motivation. The same rules apply to all employees: treat and compensate people fairly and they will be less likely to want to hurt you.
But even that doesn't work in all cases. If your staff is large enough there will always be people who feel that you are mistreating them, or underpaying them, and who will feel compelled to get what is "rightfully theirs" in other ways, large and small. And many people steal/etc. without regard to the harm it causes the company or other employees; their motivation is purely selfish, so it doesn't matter how well they are treated and paid.
So even if you treat and compensate people fairly, and trust everybody you hire, you must monitor people's activity, investigate suspicious behavior, and, when necessary, prosecute wrongdoers to the fullest extent of the law.
I probably sound cynical, but I speak from experience.
Something similar happened to my Dad's business about 15 years ago. Back then, they just trusted the employees. For some reason I can't recall, they decided to fire the sysadmin that was running their billing systems and gave him a months notice. During that month, they let him take time off from work to interview at other places and were generally pretty nice about the whole thing.
A couple weeks after he left, the system started crashing and losing data. Apparently he used a rather well-known bomb because the company they used for support was able to dial in and found it rather quickly. He was charged, arrested, tried, and found guilty. It was a big deal because the state (South Carolina) had just passed some really though computer crime laws at the time, and the Attorney General wanted a "test case" for the law.
My Dad and his partner's requested that the guy not get any jail time since he had a wife and some kids, but he got major probation and a huge fine (something like $60,000, which was a lot back then). Plus he now has a felony charge on his record. Last I heard, he had gotten out of the computer biz and was working in a family business.
Anyway, the short lesson is: if you're a company firing someone with privileges, pay them the two weeks or whatever but don't let them back on site. And if you're the guy getting sacked, don't try to get revenge through sabotage; it's just not worth it.
As an aside: every place I've worked had a policy that whenever someone was fired they were led to their desk with a cardboard box, then escorted out of the building that very moment.
... pull a stupid crime and spend the rest of your life in a state-funded institution.
For critical systems, nothing gets changed without an approved change request. All changes must be examined, tested and approved by someone other than the programmer. You can also have a separate group to maintain the source libraries and to do builds.
Mea navis aericumbens anguillis abundat
With the Paine Webber guy, I was amazed this guy didn't think the SEC could put 2 and 2 together.
"Hmmm, there's the guy who had access to the company's computers and made all those put options, but I don't know if there's any way we can prove motive or opportunity."
Trust in God; Everybody else pays cash
Who can you trust? -- Nobody. As our master said:
Machievelli, The Prince Ch 17.The answer to the question is no one, not even your mother. If you are not secure against being hacked by an insider, you are not secure. And that means everybody, Newspapers are full of headlines about CEO's ripping off their companies. Stories about long-time trusted employees who embezzle a few hundred thousand dollars are so common that they usually wind up on page 7 of the Metro section.
SysAdmin, as the word says, it's the Administrator of the System.
there's no technical way to restrict their actions, or we should restrict the computer's capacity.
people do bad things for money, that's all, how could we prevent this happen? how could we prevent crime? how could we prevent people shoot each other? these are analog.
it's political or human issue. not technical.
Don't keep disgruntled employees or employees that you keep hidden away in a back room and ignore. Management that keeps good relationships with its employees don't have as many problems with this sort of thing.
This means:
1) Help work to keep employees happily employed (not with bribes - with real career paths, personal interest, etc.). If you keep wage-slaves, expect mutiny.
2) Actively replace employees who can't be kept happily employed. Get others who are competent and glad to have the spot (which shouldn't be too hard in this economy). Keeping people around who don't want the position isn't doing them any favors. If no one who would be qualified would also be glad to have the spot, rethink the position.
"Management" should be helping manage situations like this. If this guy had been disgruntled for a long time, it seems to be their fault for keeping him (and keeping him unhappy and ultimately vengeful). Sounds like someone did a bad job at people-management . . . sounds like the type of willfull neglect that is inexcusable but all too common. Many people think that "management" is watching the bottom line -- that is a lazy, oversimplified way of looking at an important job.
C'mon -- this is really small potatoes ...
Who can you trust?
Microsoft. Trustworthy computing.
At Microsoft, we make operating systems that administer themselves, so you don't have to hire those untrustworthy and expensive system administrators. Nearly any high-school graduate, or poo-flinging monkey, with the proper brainwa^H^H^H^H^H^H^H training can become a Microsoft-Only Operations Certified Omnipotent Worker. Get your own MOOCOW today, and let us handle your security problems. You shouldn't have to worry about these computer dealies - that's our job.
Microsoft. Trusted Computing since 2002.
My beliefs do not require that you agree with them.
Put option quick explaination:
Suppose that the stock of company FooBar is worth $80 today.
I buy the *option* of selling that stock at $80 in one weeks time (this of course cost me something since there is a risk involved for the entity that I buy this option from).
Let's say that priviledge costs me $1 (since everybody considers company FooBars stock prices to be quite stable).
Now, one week later the "bomb" has blown up their computer system and the stock has plunged to $40.
The option of selling one stock at $80 is now worth $40 since the stock is currently priced at 40$. I don't even have to own the stock since someone who does can buy the option from me instead.
In total I've made 39$ on an investment of 1$ in one weeks time.
Forget the sysadmins hosing the company, how many friggin execs run the thing into the ground looking to pad their stock options, then leave?
At a big EDA firm I worked at the sysadmin got into big trouble (I think he was fooling around on his old lady and was trying to run away with some other chick). He decided to hose the backups by placing a small magnet on the read/write head (IIRC). Then he did real backups, which he hid in the drop-down ceiling. His stupidity led him to try to blackmail the company (gold coins). The episode ended badly--high speed chase, crash, prison. Now that I think about it, yeah, a Fox mini-series!
doug
>> How can they prevent it?
> They can't.
They can at least reduce the chance a lot with redundency.
If you have a team of sys-admins, you have a good chance that the other might catch the bad one before it's too late. And if they feel treated well by the company and don't share the sentiment of the saboteur, the damage is usually contained.
Another policy I've seen in some banks is that all employees have to take 2 continuous weeks paid vacation each year (the rest of the paid vacation time can be distributed at will). This promotes cross-training and redundancy.
All of this costs money, but think of it as cheap insurance, compared to the cost of rogue sysadmin. Is it worth penny-pinching on salaries and benefits, while maxing out the workload if that results in disgruntled employees who timebomb your systems as they head for a new job?
If you paid the sysadmins $1 million per year, there would be zero theft, zero funny business, and zero turnover. Of course, nobody can do that and stay in business. At some level less than $1 million and higher than fast-food wages, you can retain decent people and discourage malicious tactics. The key to avoiding a technological meltdown is to treat people well enough so that your recruiting process lets you avoid the marginal candidates. Once hired, a properly compensated person should feel as if the "have something to lose", and therefore you can expect such a person to act as a professional. Paying hamburger wages and putting a person in the sysadmin seat would be like staffing a nuclear power plant control room with random selections from the phone book.
This is a very interesting topic, especially right now. We are in a down market, and there is an irresistable temptation for some employers to make lowball offers to currently-unemployed candidates. This allows the employer to cheaply refill vacancies (or exert leverage against current employees). Those employers who are gung-ho about bottom-feeding are setting the stage for big trouble later. Employee turnover is just the tip of the iceberg.
My take on it would simply be that your employer did not pay enough attention to your activities abd subsequently due to their mismanagement you would not be at fault. Comments?
--Chag
Here in Venezuela, when the Oil strike begun some sysadmins blocked and placed logic bombs in the critical computers. It is costing the country an average of US$ 15 million a day. The computers that control the fuel-load process in the tankers where so sabotaged that any try to get the system up would end up spilling fuel on every "island" (the place where the fuel truck loads). The only way to stop the spill would be to activate the emergency system in the plant. Gladly (it's already very known worldwide) the goverment set up a "hackers team" to take over all the sabotaged industry computers. Most of them are running Solaris or Windows NT 4, so it wasn't too hard to break all the systems. If you calculate: US$ 15 Millions * 16 days = 240 Million US$ ... and most of it is because the admins who sabotaged the critical computers.
1985: A travel company with several offices (local big group) had only one sysadmin for their computerized booking system. He was this nasty guy who was related to one of the founders, and no one wanted to fire the guy because only he knew how to run the damn things. Not that he did a good job. He was lazy, rude, and demanding. Well, one day, new management got sick of him, and tried to get an "assistant" for him (read "learn his job so we can fire him"). Sysadmin was wise to that, and basically they went through several employees in as few months. Finally, they decided to fire the guy, and hire a contractor to replace the systems. The firing was ugly, they ex-admin had to get dragged out by the police in the end. Days later, the whole system went down. Guess what? No backups. No one knew how it ran, and years of data was lost, chaos among their customers ensued, and six months later the company went out of business.
1996: Our company bought out a competetor. They guy in charge of the call center was the only one we didn't lay off right after the merger because he was the only one who knew what went where, and he used this knowledge to leverage his job security. He was impossible to work with, never did anything on time, never answered his pages, and did just enough work not get fired, but it was really, really hard to get him to do anything else. Finally, we gathered a team of experts (our staff plus vendors) to go as a group, figure out what he was doing, then fire him. His response? He deleted all the call center tables, databases, and destroyed all paperwork... then quit. We had him arrested, but he posted bail, and we never found him again. It took half a month to get everything working right, which meant we had to tell 300 call center employees they couldn't come to work or get paid until we called them back. Boy, was that a clusterfuck.
I saw this button once, "Now that I have changed the master password for the database, it is time to discuss my salary." Heh.
1997: The head of our HR department was fired due to some political bullshit. Standard procedure was to take an ex-employee's computer, wipe it, and give it back to the tech department. Guess what we lost because no one thought about it? All employee records for the department. Backup was on a single floppy that wouldn't load, and she hadn't done backup since the first of the year anyway. We had to have every employee resubmit 1099s and W4s, plus tell us honestly what vacation and sick they already took.
1999: Same company, same situation, but this time it was the guy who kept the entire tech department hardware inventory records. It took a year to recount what we had, and re-enter serial numbers and license keys into a new database. The stupid thing was, this guy made regular backups on the network drive... which was on a server they wiped by accident. Doh!
2001: After a round of layoffs, one of our more brilliant and inspired programmers had "expiration dates" on all his compiled software. He wrote most of the tools we still use today. Months after he was laid off, all of them stopped working on September 17th, 2001 at 12:00 midnight. The only way we got saved was that no one wiped his original desktop box (which had the source code on it, which is how we found out about the "expiration date"). So we recompiled without the date, and everything worked again. Due to WHEN it happened, our whole company thought we'd been attacked by terrorists (the clever generic error only said there was a "network failure") until the truth was revealed. Later we found 9/17 was his birthday, and it was just coincidence it happened so close to 9/11; the layoffs were in March, and they were unexpected and sudden. I doubt this guy had Al-Queda (sp?) connections, so he must have been planning this "job security" (as the comment in the code labeled it) way in advance.
Not quite. You've described a short sale.
With a "short sale" you can borrow stock that you don't own, sell it, then later on, after the value has fallen, buy it, and give it back to its owner. Think of borrowing your neighbour's lawnmower in April when lawnmowers are expensive. Sell it for $200. Then in November when lawnmowers are cheap, buy a lawnmower on clearance for $100 and give it back to your neighbour.
Options (a put option is one of two kinds of option) are a bit different in that you don't actually buy any stock. You only buy *the right* to buy (call options) or sell (put options) the stock at a given price.
What's the difference?
Well, for options, you have a limited risk (it's impossible to lose more money than you put in -- the worst that can happen is that your options become worthless and you throw them away). But with a short sale, the risk is potentially limitless, since it's possible for the stock price to be infinitely high when you have to buy them back and repay the lender.
- In Capitalist America, law violates YOU!
Remember the lesson of "Jurassic Park":
If you don't pay your programmers enough money, a tyrannosaurus rex will eat your lawyer.
He can't. I've had this happen to me one or two times. I've been pushed in to sysadmining (dammit, Jim, I'm a programmer, not a sysadm!) in this small association (about 60 employees, about 60000 members), and initially just assumed the system I took over was OK. After a year or so I discover, quite by accident, the first horrible thing... Every user PC has a small script on it, that contains the root password to the main server in plaintext.
Apparently, no-one knew. I was responsible, even if it was my predecessor (or his) that had written that script. What to do? Go up to the boss and say "Hey Joe! Funny thing, any employee may have had root access to the DB in the last five years! Ain't that funny?". No. Fix it. Shut up.
There were a few almost as horrible things I fixed quietly over the next few months.
I also have to confess that I have did a horrible blunder myself, that has gone undetected. What do you do when you find that a bug in an old program you wrote has lead (over the last six months) to >4% of your members mailing addresses beeing slowly mangled? When membership dues are mostly collected by mail? Which has lead to large losses for the association, and great unhappiness among the members?
Fix the bug, correct the adresses as much as possible, delete the evidence, lie when confronted. That's what you do.
If you discover them in the normal course of business, you explain what you were doing and how you discovered them. Do it on paper, sign and date the paper, keep a copy on your person, send a copy to your boss and whoever else it makes sense to send it to.
If you took it upon yourself to "audit" the system without specificly getting permission, then you probably violated a policy and potentially broken the law. The real answer is "don't do that."
Obviously "good" is tied to "doing what you're authorized to do," NOT "finding things that could potentially be held over someone's head but not yet taking advantage of them.
The company is repsonsible for ensuring its shareholder value is protected from people who violate policies and laws.
Randall Schwartz got a felony conviction- I don't believe anyone argued that he was going to maliciously use the information he gathered, but he violated policy and the current law in that jurisdiction. Exceeding your authority accessing computer systems is wrong. If you want to look around *get written permission* from someone who's authorized to grant it.
I do computer forensics relatively often on behalf of corporate clients. If something ominous happened to a machine you'd just probed that evidence wouldn't do you any good- even if you weren't linked to the orginal problem.
If the work environment is right, go in and admit improper access, explain why it won't happen again without permisson and explain the findings. Otherwise, an unrelated event could put a bad spin on it that could do you real damage.
Paul
http://www.pauldrobertson.com