Slashdot Mirror
When Sysadmins Go Bad
An anonymous reader writes "Here is a story about what can happen when you think you're being oh so clever. This sysadmin planted so-called logic bombs on the systems he was responsible for and then quit. He also tried to game the stock market, buying put options on his former company, hoping to cash in when the disaster he engineered struck. Who can companies trust if they're afraid that this kind of thing can happen? How can they prevent it?"
Obviously, in the sake of security, you should NEVER provide system administrators with dangerous tools such as root passwords!
Seriously though, security is a very delicate matter which is entirely built on trust.
Ways to improve security is to limit access to only what you actually need to use. In the case of system administrators and the like, it's not quite as easy as they obviously need a high level of access.
One solution would be to have third party audits of the systems, perhaps with read-only access in order to prevent tampering, but even then you need to trust the integrity and skill of the auditors.
Another thing to remember is to have a solid disaster recovery plan, but that's only good AFTER something happens and the person designing and implementing this plan will likely be the person that has the most access.
There's no universal answer to this problem. If I knew of one, I'd be rich as heck from selling it to companies.
Many years ago one of our staff left at the end of the summer. Our boss said "Thank you very much for working for us ... [pause as the door closed, then turned to a coworker] ... delete his account."
Slashdot monitor for your Mozilla sidebar or Active Desktop.
Have two sysadmins, who work in different areas, and who a la "missle key firing system" both have to approve additions to important code bases.
Obviously, you could get two bad apples and have the same thing happen, but odds are slim.
Problem is, it tough to find ONE good admin, much less two, esp. with tough times for business... having to dole out twice the budget to protect yourself "just in case". Then again, it would double the job market =)
OR mabye CVS everything, and look through all changes an employee made after they quit... then again, the clever get around this, etc.....
*sigh* People just suck sometimes.
Department of Homeland Security: Removing the rights real patriots fought and died for since 2001
When you have reasonable salaries, reasonable work hours, and no one that runs everything.
First of all you'd have less disgruntled employees.
Second, you'd have less disgruntled employees.
Third, you wouldn't need to trust anyone 100%. Most egos of sysadmins wouldn't let them let someone else compromise their system. If you have 2 or more admins 100% responsible for the integrity of a system, and each performing checks on each other, you would reduce the occurences of these types of attacks.
> Who can companies trust if they're afraid that
> this kind of thing can happen?
Nobody.
> How can they prevent it?
They can't.
Employee misbehavior spans an entire spectrum of seriousness, from stealing paper clips to embezzling billions. You can't prevent a determined and dishonest sysadmin from sabotaging a system any more than you can prevent an accountant from diverting funds or an after-hours custodian from taking things off peoples' desks.
There is no panacea, technological or otherwise.
Preventing employee misbehavior has several parallels with Copy Protection. No affordable and practical scheme is bulletproof if the person is determined enough, so the best method is to remove the motivation. The same rules apply to all employees: treat and compensate people fairly and they will be less likely to want to hurt you.
But even that doesn't work in all cases. If your staff is large enough there will always be people who feel that you are mistreating them, or underpaying them, and who will feel compelled to get what is "rightfully theirs" in other ways, large and small. And many people steal/etc. without regard to the harm it causes the company or other employees; their motivation is purely selfish, so it doesn't matter how well they are treated and paid.
So even if you treat and compensate people fairly, and trust everybody you hire, you must monitor people's activity, investigate suspicious behavior, and, when necessary, prosecute wrongdoers to the fullest extent of the law.
I probably sound cynical, but I speak from experience.
How is this different from any other kind of sabotage by employees or ex-employees? As long as there have been accountants, there has been embezzlement. A short-order cook could forget to wash his hands. A construction contractor can use sub-standard building materials.
You gotta trust somebody; just make sure it's somebody worthy of trust.
As for preventing this particular kind of sabotage, use the same principles as everywhere else: supervision, audits, bonds, insurance, and the threat of jail time if the rest fails. Oh--a good disaster recovery plan sure doesn't hurt, either.
Cheers,
b&
All but God can prove this sentence true.
SysAdmin, as the word says, it's the Administrator of the System.
there's no technical way to restrict their actions, or we should restrict the computer's capacity.
people do bad things for money, that's all, how could we prevent this happen? how could we prevent crime? how could we prevent people shoot each other? these are analog.
it's political or human issue. not technical.
Don't keep disgruntled employees or employees that you keep hidden away in a back room and ignore. Management that keeps good relationships with its employees don't have as many problems with this sort of thing.
This means:
1) Help work to keep employees happily employed (not with bribes - with real career paths, personal interest, etc.). If you keep wage-slaves, expect mutiny.
2) Actively replace employees who can't be kept happily employed. Get others who are competent and glad to have the spot (which shouldn't be too hard in this economy). Keeping people around who don't want the position isn't doing them any favors. If no one who would be qualified would also be glad to have the spot, rethink the position.
"Management" should be helping manage situations like this. If this guy had been disgruntled for a long time, it seems to be their fault for keeping him (and keeping him unhappy and ultimately vengeful). Sounds like someone did a bad job at people-management . . . sounds like the type of willfull neglect that is inexcusable but all too common. Many people think that "management" is watching the bottom line -- that is a lazy, oversimplified way of looking at an important job.
If systems are so critical and secure, then you need to separate responsibilities, and dispense information to those holding the keys on a need to know basis.
--- have you healed your church website?
It is not equivalent to a real bomb. There was no destruction of property, no casualties. It's in a completely different league. The real solution here is to treat your employees with respect and not treat them as slaves.
yeah, but the difference is, the sysadmin is a criminal, a CEO that's stealing is just unethical...
Wax-Museum Fire Results In Hundreds Of New Danny DeVito Statues
> and a huge fine (something like $60,000, which
> was a lot back then).
Wow. I must not be making enough money, because I think that is still a lot.
TANSTAAFL.
>> How can they prevent it?
> They can't.
They can at least reduce the chance a lot with redundency.
If you have a team of sys-admins, you have a good chance that the other might catch the bad one before it's too late. And if they feel treated well by the company and don't share the sentiment of the saboteur, the damage is usually contained.
Another policy I've seen in some banks is that all employees have to take 2 continuous weeks paid vacation each year (the rest of the paid vacation time can be distributed at will). This promotes cross-training and redundancy.
All of this costs money, but think of it as cheap insurance, compared to the cost of rogue sysadmin. Is it worth penny-pinching on salaries and benefits, while maxing out the workload if that results in disgruntled employees who timebomb your systems as they head for a new job?
If you paid the sysadmins $1 million per year, there would be zero theft, zero funny business, and zero turnover. Of course, nobody can do that and stay in business. At some level less than $1 million and higher than fast-food wages, you can retain decent people and discourage malicious tactics. The key to avoiding a technological meltdown is to treat people well enough so that your recruiting process lets you avoid the marginal candidates. Once hired, a properly compensated person should feel as if the "have something to lose", and therefore you can expect such a person to act as a professional. Paying hamburger wages and putting a person in the sysadmin seat would be like staffing a nuclear power plant control room with random selections from the phone book.
This is a very interesting topic, especially right now. We are in a down market, and there is an irresistable temptation for some employers to make lowball offers to currently-unemployed candidates. This allows the employer to cheaply refill vacancies (or exert leverage against current employees). Those employers who are gung-ho about bottom-feeding are setting the stage for big trouble later. Employee turnover is just the tip of the iceberg.
You know, this story was fairly well reported as this type of technology story goes... until they got to this part:
Duronio's logic bomb, the government charged, deleted files and led to $3 million in costs for PaineWebber to assess and repair the damage.
To which I say Bullshit. If $3 million was done by this thing, it's their own damned fault for not having a backup system, and I'm sure they DO have a backup. There is no way that there was $3 million in damages done, because they should just have needed to load their backup. Sure, they would have needed to audit their code to find the crap he put in there, but that couldn't possibly have cost $3 mil.
You must not be a sysadmin...Or you must be working for the government?
This is unrealistic. When the fire is burning, you can't take 5 minutes to sit down and follow the procedures, you just jump in and fight it.
-- Leeeter than leet
You can't. Next question.
"Be kind to your enemies; be peaceful. But if they lay a finger on you, send them to the cemetary."
I presumed you're the type that think that corporate CEO who looted pension fund shouldn't get any time in jail, since they didin't actually use physical violence?
...is 20 years in prison. It doesn't hurt to have national press coverage of the guys who have tried this and have failed. It's not like you can get away with this very easily.
Let's see? Who has had access to all of these systems? Who has recently quite or been fired? Who just sold a boatload of stock when we got hit? A smart admin realizes that there are other admins as smart or smarter. People can piece these things together, and obviously this employer and the government are taking this crime very seriously.
RP
20 years seems harsh only when viewed in the context of a "victim-less" crime. However, most white collar crime has the potential to affect a larger number of innocents than most people consider.
Consider the consequences of an irrevokable malicious act on a trading company. If damage is broad enough the perp shuts down said company for days on end. Thousands of clients are unable to do anything during this time. Employees waste thousands of man hours attempting to rebuild wasted systems. If the damage is extensive enough, it could put the entire company out of business.
Just take a look at the fallout of the Enron situation and you'll find countless people who have lost entire life savings because of some "victim-less" white collar crimes. Not only is Enron dead, their consulting firm has died, thousands of people are out of work, numerous support companies have gone under, and thousands of people have lost millions upon millions of dollars in retirement savings. The consequences of Enron's illegal practices touch many people who did not have anything to do with the crimes being commited.
Don't assume because a crime doesn't physically harm someone that it has fewer consequences or requires lesser punishment. In the broad perspective of total social impact, white collar crimes have the potential to an aweful lot of harm to a large number of people.
As a rule I never delete an account or remove user identification information.
Nuking an account kills part of your auditing trail and/or proper file associations when you do it. Besdies, if you need to check something as a specific user it can be a bear to undo the dammage. Temporarily suspending access can happen just as often depending on the environment, so why not simplify it to one process?
Besides the practical option of re-enabling the account if the person comes back, disabling accounts is a good habit preventing nasty problems fixing mistakes (John Smith vs. Johan Smith).
A firewall can not protect you from yourself. Turn off what you do not need. Do not use the firewall to do your work.
... which is why the SEC investigates any large options purchases which occur shortly before large short-term movements in stock prices. If you're one of these lucky devils, they will probably get your name and address from your broker and see if you are employed by the company in question, if you work for a law or accounting firm retained by that company, if you have the same last name or home address as someone who works for the company, etc., etc.
There is nothing sinister about this kind of investigation; it's routine police work. (Likewise, if you're the town layabout, and the day after a masked man robs the town bank you start spending money like it was going out of style, the sheriff will probably peg you as a suspect.) What is amazing is that people do not realize that it is the SEC's job to do this sort of investigation: they just blithely go ahead with their stupid criminal plans. Even lawyers, who ought to know better even if they are unwilling to behave better, do this sometimes.
The perfect inside trader would have 10 loyal friends located around the country willing to make small purchases of options on his behalf, to forward him all the profits, and to stonewall the SEC investigators who come knocking. Believe me, you don't have 10 friends like that.
Part of the problem is "lone ranger sysadmins". No serious system should be vulnerable to the whims of a single individual with the root password. The root account should only be allowed to activate if two separate passwords are typed in (one for each person). You can have a pool of admins each with their own password, but at least two of them would be needed to log in as root. You then require via company policy that for the duration of the session that both persons are present for the work that needs to be done.
You still need some sort of emergency brake so that a lone admin can stop a haywire system from further corrupting itself, but to actually fix or change the system there should be oversight.
At the same time, forcing two people to do this work means that you get all the other advantages of pair programming: 1) two heads are usually better than one, 2) two people are now familiar with the status quo, 3) less mistakes due to simple errors (as one person can catch typos, etc, before they're committed to disk), 4) others? There is plenty of documentation that programming in pairs is a highly successful strategy, and I suspect that it's a good idea to do major systems administration in pairs as well.