Predicting User Behavior to Improve Security

CitizenC writes "New computer-monitoring software designed to second-guess the intentions of individual system users could be close to perfect at preventing security breaches, say researchers. Read more." The paper (pdf) is online as well.

Comments

hmmm...

[Britissippi]

Sounds great in theory, however, what happens when users change roles, get promoted, demoted..... and what they have to do with their terminal changes as a result. You'd have to have a staff working full time at any average sized company making the system changes to keep this thing from triggering constant alerts.

Does sound promising though.

Re:hmmm...

[Angry White Guy]

Hell, even without promotions, added staff, etc. Everyone in my office acts irrationally enough to screw the system up completely in an hour or so.
They can't fire us all.

Re:hmmm...

[CoffeeDad]

I'd guess that clearing out the learned habits of any given user, say for example when roles or responsibilities change, would be a rather routine and trivial administration task? Not unlike resetting a password or adding someone to a print queue that's not so far down the hall...

- Just my $0.02

Re:hmmm...

[bmwm3nut]

i don't think they mentioned the method in the article. but i can imagine using something like a neural network to learn the users' behaviors. from my limited work with nerual networks, i've discovered that they're really robust when they learn a problem. it's totally concievable that a neural net could learn irrational behavior too.

promotions wouldn't be a problem either. you have the network have a parameter for the type of job that a user is supposed to be doing. when they get a promotion that job type will change. their new behavior will not be marked as bad until the system learns the new behavior.

of course everything i said is under the assumption that they'll be using neural networks.

Re:hmmm...

[dubious9]

My guess is that it wiil take a statistical look at commands a la Bayesian Spam Plan

After all, probing port looks different than fixing network problems, package manangement/installation looks different than maliciously deleting files, trying to find memory leaks looks different than trying to access another process's memory space. They all us similar commands/system resources, but it should be possibile by look at a few tens of instructions whether a user is try to be malicious or not.

These may not be the best examples but the general idea is that it should be possible to determine user's intent because the probability of a sequence of commands having both a normal and malicous role, should go quite down the more instructions the user executes.

Even false positives should be useful to admins by telling about inadvertant, i.e. acidentally typing rm -rf *,users as well.

Re:hmmm...

[DunbarTheInept]

I have my doubts:

for example: which is the malicious activity?
User A types: rm -rf *
User B types: rm -rf *

(User A was in the root dir at the time. User B was in a subdirectory of his home directory at the time.)

Okay, that's easy- just remember to track the context of where the user currently is. But then what about this?

User A types: rm -rf /shared_network_drive
User B types: rm -rf /shared_network_drive

The difference is that User A was trying to delete everyone's stuff, while User B, knowing how the permissions on the files work, was just trying to find a lazy way to delete those files that he has permissions on because he was trying to clear his own junk out of the /shared_network_drive. He was being sloppy, but not malicious.

How does the software know the difference?

Home Security

[McFly69]

-Note to Self-

Keep doors locked at my house to prevent other people from coming in.

what if

[Diclophis]

the first action you take breaches security?

Arms Race

[queh]

Surely this will just prompt crackers to stealth their actions in commands that are similar to how the system is used normally?

Well, um

[Roadmaster]

if they had any clue about real-world users, they'd know they're absolutely unpredictable. A user's creativeness to mess things up never ceases to amaze.

Re:Well, um

[qengho]

A user's creativeness to mess things up never ceases to amaze.

Or as one of the corollaries to Murphy's Law states: "No matter how idiot-proof you make something, an ingenious idiot in the field will find a workaround."

Sounds good for other people.

[teamhasnoi]

Might be something to install to prevent me from reading /. all day long.

Oh wait. That wouldn't be unusual. DAMMIT!

Gee,

[He Was Gamecubed]

This would work fine, with windows, you know. those 'illegal operations' have a really obvious prompt, it's easy to tell when someone is up to something.

aliasing

[Brandon T.]

Wouldn't it be relatively easy to get around this by aliasing shell scripts to frequently used commands? Sure, the admin might be able to find the shell scripts lying around, but if an intruder was trying to do a one-off attack, it might be viable.

Brandon

Re:aliasing

[halftrack]

I think that's untrue such a scam is not viable. The shell scripts would call commands that get registered by the system and plain alias will only affect the user, the system still sees the original command.

Re:aliasing

[DunbarTheInept]

But what about making new programs to imitate existing ones, but just in a way that isn't noticed by the snooper? (for example: myFuzzySlipperProgram could be a renamed "rm" program compiled from source.)

Or, just do your malicious cracking using system calls from your own C programs. Don't use the rm command in a script, use a program that calls unlink().

To even have a chance of being effective, the system would have to be watching not the commands you type, but the system calls you make. (In unix terms, any time you do something using one of the functions on man page 2, the system library would have to log that.)

Credit card / phone companies...

[monadicIO]

How is the system used by credit card and phone companies different than the one proposed by this paper?

Stifle creativity

[nut]

This would encourage users not to experiment and find new ways of doing tasks, if everytime you tried something new a sysad came round to ask you what you were doing.

Re:Stifle creativity

[Damion]

Keep in mind that the sysadmin can see quite well what the user is doing. The point of this is just to raise a flag if someone does something outside of their daily pattern, not to mark them for inquisition.
All the sysadmin has to do is look at the log and say, "Ah, he's just trying to figure out how to filter his email" and dismiss it, whereas trying to get acquainted with an unfamiliar system and all of its configuration files would be extremely obvious.

Not bad but...

[aridhol]

At first glance, this looks like something that may be useful. However, what happens if a user knows about the system and its patterns, and plans out the attack over a large period of time?

The user could "poison" the information by slowly changing his working habits. If done properly, the AI would probably think this was no different than the user just learning to do things in a different way. When the habits are close enough to the infringing behaviour, the user can probably do anything without setting off alarms.

In addition, if this is the only line of security, the user can then gradually return his patterns to normal. The logs from this system won't show anything. The PHBs may well decide that, when using something as smart as this, traditional logs won't be needed.

Re:Not bad but...

[Damion]

Well, this could never be the only line of defense. Applying patches regularly and maintaining sane security guidelines could never be obviated by an automated system. Think of this as just another level of intrusion detection software. The methods used to stop intrusions from happening in the first, and those to mop up afterward, would remain unchanged.

Re:Not bad but...

[aridhol]

Nothing can ever be the only line of defense. How many PHBs know that? When they see/hear from media/rumours that this is the ultimate defence, how many of them will rush out to get it and tell their IT staff that this is all they need?

Better yet, do this at the kernel level

[pcraven]

See CylantSecure. Run your apps for a while and have it learn your apps typical behavior. Then when something unusual happens it kills off the process. Interesting concept.

Minority Report?

[zoward]

And how long will it be before users start losing privileges for things that they "potentially might do" (with a 94% accuracy rate). About one in 20 of us is really going to suffer for this one.

Not as crazy as it sounds

[Damion]

There are/were some people working on something like this here at CMU. They had posted up bunch of the raw data that they had collected (basically just shell histories with each command run being assigned to a number, and then plotted as number of command (for instance, the 40th command the user entered) against the number value of the command). The results were extremely regular, and in many cases, downright periodic. People are far more predictable than they would like to think.

Intelligent pr0n filters..

[grub]

..are what we need. If someone could come up with a box that could filter pages based on the amount of pink within the images I could delete 80% of my outgoing firewall rules at work!

Re:Intelligent pr0n filters..

[Anonymous Coward]

They use something like this where I work. They have a script that filters all images over XX bytes into a program that then scans for flesh tones within the images. Possible offending images are then forwarded onto an admin who checks the image out and with a few clicks can either add the site the picture came from to the block list, send a warning letter to the logged in user, or both. Does the same thing for image attachments on email.

God I would love to have that guys job!

Re:Intelligent pr0n filters..

[grub]

I assume the other 20% of rules cover the interracial and black pr0n sites?

They're all pink on the inside. :)

These guys should learn from history...

[Anonymous Coward]

computer-monitoring software designed to second-guess the intentions of individual system users could be close to perfect at preventing security breaches

I don't think so... MS software constantly second-guesses users, and decides things for them, and it's pretty much as far from 'perfect' at preventing security breaches as you can get!

These guys have never used MS word have they?

From Clippy, to the damn 'auto-correct' which always decides to turn "MHz" into "Mhz", all they need to do is install MSOFFice, and see how wrong this idea is..

94 percent success rate

[He Was Gamecubed]

"Bruce Schneier, head of US computer security firm Counterpane, says the research is interesting but warns that a 94 percent success rate would be useless at maintaining good security on its own." Well.. 94% x 100 users on the network (.94 ^ 100) = %0.2 chance of detecting all suspicious behavior. Nice odds, i wouldn't depend on it to protect my network, though.

Re:destined to failure

[distributed.karma]

> The Heisenberg uncertainty principle states that there will always be true statements within a system that cannot be proved within that system.

Um, that's Godel's Theorem.

> Wish I was a Physics Genius

I think that just about sums it up. ;-)

Remember that this is network security

[complexmath]

The average user may be adept at breaking his PC, but he's much less likely to, say, flood the network with bad packets.

Obligatory

[scott1853]

Clippy: It appears as though you are trying to hack into an IIS box.

Would you to start the IIS hacking wizard?

Would you like to view a list of the top 1,000 exploits?

Never show this prompt again, its already too easy to hack IIS.

Re:After reading the PDF intently (skimming)

[Sludge]

This seems to stop people from using an account that has access to certain data, which is not their account. If a user usually accesses files with Explorer, and someone sits down at their logged in machine and brings up a command prompt, CDs to the dir, and types 'start .', that would trigger a variant in behaviour.

You could go even further and log a typing rate jump or dip of 30 WPM.

Good idea but doesn't solve the old problem

[Ektanoor]

I took a brief look at the paper and sincerly the idea is not bad at all. However that 94% is pure hype.

The biggest problem in computer security, in what is related to users, is not anomalies, but the usual practice. Remember that experts say that 90% of flaws is due to insiders and not outsiders. And why? Because 99% of these insiders don't care a nail for security. Most of them keep using the wife's name for password and sharing C: to everyone. And no matter the efforts, policies, orders and instructions keep gaining dust. If you try to enforce them then you get a crowd in front of the boss with a rope for your neck.And if even the boss comes up to defend your work, everyone start to mine all your job. All they want is Internet, passing documents and hoping that you finally get out and Microsoft comes in to solve all the problems. That's what the lamers think about security. And in this mess, no matter the expert you are, no matter the tools you have, no matter the hours you loose on the net, you always get trouble every week.

Besides I noted that if someone is going for the break-in, he will mostly go from start. It starts up with this guy "playing" with the computer, then it goes up to the net. Later he thinks he's smart enough to break the server and show that the security admin is a LaMeR.And it ends up with you looking at his desktop and writing the final document to fire or put him into court. You may ask why this guy could go so far. Because he's smart, because no matter the lamerness he is good on something. So the boss will think twice before firing him. If you are in a corporation, then the boss will hang you up with this "unreplaceble" expert because in the city where he lives there's no one else to do his job. Besides, the corporation lost too much money on training him and doesn't want to start from zero on this. So you continue to see the bastard for a few monthes more before you catch him on the red spot.

I saw this and I know that this is a problem on many companies and state institutions around the world. So how this system will help you in such cases? It will, with a large margin of error as the main anomaly, the user, is there from the very start..

Nice in theory but

[JeanBaptiste]

The users I manage are completely unpredictable. Not to sound like a Luddite, but there is no technology that will ever predict what my users do. If there is a way to do it, it will be done. Millions of monkeys with millions of typewriters, and that is a great analogy for what I have seen...

Profiling crackers? Brilliant!

[jabber01]

Sounds like profiling terrorists. It'll work great, and everyone will feel secure and all, until somone flies a plane into their "secure" server.

Bruce Schneier

[elb]

...was recently featured in this article about US security policy, and primarily on the dangers of relying too much on technolgoy. the article is great -- not super-techy, but a great explanation of technology and security policy; it makes an intimidating topic accessible to the intelligent non-tech. a couple of good points from the article:

• "[the leading / best face recognition] software has a success rate of 99.32 percent--that is, when the software matches a passenger's face with a face on a list of terrorists, it is mistaken only 0.68 percent of the time. Assume for the moment that this claim is credible; assume, too, that good pictures of suspected terrorists are readily available. About 25 million passengers used Boston's Logan Airport in 2001. Had face-recognition software been used on 25 million faces, it would have wrongly picked out just 0.68 percent of them--but that would have been enough, given the large number of passengers, to flag as many as 170,000 innocent people as terrorists. With almost 500 false alarms a day, the face-recognition system would quickly become something to ignore."
• "The most important element of any security measure, Schneier argues, is people, not technology--and the people need to be at the scene. Recall the German journalists who fooled the fingerprint readers and iris scanners. None of their tricks would have worked if a reasonably attentive guard had been watching. Conversely, legitimate employees with bandaged fingers or scratched corneas will never make it through security unless a guard at the scene is authorized to overrule the machinery. "

Expected Behaviour vs. Modified Behaviour

[thatguywhoiam]

I would be interested to know just what happens when a user is merely aware that this system is running.

The described system seems to base it's rules on learned user habits; obviously, this strikes one as being incredibly fallible. Assuming their 94% figure is correct for the sake of argument, how do you think *your* behaviour would change knowing full-well that you are being watched?

There are laws in certain places that say a user (in a corporate environment) must be notofied that they are being monitored at that very second. Some software places a pair of eyeballs - how creepy is that - in the toolbar when this occurs.

If the thing's purpose is to sniff out 'suspicious' behavious, I can't see how it could work properly. I mean, how can it sniff out 'motive'?

Hm

[E_elven]

$ r00t machine
Ush: command not found: r00t

*meanwhile, in the Secret Command Centre*
#QUEER#COMMAND##INVESTIGATE##

$ owNz0rz machine
owNz0rz: unknown parameter machine

*SCC*
###THERE#MIGHT#BE#SOMETHING#GOING#ON##

$ owNz0rz r00t
Ush: j00 owNz0r d4 r00t!

*SCC*
#####ALARM#ALARM#####

$
Ush: Someone trying to use 'alarm()', authorize? n0
Ush: Killing alarming process.
$ 1337

"Success" - "false positive" = garbage

[dpbsmith]

Any time someone mentions a "success rate" without also mentioning the false positive rate, they're feeding you garbage

I'd be much more impressed by a claim of an 0.001% false alarm rate than I am by a 94% success rate.

Yet, on a per-line basis, if you assume that a user averages, say, three typed lines per minute, that's 180 lines per hour = 360000 lines per working year.

A .001% false alarm rate means that an innocent worker is going to be interrupted THREE TIMES A YEAR by burly security people at the cube doorway shouting "Hands off that keyboard RIGHT NOW!"

so laik

[digitalsushi]

it'd be like...

while :;do for IP in `cat /var/log/httpd/access_log|awk '{print $2}'`; do /usr/sbin/iptables -t filter -A INPUT -p tcp -s $IP/32 -j DROP;done;done or something like that. Yeah. I got your AI right here. I can sell you a tarball with a digitally signed dignature- I'm quite digil when it comes to being a digilante.

Won't work for certain categories of users

[phsolide]

I don't think the proposed system will work for every one. I think that most workers in development groups will end up getting spanked for what the system interprets as "misbehavior". A developer unit-testing pieces of an application may end up deleting large swaths of files to see how a routine responds to missing files. A developer may write a "dummy server" that just sends streams of random bytes to test how a client process responds to bad input data. Testers may have to reset dates on machines to verify leap year compliance. Testers may make a bunch of files read-only to see how an app handles a log file that has bad permissions.

These are all legit operations - I've done every single one as part of testing or unit-testing in the past. They're also all operations that might be part of a local or remote root exploit.

The Management will have to turn off the profiling for certain users to avoid periodically getting swamped with false alarms or cutting off testing during the final phases of product development.

I have to conclude that it's just more snake oil

Do The Math

[Lucas Membrane]

They claim "up to 94 percent reliable". (You get those emails that say "earn up to $300/hour stuffing envelopes at home"?). "Up to" is a weasel word, just like "arguably", eg it ain't gonna happen.

But, even if it is 94%, if you've got a system that runs around 100 users, then 94% equals approximately 1 million mistakes per year. Where does the budget come from to timely track down 1 million false alarms annually? How is any analyst going to seriously follow every machine-generated warning when 99.99% of the machine-generated warnings are spurious?

Let us now return to reality, which is already in progress.

Solution

[Some Dumbass...]

New computer-monitoring software designed to second-guess the intentions of individual system users could be close to perfect at preventing security breaches

Prediction: Users cause security breaches.

Near-perfect solution: Eliminate all users.

-- Skynet, 09-29-1997, 02:14 hours :)