New Approach To Malware Modifies Linux Kernel

Hugh Pickens writes "Professor Avishai Wool has unveiled a program to watch for malware on servers with a modification to the Linux kernel. 'We modified the kernel in the system's operating system so that it monitors and tracks the behavior of the programs installed on it,' says Wool. Essentially, Wool says, his software team has built a model that predicts how software running on a server should work (pdf). If the kernel senses abnormal activity, it stops the program from working before malicious actions occur. 'When we see a deviation, we know for sure there's something bad going on,' Wool explains. Wool cites problems with costly anti-virus protection. 'Our methods are much more efficient and don't chew up the computer's resources.'"

Comments

Ummmm.....

[Hasai]

....I thought that was the philosophy behind AppArmor (http://en.opensuse.org/Apparmor).

It's been deployed in SuSE products for years.

Re:Ummmm.....

[causality]

....I thought that was the philosophy behind AppArmor (http://en.opensuse.org/Apparmor).

It's been deployed in SuSE products for years.

Apparmor seems to be a relatively sophisticated least-privilege system, i.e. the idea that if a BIND DNS server should never need to (for example) modify the routing table, then it also should not be able to modify the routing table. That way, if an attacker compromises said DNS server, he won't be able to do very much with it that isn't directly related to serving DNS requests (this is why I would personally refer to such a system as damage control, useful for containing/limiting an attacker who has already compromised something). The system discussed in the article is different in that it seems to be less concerned with what specific tasks a program should or should not be doing and more concerned with whether the code that is executed and the way that it is executed is what you would expect from the program's source. That way, if someone exploits i.e. a buffer overflow and inserts their own shellcode, it would deviate from the pattern that you would have expected from the exploited program and this deviation would be detected.

Both can be compared to systems like PaX (kernel) and SSP (userspace) which are intended to make sure that an attacker will fail to exploit an existing vulnerability, such as an unpatched buffer overflow, in the first place.

Linux??

[fxkr]

I really don't think Linux has problems with malware. I think there is an other operating system having more trouble.

As far as I know virus scanners are used on servers mostly to check data that goes through it (example: email server); this data will however not be executed on the server.

Re:selinux

[Kjella]

Well, from my basic reading of the paper it sounds like it won't have false positives but it also will miss many negatives. Essentially when you build it it'll make a map of what system calls can be made and in what sequence. If an application makes a system call it never calls or never can call in that order because it's been hijacked then this thing will stop it. If you manage to do your nasty business using the system calls it normally uses, it won't. Think of it as a auto-hardening system turning off any syscalls or combinations that the application doesn't use anyway. One of the downsides is that if you know this system is in place, you can probably add dummy syscall patterns to your exploit to match the application's behavior unless it's a syscall it never does. Still, there's little reason assuming an attacker is perfect and this is a worthwhile protection for the cases where it does work.

Re:selinux

[debatem1]

This is actually quite different from selinux, although they both hinge on controlling what syscalls a program can make. The major difference here is that this system automatically builds an equivalent to the selinux rules at compile time, and therefore avoids a lot of the problems caused by poorly written selinux rules. An associated advantage is the ability to track whether a chain of syscalls is legitimate. Such context analysis could be very powerful if used properly.

However, you've probably already spotted the major flaws with this approach: the first is that it only works on compiled programs, which strikes me as a serious problem when you're talking about webservers. The second is that it doesn't work on certain classes of programs whose execution pattern is extremely difficult to predict: self-modifying code, highly dynamic code (longjmp is not allowed), etc. Another limitation is that it only works on statically linked libraries. Finally, it is totally dependent on GCC and friends, which could be a problem for it moving forward, and in groups where the intel compiler is preferred.

As for false positives, the entire point of this is that inside of these admittedly limited confines, it has no possibility for false positives. This system is not statistical in nature. It depends entirely upon the program itself to determine correct syscall behavior.

All in all, while it is a long way from a practical security model, it does offer the promise of powerful, accurate protection from certain classes of attacks. When combined with selinux and pcap on a system with a slim attack profile it could help to narrow the gap between being a zero-day compromise and having full protection.

Re:selinux

[Kjella]

Or the application has been legitimately updated to do new things...

Well that part is handled. The call map is made when the application is compiled, so new source equals new map. That's at least one advantage over SELinux, this is completely automatic and it's always correct in that sense. Though if you want to get this with precompiled binaries, you also have to get precompiled maps so the distros have to help you out - it's not something you can set up on existing binaries on your own.

Re:Just trying to understand

[causality]

It creates (at compile time) an automaton representing the system call activity of the program

At compile time of the program? So in addition to a modified kernel you need a modified gcc and to compile everything from source or have a specialised distro? It doesn't surprise me that the summary should be lacking such details, but it would be nice if for once it gave a decent overview.

I agree that this was a poor summary but instead of complaining about the summary you could always do something crazy like read the article.

Re:Completely incorrect basic assumptions

[Anonymous Coward]

The halting problem is a nice thing, but don't throw it at everything that looks like static analysis.

You can take any program and build a directed graph. The nodes are instructions. You add an arrow a=(x,y) to it, if after executing x the instruction pointer can be at y. No halting problem here. To be more precise the graph is build in O(n) with n being the number of instructions in the program.

Now imagine you leave out most instructions, and consider system calls only. Still no halting problem.

Now if you are given a list of system calls you can use the graph to decide if it is possible for the program to execute in that order. You get zero false positives.

Re:Looks like something AV has done for a while

[Delkster]

Their method is to automagically profile the software when it's compiled. That's obviously something antivirus software doesn't do.

Antivirus software generally doesn't know how a particular piece of legitimate software is supposed to behave. The idea here is that the approach these guys are taking is exactly to try to build a profile for the normal behaviour of the application (or some parts of it, namely the pattern of syscalls). The analysis would be done at compile time, and when the application is being run its pattern of making syscalls would be matched against the known profile. That way the system could supposedly notice a compromised application or service because its pattern would be different than expected.

Also, the method would be powerless against trojans, for example, because they don't work by exploiting vulnerabilities in legitimate software and changing the way it's executed. To me the suggested system seems more like just a potentially boosted intrusion detection system, not general malware-pattern detection, which is what antivirus software does.

How well their profiling would actually work, that I don't know. Theoretically you can't make a complete analysis of all paths the execution of a program could possibly take (for an arbitrary program anyway). Multithreading and inter-process communication might make that even more difficult. But then, maybe it would be just possible to do it for some cases that practically occur with some decent probability.

Re:selinux

[debatem1]

If you're looking into securing a shared hosting solution, you might look into libpcap. I mentioned it before, but it gets far less attention than it deserves, especially among admins. It has the capability to forbid forming new chroots, making modifications to your networking properties, the insertion of kernel modules, even escalations to root. Between libpcap, something like this, a well-constructed jail, a good vserver solution, and a very carefully secured network, you're going to be as secure as just about anything out there.

Re:Not a good article, but an interesting paper.

[argent]

if my exchange server is compromised and starts to serve webpages by binding to a port other than port 25, this method would catch it and kill the process in its tracks.

On the other hand, if your exchange server is compromised and starts shipping spam it won't do a damn thing to stop it. And if your exchange server is compromised and loads a DLL using its normal API for loading modules, then it can do anything that module will attack.

You application, which normally never spawns a procses while in a certain function, will be killed as soon as it attempts to execute because it violates the model that was created at compile time.

It only tracks system calls, not functions (that would have insane levels of overhead), and it cant even see what the interpreter thinks of as functions. Any time an interpreter is involved, the call graph will be "bushy" because all possible operations that the interpreter supports will be legal as the "next step" from each step in the interpreter loop.

it is designed to stop "cross zone" attacks, but it depends on the application developer's approach. i.e. ie7 would not benefit from this trick, but ie8 would (each zone is spawned in a new thread)

I think you're confusing cross zone attacks (attacks on the security zone model in Windows) with cross-site attacks.

this is an os-agnostic approach to stopping malware, they just used the linux kernel because its free.

Also, Linux has a formal "system call" interface, so they have a single point of entry with well behaved semantics and known overhead, so they can control the impact of their technique very closely. They can't put this in all Windows call gates, they'll have to pick particular DLLs that they track, and maybe even limit the calls they track (you don't want anything that's doing frequent callbacks to be tracked, for example).

Re:Just trying to understand

[Rich0]

I wonder if a fix for the buffer overflow (besides languages that make it harder) would be to separate the stacks used for local variables and return addresses.

The problem is that when a call is made to a function the compiler pushes the return address onto the stack. Then the function allocates space for its own variables on top of the same stack. If one of those variables overflows it can hit the return address. That essentially is a mixing of code and data. If you had two stacks then the processor could trigger an exception if anything writes to one of them except via a call or return. You could probably accomplish this via changes to the compiler without a processor change - the processor will always use the regular stack but a compiler could be designed to maintain a separate stack for local variables. You wouldn't have that read-only protection on the regular stack, but the two would be in different segments making an overflow impossible.

Other tricks that are used are things like canarys - values written onto the stack and then checked before a return - if there was an overflow the canary would not be intact. GCC has an option to do this which works most of the time.

Re:What about plug-in based systems?

[Jerry Smith]

How does this thing deal with plug-in/add-on based systems like Firefox or Eclipse, where new capabilities get added to the executable through dlls (or java classes, I guess, in the case of Eclipse? - Although, with regards to Java, I wonder if this system would work at all, since I think the kernel never exactly 'sees' Java programs or classes as executables, but only the JRE, which already has all the system calls built into it?)

It's about servers here, I personally think one should really think thrice before installing plug-ins and add-ons on a server, and rather go browsing on a desktop machine. Regarding Java, I can see your point.