New Approach To Malware Modifies Linux Kernel

Hugh Pickens writes "Professor Avishai Wool has unveiled a program to watch for malware on servers with a modification to the Linux kernel. 'We modified the kernel in the system's operating system so that it monitors and tracks the behavior of the programs installed on it,' says Wool. Essentially, Wool says, his software team has built a model that predicts how software running on a server should work (pdf). If the kernel senses abnormal activity, it stops the program from working before malicious actions occur. 'When we see a deviation, we know for sure there's something bad going on,' Wool explains. Wool cites problems with costly anti-virus protection. 'Our methods are much more efficient and don't chew up the computer's resources.'"

Comments

Help!

[i_liek_turtles]

It's stopped me from running Vista in a VM...

Re:Help!

[Anonymous Coward]

How is this off-topic? The mods must have been infected!

Re:Help!

[Anonymous Coward]

That's nothing. I ran it in a virtual machine on Vista. It broke out, took over the machine, and put vista in a vm.

premise to shutdown

[thatchman1]

Is this not the very premise that caused the Amazon cloud shutdown? A failure to communicate back proper activity illogically deduced that there was an improper activity?

Re:premise to shutdown

[Qzukk]

This has greatly increased the online sales of weight-loss products although mostly from browsers identifying themselves as Internet Explorer

Linux users were terminated by their modified kernel after it detected that they were exercising ;)

selinux

[perlchild]

Great, sounds exactly like what people have been doing with selinux and capabilities. But selinux acknowledges we don't always do the same things with our computers as the next guy... Will this approach be as flexible?

I don't want to boohoo his research, it's probably fine, but the article summary just gets my goat. Malware is a lot more complicated than most anti-malware software authors make them sound, and false positives are the biggest/most complicated problem they have to deal with, especially in automated systems that block like this...

Re:selinux

[icsx]

If you can detect and isolate a program that doesnt do what it is supposed to be doing and attracts attention with unusual behaviour, like scanning certain ports or trying to send packets etc, then it may be something that you dont have to have on your server. Most security programs today arent looking up what software does. They look what ports are used, what traffic is routed and so on.

These malware programs today try to hide themselves into deep that you just couldnt find them if you dont know what you are looking for. This system here as i understand it tries to identify what are the normal parameters for a certain program to work. If the program doesnt do or behave like a normal software, then there must be something wrong with it and alarms go off, lights are blinking and all hell brokes lose.

Oh crap, red lights and i hear noises. Oh it's only the cops.

Re:selinux

[Kjella]

Well, from my basic reading of the paper it sounds like it won't have false positives but it also will miss many negatives. Essentially when you build it it'll make a map of what system calls can be made and in what sequence. If an application makes a system call it never calls or never can call in that order because it's been hijacked then this thing will stop it. If you manage to do your nasty business using the system calls it normally uses, it won't. Think of it as a auto-hardening system turning off any syscalls or combinations that the application doesn't use anyway. One of the downsides is that if you know this system is in place, you can probably add dummy syscall patterns to your exploit to match the application's behavior unless it's a syscall it never does. Still, there's little reason assuming an attacker is perfect and this is a worthwhile protection for the cases where it does work.

Re:selinux

[Plouf]

If an application makes a system call it never calls or never can call in that order because it's been hijacked then this thing will stop it.

The problem is that, for any non-trivial program, it is impossible for a static analyser to decide whether the software will call or not and in what order a specific trap. For instance, that static analyser is not able to tell you in advance at what time the exit syscall will be called.

Re:selinux

[debatem1]

This is actually quite different from selinux, although they both hinge on controlling what syscalls a program can make. The major difference here is that this system automatically builds an equivalent to the selinux rules at compile time, and therefore avoids a lot of the problems caused by poorly written selinux rules. An associated advantage is the ability to track whether a chain of syscalls is legitimate. Such context analysis could be very powerful if used properly.

However, you've probably already spotted the major flaws with this approach: the first is that it only works on compiled programs, which strikes me as a serious problem when you're talking about webservers. The second is that it doesn't work on certain classes of programs whose execution pattern is extremely difficult to predict: self-modifying code, highly dynamic code (longjmp is not allowed), etc. Another limitation is that it only works on statically linked libraries. Finally, it is totally dependent on GCC and friends, which could be a problem for it moving forward, and in groups where the intel compiler is preferred.

As for false positives, the entire point of this is that inside of these admittedly limited confines, it has no possibility for false positives. This system is not statistical in nature. It depends entirely upon the program itself to determine correct syscall behavior.

All in all, while it is a long way from a practical security model, it does offer the promise of powerful, accurate protection from certain classes of attacks. When combined with selinux and pcap on a system with a slim attack profile it could help to narrow the gap between being a zero-day compromise and having full protection.

Re:selinux

[Kjella]

Or the application has been legitimately updated to do new things...

Well that part is handled. The call map is made when the application is compiled, so new source equals new map. That's at least one advantage over SELinux, this is completely automatic and it's always correct in that sense. Though if you want to get this with precompiled binaries, you also have to get precompiled maps so the distros have to help you out - it's not something you can set up on existing binaries on your own.

Re:selinux

[QuoteMstr]

It can't catch every case, but it can rule out a lot of system calls. If setuid never appears in a program or library's source, it can't be called. If a program calls a bunch of things in an initializer function and then enters a more restricted main loop, the static analyzer should be able to catch that too.

I can't see how this approach could hurt, though of course it won't catch everything.

Re:selinux

[debatem1]

If you're looking into securing a shared hosting solution, you might look into libpcap. I mentioned it before, but it gets far less attention than it deserves, especially among admins. It has the capability to forbid forming new chroots, making modifications to your networking properties, the insertion of kernel modules, even escalations to root. Between libpcap, something like this, a well-constructed jail, a good vserver solution, and a very carefully secured network, you're going to be as secure as just about anything out there.

Oh great.

[bigtallmofo]

They recently unveiled a unique new program called the "Korset" to stop malware on Linux...and once it reaches its full potential it could put anti-virus software companies out of business.

Doesn't our economy have enough problems? Do we really need to put Linux anti-virus vendors out of business? Next we'll probably drive the ice vendors in Alaska to bankruptcy!

Re:Oh great.

[fuzzyfuzzyfungus]

Oh, don't worry. Things are just warming up for the ice vendors...

Re:Heuristic scanning v2.0?

[ThinkingInBinary]

this sounds like the heuristic scanning features that has been in Norton Antivirus and other A/V utilities for almost a decade now, where it searches for out of the norm items and reports or blocks them, such as a program deciding to write to the MBR, or a program using raw disk I/O to write to the hard disk.

Wow, those "heuristics" sound like a simple blacklist of "virus-like" activities.

No, what this does is cleverer. It creates (at compile time) an automaton representing the system call activity of the program, and if the program tries to make a syscall that does not have a matching edge in the automaton, it kills it. Basically, if there is not a code path that should lead to execution of a certain syscall, the program gets killed.

How it works

[Animats]

From the papter: "The resulting model is an automaton that represents the legitimate order of system calls that an application may issue. This automaton is then enforced by Korset's monitoring agent, which is built into the Linux kernel, by simulating every emitted system call."

This is not likely to work for scriptable applications (Apache, Java-based servers, etc.) The order of calls is determined by the script, not the underlying executable.

Ummmm.....

[Hasai]

....I thought that was the philosophy behind AppArmor (http://en.opensuse.org/Apparmor).

It's been deployed in SuSE products for years.

Re:Ummmm.....

[causality]

....I thought that was the philosophy behind AppArmor (http://en.opensuse.org/Apparmor).

It's been deployed in SuSE products for years.

Apparmor seems to be a relatively sophisticated least-privilege system, i.e. the idea that if a BIND DNS server should never need to (for example) modify the routing table, then it also should not be able to modify the routing table. That way, if an attacker compromises said DNS server, he won't be able to do very much with it that isn't directly related to serving DNS requests (this is why I would personally refer to such a system as damage control, useful for containing/limiting an attacker who has already compromised something). The system discussed in the article is different in that it seems to be less concerned with what specific tasks a program should or should not be doing and more concerned with whether the code that is executed and the way that it is executed is what you would expect from the program's source. That way, if someone exploits i.e. a buffer overflow and inserts their own shellcode, it would deviate from the pattern that you would have expected from the exploited program and this deviation would be detected.

Both can be compared to systems like PaX (kernel) and SSP (userspace) which are intended to make sure that an attacker will fail to exploit an existing vulnerability, such as an unpatched buffer overflow, in the first place.

I'm definitely not an expert...

[fuzzyfuzzyfungus]

But this looks a lot like SElinux or AppArmor, except that the application profiles are constructed by static analysis of program code, rather than by hand, or by observing the app during a "training" period. The linked paper indicates that it is still in a rather rough state; but it looks quite promising.

Linux??

[fxkr]

I really don't think Linux has problems with malware. I think there is an other operating system having more trouble.

As far as I know virus scanners are used on servers mostly to check data that goes through it (example: email server); this data will however not be executed on the server.

Not a good article, but an interesting paper.

[argent]

OK, what this is doing is watching for code injection attacks (buffer overflows, stack smashing, etcetera) by building a model of how each specific application is going to operate, and blocking system calls that the model of the application would never make. It seems like an interesting approach, though it may not be as useful on Windows where there's not such a formal distinction between system calls and other kinds of calls.

It won't do anything about interpreter code injection (eg, SQL injection or shell code injection) or script privilege escalation attacks (eg, ActiveX and other "cross zone" attacks in Internet Explorer), or attacks that involve complete executable code drops.

Still, this is useful and not nearly as dodgy as the article made it sound.

Re:Not a good article, but an interesting paper.

[argent]

if my exchange server is compromised and starts to serve webpages by binding to a port other than port 25, this method would catch it and kill the process in its tracks.

On the other hand, if your exchange server is compromised and starts shipping spam it won't do a damn thing to stop it. And if your exchange server is compromised and loads a DLL using its normal API for loading modules, then it can do anything that module will attack.

You application, which normally never spawns a procses while in a certain function, will be killed as soon as it attempts to execute because it violates the model that was created at compile time.

It only tracks system calls, not functions (that would have insane levels of overhead), and it cant even see what the interpreter thinks of as functions. Any time an interpreter is involved, the call graph will be "bushy" because all possible operations that the interpreter supports will be legal as the "next step" from each step in the interpreter loop.

it is designed to stop "cross zone" attacks, but it depends on the application developer's approach. i.e. ie7 would not benefit from this trick, but ie8 would (each zone is spawned in a new thread)

I think you're confusing cross zone attacks (attacks on the security zone model in Windows) with cross-site attacks.

this is an os-agnostic approach to stopping malware, they just used the linux kernel because its free.

Also, Linux has a formal "system call" interface, so they have a single point of entry with well behaved semantics and known overhead, so they can control the impact of their technique very closely. They can't put this in all Windows call gates, they'll have to pick particular DLLs that they track, and maybe even limit the calls they track (you don't want anything that's doing frequent callbacks to be tracked, for example).

Will fork bomb do work still?

[DpakoH]

I'll try to run famous :(){ :|:& };: shell example

I might give it a spin

[FudRucker]

When & if Linus Torvalds (or whoever the benevolent dictator of the kernel is nowadays) includes it in to the main kernel source tree...

Sounds like a good idea to me, I just want to see what the Linux kernel pros think of it...

Re:I might give it a spin

[ZachPruckowski]

It seems too specialized to go in the main kernel tree. Additionally, I can only imagine that it hurts performance more than some users would like. Additionally, it requires a fundamental change to the system of distributing linux apps. Maybe a distro will include it in their kernel and modify their repository to include pre-built maps, but I can't see it becoming fully mainstream.

It's kind of like Vista

[TheRealSlimShady]

Sounds from the summary at least (hey, it's slashdot, I haven't read the article) that it's similar in some ways to the service profiling in Vista. The service profiling means that the dev looked at what the service needed to do to be able to run and gave it only those permissions, restricting the damage it could do if it were compromised. This seems to extend that to give the kernel the intelligence to baseline the services itself, and then restrict activity when the baseline activity changes.

Re:It's kind of like Vista

[Delkster]

These guys make a point of avoiding the labour involved in manually building the profile. (FWIW, I don't know about anything service profiling in Vista -- in fact, I had never heard about it -- but your description also sounds somewhat reminiscent of AppArmor.)

Re:Just trying to understand

[causality]

It creates (at compile time) an automaton representing the system call activity of the program

At compile time of the program? So in addition to a modified kernel you need a modified gcc and to compile everything from source or have a specialised distro? It doesn't surprise me that the summary should be lacking such details, but it would be nice if for once it gave a decent overview.

I agree that this was a poor summary but instead of complaining about the summary you could always do something crazy like read the article.

Re:Completely incorrect basic assumptions

[QuoteMstr]

You're right. You can't exactly predict the behavior of a program without running it.

But that's not what this package is trying to do. Instead, it's trying to rule out large swaths of the behavior space of a program based on static analysis. Of course there will be false negatives -- i.e., malicious actions that remain undetected. But I don't really see how false positives would be a danger, modulo bugs in the static analyzer.

I imagine this package would be nearly useless for something like firefox, which does many varied tasks. But for programs like exim, or bind, or vsftp -- which do one task over and over again -- the degree of protection should be pretty good because there's a lot these programs don't do.

Re:Completely incorrect basic assumptions

[Anonymous Coward]

The halting problem is a nice thing, but don't throw it at everything that looks like static analysis.

You can take any program and build a directed graph. The nodes are instructions. You add an arrow a=(x,y) to it, if after executing x the instruction pointer can be at y. No halting problem here. To be more precise the graph is build in O(n) with n being the number of instructions in the program.

Now imagine you leave out most instructions, and consider system calls only. Still no halting problem.

Now if you are given a list of system calls you can use the graph to decide if it is possible for the program to execute in that order. You get zero false positives.

Tron

[nsayer]

Somehow, this technique reminds me of the (obviously rather simplistic) description of the functionality of the Tron program from the movie of the same name. From the script:

                    DILLINGER
  [...]
                                        What's the thing you're working on?

                                                                ALAN
                                        It's called Tron. It's a security
                                        program itself, actually. Monitors
                                        all the contacts between our system
                                        and other systems... If it finds
                                        anything going on that's not scheduled,
                                        it shuts it down. I sent you a memo
                                        on it.

                                                              DILLINGER
                                        Mmm. Part of the Master Control Program?

                                                              ALAN
                                        No, it'll run independently.
                                        It can watchdog the MCP as well.

As I understand it

[slimjim8094]

this isn't anything specifically to do with malware.

As far as I can see, this verifies that the binary currently running is the same binary that was compiled from a (trusted) source.

When you compile it, it knows (from the source) what the program will and won't do. If the program deviates from that, it dies (as it's been replaced by malware, presumably)

If I'm wrong, please correct me...

Re:Heuristic scanning v2.0?

[liquiddark]

So basically it shuts down malware and buggy software. Holy fuck, somebody kill this thing - we're all out of a job if this catches on.

Re:Looks like something AV has done for a while

[Delkster]

Their method is to automagically profile the software when it's compiled. That's obviously something antivirus software doesn't do.

Antivirus software generally doesn't know how a particular piece of legitimate software is supposed to behave. The idea here is that the approach these guys are taking is exactly to try to build a profile for the normal behaviour of the application (or some parts of it, namely the pattern of syscalls). The analysis would be done at compile time, and when the application is being run its pattern of making syscalls would be matched against the known profile. That way the system could supposedly notice a compromised application or service because its pattern would be different than expected.

Also, the method would be powerless against trojans, for example, because they don't work by exploiting vulnerabilities in legitimate software and changing the way it's executed. To me the suggested system seems more like just a potentially boosted intrusion detection system, not general malware-pattern detection, which is what antivirus software does.

How well their profiling would actually work, that I don't know. Theoretically you can't make a complete analysis of all paths the execution of a program could possibly take (for an arbitrary program anyway). Multithreading and inter-process communication might make that even more difficult. But then, maybe it would be just possible to do it for some cases that practically occur with some decent probability.

Re:Just trying to understand

[Rich0]

I wonder if a fix for the buffer overflow (besides languages that make it harder) would be to separate the stacks used for local variables and return addresses.

The problem is that when a call is made to a function the compiler pushes the return address onto the stack. Then the function allocates space for its own variables on top of the same stack. If one of those variables overflows it can hit the return address. That essentially is a mixing of code and data. If you had two stacks then the processor could trigger an exception if anything writes to one of them except via a call or return. You could probably accomplish this via changes to the compiler without a processor change - the processor will always use the regular stack but a compiler could be designed to maintain a separate stack for local variables. You wouldn't have that read-only protection on the regular stack, but the two would be in different segments making an overflow impossible.

Other tricks that are used are things like canarys - values written onto the stack and then checked before a return - if there was an overflow the canary would not be intact. GCC has an option to do this which works most of the time.

What about plug-in based systems?

[JSBiff]

How does this thing deal with plug-in/add-on based systems like Firefox or Eclipse, where new capabilities get added to the executable through dlls (or java classes, I guess, in the case of Eclipse? - Although, with regards to Java, I wonder if this system would work at all, since I think the kernel never exactly 'sees' Java programs or classes as executables, but only the JRE, which already has all the system calls built into it?)

Re:What about plug-in based systems?

[Jerry Smith]

How does this thing deal with plug-in/add-on based systems like Firefox or Eclipse, where new capabilities get added to the executable through dlls (or java classes, I guess, in the case of Eclipse? - Although, with regards to Java, I wonder if this system would work at all, since I think the kernel never exactly 'sees' Java programs or classes as executables, but only the JRE, which already has all the system calls built into it?)

It's about servers here, I personally think one should really think thrice before installing plug-ins and add-ons on a server, and rather go browsing on a desktop machine. Regarding Java, I can see your point.