Sandia Studies Botnets In 1M OS Digital Petri Dish

Ponca City, We love you writes "The NY Times has the story of researchers at Sandia National Laboratories creating what is in effect a vast digital petri dish able to hold one million operating systems at once in an effort to study the behavior of botnets. Sandia scientist Ron Minnich, the inventor of LinuxBIOS, and his colleague Don Rudish have converted a Dell supercomputer to simulate a mini-Internet of one million computers. The researchers say they hope to be able to infect their digital petri dish with a botnet and then gather data on how the system behaves. 'When a forest is on fire you can fly over it, but with a cyber-attack you have no clear idea of what it looks like,' says Minnich. 'It's an extremely difficult task to get a global picture.' The Dell Thunderbird supercomputer, named MegaTux, has 4,480 Intel microprocessors running Linux virtual machines with Wine, making it possible to run 1 million copies of a Windows environment without paying licensing fees to Microsoft. MegaTux is an example of a new kind of computational science, in which computers are used to simulate scientific instruments that were once used in physical world laboratories. In the past, the researchers said, no one has tried to program a computer to simulate more than tens of thousands of operating systems."

Comments

Life imitates XKCD

[Tackhead]

Once again, life imitates XKCD: Network.

Re:Life imitates XKCD

[The_mad_linguist]

Well, given that XKCD was imitating an old hacker competition...

Re:Life imitates XKCD

[dintlu]

Goes to show that ideas are a dime a dozen.

Implementing something like this is what makes the news.

Re:Life imitates XKCD

[Ambvai]

Is there any serious implementation of that XKCD comic, or even just in an imitation of what looks like computers fighting for control of a network?

I've got an easier way

[iamapizza]

what is in effect a vast digital petri dish able to hold one million operating systems at once in an effort to study the behavior of botnets

If they've set up this mini-internet and have set up this botnet, then the easiest way to understand its behavior would be to look at the source code

Re:I've got an easier way

[Ant P.]

OK, here's seven hundred million lines of source code. Come back when you've solved the halting problem.

Re:I've got an easier way

[leuk_he]

The source code does not help you to imange what happens in peer to peer network with very large amounts of cleints that have a different kind of environment. Not to mention software that has bugs.

BTW... who is the first to post to the xkcd comic about it normal people have aquaria

Re:I've got an easier way

[dotgain]

BTW... who is the first to post to the xkcd comic about it

Uhh, the First Post?

Re:I've got an easier way

[Sta7ic]

Just like the easiest way to understand how a dog works is to dissect them.

In short, no. You can figure out how some of the parts work, but there's a lot within complex software that is non-deterministic, whether for internal, external, or thoroughly inadvertant reasons on either side. Just because you _think_ you know what it's doing doesn't mean it'll act the way you expect it to.

Also, see http://xkcd.com/397/

Re:I've got an easier way

[caramelcarrot]

Simple rules can give rise to complex behaviour. Who knows what the botnet might do? It could have harmonic resonances, it could have phase changes at critical infection rates, it could do all sorts of interesting and complex behaviour. Looking at the source code won't tell you any of this.

Re:I've got an easier way

[coreboot]

not really. Source code analysis goes just so far. Multiplied by 1M, it goes less far still. And then there's this little issue: http://en.wikipedia.org/wiki/Halting_problem
ron

Re:I've got an easier way

[voidphoenix]

You can't study emergent behavior by studying source code. Even within one host, the interactions between malware, applications and every the piece of the OS would already have emergent properties. Magnify by tens of thousands to millions (exponentially, not additively or multiplicatively), and the sheer complexity of the entire system would overwhelm our ability to understand it.

We have ~100 billion neurons and ~100 trillion synapses. At 2^N - N - 1 subgroups, how many pieces before the system's complexity outruns our brain's processing power? A network of 47 pieces has ~140 trillion subgroups. With several million pieces...

Re:I've got an easier way

[Vellmont]

Maybe. But why use ACTUAL botnets for this purpose and not study the underlying algorithms and infection behavior directly? That would give you the ability to generalize instead of relying on -botnet X, version z-

If that's what you care about, study it. Why rely on botnet authors to code some arbitrary botnet spreading code when you can write your own and study various different scenarios at will?

Re:I've got an easier way

[swillden]

If it's unclear what the code does, run it in a debugger and control the inputs. Step through the code line by line. If the debugger doesn't do everything you want, write a better debugger.

Is that right?

Here, I'll describe a program so simple it can be coded in under 100 lines, and can be fully specified in a few sentences, then ask you a question about its behavior. It should be easy, right?

There is a 100x100 grid of cells. Each cell is in one of two states "live" or "dead". Each cell has 8 neighbors, the cells horizontally, vertically and diagonally adjacent (the edges of the grid "wrap", so this is true even for edge cells). Each "generation", the state of the cells is updated according to the following rules:

1. Any live cell with fewer than two live neighbours becomes dead.
2. Any live cell with more than three live neighbours becomes dead.
3. Any dead cell with exactly three live neighbours becomes live.
4. All other cells remain unchanged.

That's it. Now, given an initial state of the grid, tell me what the state is after 100, 500 and 1000 generations. Further, tell me whether or not any patterns of live cells will survive across across generations. Will patterns repeat? Can patterns move? Interact?

Amazing complexity can arise from very simple rules. In this case (known as Conway's Game of Life, if you hadn't recognized it), the above rules contain enough power that if you make the grid infinite in size, the result is a Turing-complete computation system. In addition, the shifting patterns it creates are bewildering in their number, complexity and behavior.

Now scale that up to thousands of lines of code. Granted, not code specifically chosen to create interesting interactions, but still 2-3 orders of magnitude more complex. Further, code that itself lives in and interacts with a complex and varied ecosystem of other code, some of which is trying to detect the code and kill it -- so the code is written to be self-modifying, to "mutate" a bit, after a fashion. Also add in the ability to migrate between "ecosystems", reproduce, receive deliberate external updates and instructions, etc.

Simulation is the only way to get a handle on this sort of thing. And that's why the very smart people who designed and built the world's first million-machine simulator decided to do it.

Re:I've got an easier way

[AliasMarlowe]

OK, here's seven hundred million lines of source code. Come back when you've solved the halting problem.

Power switch. Halts that sucker every time.

First Findings!

[CorporateSuit]

The first thing the researchers noticed is that within 30 minutes, the botnet had sent over 6 billion emails out of newly-registered gmail and hotmail accounts, and continued to send millions more each hour. The researchers say the botnet thrives on pain and misery, and probably shouldn't have been given access to the real internet.

They can't afford an MSDN subscription?

[n0tWorthy]

Then they can run 1 million copies without a subscription.

Re:They can't afford an MSDN subscription?

[Anonymous Coward]

Someone marked this as 'funny' but it is true. Read the license it is per user... If your creating a cluster with THOUSANDS of nodes and testing things you are perfectly within your rights to do this. You can even get most of the different versions of the OS going. 98, 98se, 95 (shudder), ME (double shudder), NT4, 2k, XP, Vista, 7, etc... Putting different versions at different patch levels etc...

http://msdn.microsoft.com/en-us/subscriptions/cc150618.aspx

They lost me at Wine. As that would not truly create the environment they are trying to describe.

I have had up to 100 desktops all going from 10 msdn licenses (10 users). With different levels of the OS to test install and different configurations. They probably dont even need a very high level of it.

Is that really a windows environment?

[damn_registrars]

I understand not wanting to buy 1M windows licenses; I am of the persuasion that is not inclined to buy 1 license.

However, the summary seems to claim that Wine == Windows environment. I don't see how they are analogous in this sense. In particular, if you are trying to understand botnet behavior, you need infected botnet systems. Is there a way to make Wine vulnerable to the infections that frequently hit Windows systems?

Re:Is that really a windows environment?

[geegel]

Welcome to the world of open source software. The place where you can modify the code in any way you want.

Re:Is that really a windows environment?

[mcrbids]

I don't see how they are analogous in this sense. In particular, if you are trying to understand botnet behavior, you need infected botnet systems. Is there a way to make Wine vulnerable to the infections that frequently hit Windows systems?

WINE is an implementation of the Win32 API. Since the *target* of WINE is to emulate Windows, then in order to be successful, it must implement the bugs as well. So the better WINE is, the better it runs *ALL* Windows software - including the viruses and malware!

I would assume (ass + u + me) that they've done enough unit testing on the particular botnet software in question to determine its compatibility with WINE, and so long as this compatibility is sufficient, then this could be a very useful test environment. It's the botnet being studied, not Windows itself!

Another example: Windows 2000. I build data management software. I test with Windows 2000. Not because Win2000 is an example of the latest greatest from MS, but because it costs me nothing extra and runs nicely in a VM. Since the only O/S features I care about are those that are already present in Win2000, it creates a very useful test environment despite lacking many pieces present in later OS versions.

Re:Is that really a windows environment?

[EdIII]

I understand not wanting to buy 1M windows licenses; I am of the persuasion that is not inclined to buy 1 license.

However, the summary seems to claim that Wine == Windows environment. I don't see how they are analogous in this sense. In particular, if you are trying to understand botnet behavior, you need infected botnet systems. Is there a way to make Wine vulnerable to the infections that frequently hit Windows systems?

Yeah, I call bullshit that on too. If you want to study botnet behavior, which includes studying malware and viruses, then it should be a "real" Microsoft OS. I don't think WINE counts.

I am not the biggest fan of ol' M$, but considering how interesting this research is and it's possible positive impact on the greater community (which does benefit Microsoft) you would think they would at least ask Microsoft for some licenses gratis.

Microsoft would probably be reasonable, if just for the good PR, which they sorely and always need.

Re:Is that really a windows environment?

[Anonymous Coward]

I can't possibly imagine how a simulation of millions of instances of your software infecting itself would be good PR.

Re:Is that really a windows environment?

[Anonymous Coward]

I understand not wanting to buy 1M windows licenses; I am of the persuasion that is not inclined to buy 1 license.

However, the summary seems to claim that Wine == Windows environment. I don't see how they are analogous in this sense. In particular, if you are trying to understand botnet behavior, you need infected botnet systems. Is there a way to make Wine vulnerable to the infections that frequently hit Windows systems?

Yeah, I call bullshit that on too. If you want to study botnet behavior, which includes studying malware and viruses, then it should be a "real" Microsoft OS. I don't think WINE counts.

I am not the biggest fan of ol' M$, but considering how interesting this research is and it's possible positive impact on the greater community (which does benefit Microsoft) you would think they would at least ask Microsoft for some licenses gratis.

Microsoft would probably be reasonable, if just for the good PR, which they sorely and always need.

True... But if they did use *real* windows instead of Wine, then the supercomputer could only virtualise a few hundred copies of Windows XP running simultaneously, or 2-3 copies if it's Vista. :E

Re:Is that really a windows environment?

[MaskedSlacker]

I think you're misunderstanding what they are doing. They are not studying in-the-wild worms. They are trying to build theoretical models of botnets and how they propagate through networks--this is the equivalent of computer simulations of viral epidemics. You don't need to simulate what the virus does in a person to study how it spreads through a population.

Re:Is that really a windows environment?

[amicusNYCL]

Since this is a closed environment for a scientific study, it would make sense for them to use viruses which spread via exploits that they know are present.

Re:Is that really a windows environment?

[amicusNYCL]

The research isn't to determine how Windows reacts to a botnet. They're trying to figure out how the botnet itself communicates and spreads. Or, more specifically, what the botnet looks like as it is spreading. Windows is just the platform that they're running the botnet on (sort of), but they don't really care how Windows reacts to it.

In other words, they're studying the botnet itself, not the infrastructure it runs on.

Re:Is that really a windows environment?

[vux984]

I think you're misunderstanding what they are doing.

I think you are correct. However, that raises the question: why use WINE?

Since they aren't relying on 'real in the wild exploits' they could model botnets and how they proagate through networks on linux or freebsd just as easily. Its really just specialized p2p and client server software to simulate botnet behaviour and spread.

Re:Is that really a windows environment?

[Antique Geekmeister]

WINE is far less resource intensive, and typically runs far faster, than fully virtualized simulation software, especially because it leaves out the basically rewritten-VMS kernel and memory management of the Windows kernel in favor of Linux's own pretty zippy kernel. And the cost of buying and running a million actual Windows boxes to avoid the performance penalties of virtualization is simply infeasible.

Re:Is that really a windows environment?

[hairyfeet]

As an old greybeard PC repairman I can tell you that Windows bugs are screwing around with the guts of Windows more than any tweaked Wine could ever replicate. I don't see why they wouldn't just pony up for MSDN where they could then run all the real Windows versions they wanted and then get more realistic results. This seems like they are going pretty far out of their way to keep from spending a buck, when the cost of that monster PC makes being so "penny wise, pound foolish" seem extra crazy to me.

But IMHO you aren't gonna see how a real botnet works without running real unpatched Windows boxes. I used to keep a box here in the shop for dropping bugs on to find the best ways to clean them (before cleaning got to be pointless) and the amount of crap some of these bugs were screwing with was just mind blowing, we are talking fake .tmp files, stuff hidden in places like program files/ windows media player, a couple that would even rip out different windows system files and replace them with their own hacked versions, just really crazy stuff. But since Wine is primarily a very tiny subset of the Windows susbsystem I really don't see how they are gonna get any real results from this.

If it was just some guys playing in their basement I would think "okay...maybe cool" but spending the amount they did on that "Bigtux" makes it just nuts not to buy an MSDN and run a real simulation. I feel this is a moment where we need the late Graham Chapman to come out in his military uniform and tutu and demand that they cease and desist for being just too silly, because spending all that cash to study Windows botnet behavior and then cheaping out on a ...what? $600 MSDN license? It is just too silly.

Re:Is that really a windows environment?

[Antique Geekmeister]

Lease time on one of the larger botnets?

Wine?

[Facegarden]

I understand using WINE to avoid license fees, but wouldn't that potentially hinder the results of the experiment? I suppose that if they knew what functionality was needed by the botnet, they could be sure WINE provided what they needed, but it also seems like they might be able to work out a deal with MS to get a free site license for use in this test only, since it betters the computing world in general, which ultimately benefits microsoft?

Seems like a few phone calls might go a long way, if they get a hold of the right people.
-Taylor

Re:Wine?

[Facegarden]

...Except for that they basically would have to say "Hey MS, your code is broken, so broken that we need free licenses in order to show the world how broken it is". While it is a great idea and would benefit them, all MS can see is bad press, and they want to avoid that.

I'm pretty sure that the notion of windows being susceptible to malware and viruses is probably something Microsoft has come to terms with, i really can't imagine anyone getting terribly upset. Viruses exist, someone wants to do some research, it shouldn't be that offensive of an idea.
-Taylor

WINE

[Phroggy]

Can a botnet run on WINE with 100% compatibility? Doesn't malware often use exactly the same kinds of tricks that WINE doesn't fully implement? This might not create an accurate picture.

Also, are they simulating network latency between nodes? Many bots take this into account.

Re:WINE

[monopole]

I hope Microsoft issues a statement that only Genuine Windows software can fully support viruses and malware in an effective fashion.

Re:WINE

[Eighty7]

In other news, Miguel de Icaza said that he believes botnet support is a good idea. Linux should support malware because Microsoft is going to win anyway, so linux would better be prepared if it doesn't want to be locked out of the future markets, and presented a beta version of the software. Members of the Mono project are participating in the standarization.

But -- how can you infect it?

[Nefarious Wheel]

My first thought meme was "Yes, but does it run Linux?" ("Megatux". Duh.) Then I thought - hang on, how can you develop a botnet that runs on Linux in the first place? And if you did, how would it reflect the nature of real botnets if those millions of operating systems weren't running NT4 or variants?

Then it got surreal - I imagined all those bots emulating the game of life , with little dots flashing on and off, and little gliders and factories...

Ok, I'll go back to work now.

14 comments so far

[zmollusc]

and nobody yet has imagined a beowulf cluster of these? Standards are slipping!

Wine for viruses?

[Tubal-Cain]

Wine's come a long way in the past 4 years if it can run viruses now!

A few notes from Ron Minnich

[coreboot]

Hi, Ron here. Just thought I would mention a few things.
I love the "life imitates xkcd" aspect. :-)
We're well aware that Wine is not quite enough to run many windows bots. Until a year or so ago, however, there was a researcher in North Carolina running Storm under Wine, but he told me that that effort ended when Storm added a kernel driver. We've got some ideas in that area. We expect that implementing them will cost less than 1 million Vista licenses.
I was surprised to find I have become a cybersecurity expert! What I really am is an HPC expert who is using HPC tools and resources to build a system for studying cybersecurity phenomena on a millions-of-nodes scale.
Doing anything with a million of something gets interesting fast. There's a lot of interesting challenges.
Thanks
ron

Re:A few notes from Ron Minnich

[PCM2]

Well Ron, since you're here, I'm curious whether you had in fact tried to approach Microsoft for a free site license. You could explain to them that you're doing security research in a unique environment and that you'd be willing to share your results with them, etc. I could even imagine a distorted PR spin where the fact that all this major security research is being done on Windows shows that Windows is clearly the dominant operating system, blah blah...

Or if Microsoft doesn't see the value of the kind of information your research could yield, maybe someone like Symantec would be willing to buy a license and donate it to you (if that's even possible, given EULAs etc.)?

Re:A few notes from Ron Minnich

[coreboot]

We will probably approach MS at some point, if it appears to be necessary, and see if they are interested. I do have friends there who might be interested in what we're doing.
The biggest limit we've found on the VM side is memory footprint of the VM guests, and it's very easy to control that with Linux; harder with Windows. We have some ideas in that area too, but it's way too early to speculate on them.
But from my point of view, it is a lot easier to do this kind of work in Linux than in Windows (I have done NT drivers in a past life), not least because of the openness of the environment. Hence, I'd rather try to find a way to make it all work on Linux.
Consider this work the beginning of the story; it's not even chapter 1, maybe it's the preface. There's a lot of work left to do. There's a lot we still don't know.
thanks
ron

Re:A few notes from Ron Minnich

[PCM2]

Do you really think it would be easier to set up (and periodically reinstall) a million copies of Windows vs. telling Linux to virtualize a million instances?

I'm assuming they would do both. If they didn't have to individually license each Windows instance, it would be trivial to clone a million virtualized instances of a fresh Windows install. (I'm sure he's right that this would make resource management more difficult/costly than using WINE, however.)

I would guess it wouldnt' be a problem at all

[Sycraft-fu]

I work for a university and MS is extremely generous with academic licensing. When it is for academics, like education or research, it is actually no cost. For infrastructure it does cost, but not very much. I bet if they asked MS, MS would give them all the licenses they needed for little or no cost.

For that matter, they might be eligible for volume licensing. That is where you pay a fixed yearly fee and get an unlimited use of the software it is for. Often that is based on total academic headcount, which might not be very much.

Regardless, if they asked I'd give good odds MS would figure out a way to offer them a good deal.

I'm also with you that if you want to study something, you need to run it on the actual environment. Wine is a neat idea and a neat goal, but anyone who has made use of it for more than simple testing well tell you that it has some serious issues. Not only do things not run, worse is that they'll run but not completely correct. For a user this might be fine, something works in a bit of an unexpected way, you just work around it. For research though, it could mean your conclusion is invalid.

Re:Wine on Linux?

[geegel]

Not necessarily.

You might want indeed at some point to emulate an internet choke full of unpatched machines, but other times you will probably want only a percentage of them to be this way, or you might want to study a particular vector of infection, or concurrent vectors of infection to see how they interact. The combinations are endless and so will probably be the number of WINE flavors used.

What about Norton Antivirus?

[node 3]

What about Norton Antivirus? Specifically they should run a second experiment with a simulation of 1 million systems running Norton Antivirus, and compare the results of the first test to see which has the greatest adverse effect...

Not exactly.

[khasim]

A patent on an IMPLEMENTATION of an idea is a good thing.

A patent on an idea itself ... that's stupid. And that's what we're stuck with today.

Old News...

[davevr]

There is already a system running somewhere around 420 million windows machines in a semi-private walled-off version of the internet, with no license fees paid to Microsoft, hosting several botnets and just about every virus under the sun.

It is called "China".

Linux for a reason...

[Entropius]

The researcher posted up above saying he's an HPC researcher, not a computer security guy, and in that context using Wine makes sense.

HPC people typically study emergent behavior -- how a lot of nodes interacting by simple rules generate complicated phenomena. The challenge is coming up with the simple rules in a form that accurately captures whatever leads to the emergent behavior you want to model. In this case, "actually being Windows so all the viruses work exactly right" is less important than getting a lot of nodes running to capture the interesting behaviors of viruses spreading through a large network.

Supercomputing is difficult on Windows. I'm at a computational physics conference now, and everything runs on Linux just because it's bloody *easier* to make everything go. I doubt many people here would even know *how* to run our models on a Windows supercomputer.

Performance issues aside, my guess is that the fellow chose Linux because the computer *already* ran Linux.

It's "The Matrix"

[Michael Woodhams]

for bots. Poor little things think they're in the real world.

Dell Makes Supercomputers...

[longacre]

And in related news, Hummer will join the Formula 1 circuit next season.