Network Testbed Emulab.net

gseidman writes: "Have you ever needed to simulate a network? Tired of old ns? Do you just hate dealing with hardware in general? Take a peek at Utah's Emulab.Net. They have over 300 PCs, some StrongARM devices, roughly 5 miles of cabling, a huge and expensive switch, and great software for setting up a virtual LAN. They also have a gallery showing the machine room in various stages of completion (did I mention five miles of cabling?)."

Comments

Gonna loose some Karma for this...

[jeffy210]

ooooh... what about a Beo.... ah, forget it.

This is very cool.

[perdida]

This is applied research, the type that may have direct and positive effects on improving security and efficiency in the immediate future.

Unlike dark matter research, Mars colonization, and subatomic research, this stuff is the kind of thing that should attract wide funding from business. Immediate payoffs are likely.

Basic research is fine, but I wish that the money poured into it would go towards immediate business applications. More available cash would make those venture capitalists a lot nicer and less demanding of unrealistic profits in an unrealistic period of time.

Re:This is very cool.

[fnordboy]

Basic research is fine, but I wish that the money poured into it would go towards immediate business applications.

Now that's just silly. Basic research is incredibly important, and it is vital to the economic health of the country (and the world, for that matter) that money is spent on it. Where did the transistors that your nice new Intel chip is made up of come from? Basic research. How about lots of medical technology like MRI machines and x-rays? Basic research. And there are lots of indirect benefits to basic research as well. How about those snazzy digital cameras? The need for high-quality CCDs for astronomy (*cough* hubble *cough*) and for other research applications pushes that. Do you like the world wide web? Thank a bunch of physicists who put it together so they could share their data.

The point of basic research isn't the small, immediate payoff - it's the hope that somewhere along the line, some scientist is going to come up with something that will revolutionize the world - just like the transistor! So I respectfully disagree with you - while it's important for companies to be concerned with their quarterly earnings reports, in the long term, basic research is most certainly worth the investment.

Re:This is very cool.

[LS]

If businesses feel that they would gain from research, they can do it themselves. Look at IBM and Lucent. University research should no be based on short-term goals of funding, but instead on the researcher's interest coupled with the current direction of the science. I hope that the short-term thinking which placed the US economy and world environment in it's current state does not further infect the university and stain the ivory towers.

LS

Emulating the /. effect

[Brento]

It's already starting to slow down. Now would be a good chance to start dedicating some of those machines as backup web servers, eh?

This could be a nightmare

[Quasar1999]

5 miles of cables? I hope they labled them... :)

Re:This could be a nightmare

[mac.newbold]

5 miles of cables? I hope they labled them... :)

Every wire (~1000 of them) on both ends, then the endpoints get recorded in a database.

And anyone who moves a wire without properly documenting it gets shot! ;)

Mac

Re:This could be a nightmare

[b1t r0t]

As a bit of perspective, I used up about 1/3 mile (most of two 1000 foot boxes) of cat-5 wiring up my house. (eight six-wire bundles of cat-5, plus 300 feet more of RG-6) They used up fifteen times that. Of course it's a little easier to label the ends when you're only pulling six at a time.

Wow, there's nothing like a picture of a Cat-6500 switch full of ethernet. Except maybe a Cat-6500 full of gigabit fiber.

5 miles, big deal.

[scott1853]

They probably didn't have a 2 ft. high crawl space that they had to run it through.

Re:5 miles, big deal.

[mac.newbold]

I was one of the few 'lucky' people who had to run it, and no, we didn't run it thorough a 2ft high crawl space. It is even worse than that.

In case you couldn't tell from the pics, this is all in self contained racks. The large majority of the wiring is in 9 standard-sized racks, or about 7ft tall * 3ft deep * 2 ft wide * 9 racks = 378 cubic feet for about 5 miles (25,000 ft) of cable plus all the PCs and switches.
As a generous estimate, that leaves 100 cubic feet for cables and ventilation. That says the every cubic foot of open space is filled with an average of 250 linear feet.

Needless to say, it was not fun.

Mac

Re:5 miles, big deal.

[OWJones]

Having worked with the Centurion cluster at UVa for a few years, I feel your pain. And this was before we added 10 such cabinets of rack-mounted PCs. :)

-jdm

Simulation is never perfect

[euroderf]

This is the difference between book learning and experience. Sure, you can 'test' a network under idealised simulated conditions, as one might 'test' a plane in a wind tunnel, but till the network is reality, and the secretary spills a cup of coffee on the router or the chief engineer urinates drunkenly on the packet switcher, you can't tell how your network will perform.

What is it with the modern generation who think that simulations will improve their likely performance? It is all idiocy, when I was young we did things with a spanner and looked at das blinkenlights under real world conditions. This is so much nonsense, really, the sort of thing I'd expect to come out of our modern CS courses.

Computing is just an offshoot of down-and-dirty engineering, and none of us should forget it. The day we forget the feel of the netowrk cards in our hands, the smell of the overheating cat5, is the day we lose control over the netowkrks of America, the day that our economy starts going backwards.

Re:Simulation is never perfect

[sharkey]

...chief engineer urinates drunkenly on the packet switcher...

Yes, but what about urine acidity and volume?

Re:Simulation is never perfect

[John_Booty]

"but till... the chief engineer urinates drunkenly on the packet switcher, you can't tell how your network will perform"

If anyone needs help testing this condition out, I'm available.

Re:Simulation is never perfect

[bluGill]

I partially agree, partially agree.

Simulation isn't perfect, but it is a lot cheaper than full blown tests. If you solve all the problems in simulation that you can, then you have saved a ton of time any money over doing real world testing. Expirence in the real world will help you make the simulation better.

Where I worked we once modified an old router to package FDDI packets over ethernet (note, we artificially slowed the sender), transmitted them to a sun IPX (a heavy duty machine for its time, though unbearablly obsolete today), ran some hardware simluation to send it through a fddi interface, through a custom routing board, through a backplane, to a ethernet board, and then took the packet to a different router, unpackaged it, and put it on a real network. We were accually able to do telnets through our hardware simulator, so long as we keep the packet count down to about 1 a minute and could deal with long latiancys. The result is we found a lot of bugs in simulation before hardware was built. Eventially this was the first switch on the market, beating Cisco to that mark by about a year. (Of course Cisco was still the go to network provider, and their switch could do more packets, but we beat them to market)

A friend of mine works for a company that makes injection molds. the old expirenced engieers could sometimes get an acceptable mold after 3 prototypes, though 6 was considered normal for an expirenced engineer. They hired a new college grad a few years back who with simulation always gets a perfect mold on the first try. Not acceptable, perfect. With simulations he could watch hot spots, and make the changes to cool them and/or account for different contraction rates in the mold. Since a mold costs a lot of money to make, (but lasts a long time) this results in considerable costs savings for the company. The new engineer can also turn out more molds in a year because he doesn't have to analyise the failures and guess what went wrong, he knows.

that said, I agree fully that simulation can't do everything. The new engineer above still sometimes turns out a mold that fails to work, but failure analysis improved the simulation next time. When we simulated our new router previously we still found a couple hardware bugs that didn't happen in simulation. for computer simulation we can't do stress testing without real hardware, but we already have solved most of the problems by that time.

Re:Simulation is never perfect

[Minupla]

What is it with the modern generation who think that simulations will improve their likely performance? It is all idiocy, when I was young we did things with a spanner and looked at das blinkenlights under real world conditions. This is so much nonsense, really, the sort of thing I'd expect to come out of our modern CS courses.

#include humor/sarcasm.h
Ya, and what is it with those aerospace engineers who think they can simulate anything in a wind tunnel. I mean, come on, the only way to see what will happen in realworld conditions is to build a multimillion dollar airplane and see if it crashes. When will they learn?

5 Miles of sloppy cabling...

[GLX]

I don't know what brainchild designed the layout of those switches, but the fact that they went for the RJ-48 blades versus the Telco (RJ21X) blades in those switches shows sloppy planning. What if one of the blades in the middle goes out? They have to unplug 48 Cat-5 cables versus unscrewing 4 telco harmonicas and leaving the 48 ports untouched on a patch panel

If you're going to put forth that kind of effort and money, why not do it right the first time?

Re:5 Miles of sloppy cabling...

[monkeydo]

Not only that, but they have 7 slots worth of 48 port 100Mb cards (33.6Gb) and 1 Gb uplink. That might be a little of a bottleneck if more than a few of the nodes are trying to pass upstream traffic.

Re:5 Miles of sloppy cabling...

[mac.newbold]

There are several reasons we chose not to use the Telco connectors:

1. Then we'd have to go to a patch panel, and then the ciscos, which greatly increases the number of wires and the time it takes to run them.
2. The telco connectors aren't as dense enough to be worth it... 8 wires per cat5, 50 wires per telco, so we'd only consolidate 6 wires at a time.
3. We eventually want to run gigabit over copper with these cat5e wires, and if we put them into telco connectors, we can't do that.

Does that answer your question?

Mac

Re:5 Miles of sloppy cabling...

[GLX]

Not especially.... To answer in order:

1) It increases the number of wires by 4 per 48. It however lessens the trouble involved if a module fails - apparently you've never had the joy of unplugging, keeping track of, and re-plugging in 48 Cat-5 cables in a hurry. It's not fun. An increase of time in the beginning far outweighs the risks of the increase of time in an outage. I can show you pictures of rats nests and tell you horror stories all day about this.

2) The standard Cat-5e configuration still only uses 4 wires. The Telco panels are wired as such. For each Telco harmonica you get 12 ports - quite dense enough.

3) There's nothing that says that gig won't be supported over telco, just like there's still no set-in-stone standard for gig over cat-5. Nothing even says that cat-5e is going to be required.

Re:5 Miles of sloppy cabling...

[mac.newbold]

1) It increases the number of wires by 4 per 48. It however lessens the trouble involved if a module fails - apparently you've never had the joy of unplugging, keeping track of, and re-plugging in 48 Cat-5 cables in a hurry. It's not fun. An increase of time in the beginning far outweighs the risks of the increase of time in an outage. I can show you pictures of rats nests and tell you horror stories all day about this.

It increases the number of wires by 8 per 48 (see below), and yes, we've had module failures before, and I have moved 48 cat-5 cables in a hurry. These module failures are so rare that its not even worth the extra time at the beginning to try and make it easier.

2) The standard Cat-5e configuration still only uses 4 wires. The Telco panels are wired as such. For each Telco harmonica you get 12 ports - quite dense enough.

It uses 4 wires when running at 100Mbit, but like I mentioned before, when we go to gigabit over these cat5e cables, we'll need all 8, since the gigabit over copper products we're looking at use all 8 wires in the cable.

3) There's nothing that says that gig won't be supported over telco, just like there's still no set-in-stone standard for gig over cat-5. Nothing even says that cat-5e is going to be required.

Every product on the market that I've seen requires cat5e for its higher standards, and I've never seen a patch panel/telco combination that claims to be able to support the high requirements of gigabit over copper.

Now maybe you would have done it differently, and that's fine. Our needs are different from yours, and our criteria for judgement of our options are probably quite different as well. What we chose to do has worked out very well for us, and we're very happy with the way that things are set up. If you're still not satisfied, perhaps we should just agree to disagree....

Mac

Agreed

[clark625]

I totally agree with you. Simulation doesn't prove that things will be all peachy cream later on in the "real world".

Then again, if a 1/100th scale model does a nose dive in the wind tunnel when they throw a little turbulence at it, you can bet the ranch that plane will be redesigned. Testing and modeling only show the existence or non-existence of very particular problems. We just usually hope that we can "test" out as many likely problems as possible.

Just how "virtual" can it be...

[JesseL]

if it uses 5 miles of cable? ;-)

Re:spamming us /. ??

[Atilla]

Have You Ever Heard of Sarcasm?

Only 5 miles?

[yzquxnet]

The post submitter makes 5 miles of cable seem like a lot. Well, it isn't. Even a small office complex can burn through 50,000 ft a cabling in short work. Running two data cables effectivly doubles your cable requirements. Depending on the situation the installer may even lay a third cable for voice. (What standard was that?) Anayways, 5 miles of cable is not a lot.

I work with Utah

[OWJones]

A group here at Duke has just started to collaborate with the Emulab people on a similar area of research. We're pretty excited, since we think our group has a lot to share with the Utah group. It's just kind of weird to load Slashdot and find your collaborators on the front page. Here I was thinking I'd take a break from research and read Slashdot, but noooooo .....

And to respond to another poster, maybe if Emulab had Muse-like resource management of their web server, they could handle the web load. :) It would be nice if someone could work on merging the two, getting the two to leverage the best parts of both, work on .... uh, maybe I should get back to work before my advisor sees me posting on Slashdot. :)

-jdm

randomness

[johnjones]

yeah great

much of the intresting things about networks has been done through live capture of a system and then sifting the data

phone networks work well because they have had a long time trying things out and finding optermal solutions to random problems that come up in live networks (problems with GPRS networks where only seen once semi live)

also you dont get the scanning and general attacks that do strange things to routers

its like saying yep we have 1000 monkeys and typewriters but we need you to go through the data and compare it to the real world

you might as well do it in the real world how expensive is a switch/hub and network cards ?

silly but nice

regards

john jones

Just a bit surprised ?

[pruneau]

Those guys just bought those 4 huge 6509C Cisco switches for big bucks.

I assume that One is acting as the main router, and the other are using it. I assume also that the switches are interconnected with 2x1Gb/s fibers, probably full-duplex and load/sharing above the two links.

What surprises me is that those switch fabric supports up to 256Gb/s bandwidth, but they are just connected with 2Gb/s links : talk about some bottlnecks here...

But maybe I did not read enough documents ?

Any comments from the builders of that lab ?

Notice the hardware config?!?

[Geek In Training]

Boy, I wonder if one of the projects they're planning to emaulate is "wide scale hardware failure." Look at their node configuration (hard drive boldfaced):

128 new nodes:
850Mhz P3
512M ECC memoryold reliable BX chipset
40G 7200rpm IDE disk (IBM Deskstar 60gxp)
5 Intel Pro/100+ network interfaces
2 on board
1 on a single Intel card
2 on a dual Intel card
No video at all
serial console

This is the very same hard drive drive we drew and quartered here, and has gotten IBM a big fat lawsuit for rampant failures.

So, I guess their error recovery is going to be tested to the limits very shortly, especially with the space/heat issues inherent in the installation exacerbating the engineering flaws in the 60GXPs. :)

Re:Notice the hardware config?!?

[A Commentor]

Wrong... that was the 75GXPs. See the original article.

I double checked a few posts, and they mentioned that the 60GXP was much more reliable.

Simulation need not be perfect to be useful.

[FallLine]

While I agree with much of your sentiment, hardly being a fan of the Ivory Tower, if you take a look at some of the projects on the website, you'll see that many of them are things that are hard to simulate AT ALL in the real world. For instance, there are projects there to research methods of stopping DDos attacks. They can hardly start flooding a real WAN to test their software. While it is true that they can fall back to mathematical models, these are complex and can never hope to realistically simulate some of the non-deterministic elements nearly as well as a real WAN (or this network).

That said, I also can't say that this network really has SUFFICIENT usefulness to justify its existence (after all, how many people need to test denial of service attacks and such?); we'll let the markets resolve that, eh?

5 miles? pfft.

[austad]

In a 30,000 square foot office space, with only 50 employees, we have 42 miles of cat-6. 5 miles is only 25 spools of cable.

The office was wired for 150 people, so we're not using even half the ports. And since this particular office is all tech people, everyone has at least 2 machines on their desk (I have 4).