Slashdot Mirror

How about a linux based solution?

For quite a while linux has been fine for any networking need that hasn't required over 40gbps over a single link. Added bonuses of consistent interface as other linux items, multiple hardware sources so you can find the cheapest bidder for what you need, and it scales to a point where only major internet backbones and telecom providers would typically require more.

A 24-port 10/100 with 2 port 10Gb will be a killer product when it emerges, is standardised, and cheap enough. Hell, I could use it NOW.

The future is here! 10GBASE-T was standardized over 5 years ago, and fiber variants before that. Every major manufacturer's midrange fixed-config edge switch lineup has a 24/48 port 10/100/1000 switch with dual 10Gb uplinks.

Just a few examples:

http://www.cisco.com/en/US/products/ps6406/index.html
http://www.extremenetworks.com/products/summit-x350.aspx
http://www.brocade.com/products/all/switches/product-details/fastiron-gs-series/index.page
http://h30094.www3.hp.com/product.asp?sku=3981100&mfg_part=J9146A&pagemode=ca

Our department uses Extreme hardware throughout in a fairly large network deployment.

We have two Linux clusters -- Viking, a 512 processor P4 cluster, and Mars, a 400 processor Opteron cluster (being commissioned). We also host a number of large Sun machines, including SunSite Northern Europe.

The department also hosts a 250+ node teaching lab and several floors of staff and research desktops, each with 100Mbit+ to the desk and a 1Gbit uplink from each switch. At the middle of our network are two Black Diamonds, one of which is the new 10ks which we helped Extreme beta-test (now in production). A third BD is being set up at a second site with 10Gbit fibre links.

The rest of the network ('the edge') is also made up of Extreme switches, ranging from Summit 1s, 24s, 48s to some of their newer 400-series 1u switches on Mars and elsewhere. (The 400-series can run 10Gbit fibre uplinks, and each copper port runs at up to 1Gbit, which is ideal for a cluster environment.)

Plus, they're are all coloured a nice shade of purple.

Our department uses Extreme hardware throughout in a fairly large network deployment.

We have two Linux clusters -- Viking, a 512 processor P4 cluster, and Mars, a 400 processor Opteron cluster (being commissioned). We also host a number of large Sun machines, including SunSite Northern Europe.

The department also hosts a 250+ node teaching lab and several floors of staff and research desktops, each with 100Mbit+ to the desk and a 1Gbit uplink from each switch. At the middle of our network are two Black Diamonds, one of which is the new 10ks which we helped Extreme beta-test (now in production). A third BD is being set up at a second site with 10Gbit fibre links.

The rest of the network ('the edge') is also made up of Extreme switches, ranging from Summit 1s, 24s, 48s to some of their newer 400-series 1u switches on Mars and elsewhere. (The 400-series can run 10Gbit fibre uplinks, and each copper port runs at up to 1Gbit, which is ideal for a cluster environment.)

Plus, they're are all coloured a nice shade of purple.

Extreme Networks has a great line of switches.

The Black Diamond 10808 would work great for the type of envrionment you have setup from the sounds of it. Also, Extreme is usually 20-40% cheaper then Cisco and Foundry for the equivilant appliance.

We currently use an Alpine 3808 with 192 100mbps ports and it's never had a problem with uptime and configuration is a simple and straightforward.

Extreme Networks has a great line of switches.

The Black Diamond 10808 would work great for the type of envrionment you have setup from the sounds of it. Also, Extreme is usually 20-40% cheaper then Cisco and Foundry for the equivilant appliance.

We currently use an Alpine 3808 with 192 100mbps ports and it's never had a problem with uptime and configuration is a simple and straightforward.

Extreme Networks has a great line of switches.

The Black Diamond 10808 would work great for the type of envrionment you have setup from the sounds of it. Also, Extreme is usually 20-40% cheaper then Cisco and Foundry for the equivilant appliance.

We currently use an Alpine 3808 with 192 100mbps ports and it's never had a problem with uptime and configuration is a simple and straightforward.

I have never used their routers, but 3Com's switches have always been of great quality. We used to use them exclusively at the college I work at, but we have been phasing them out in favor of Extreme Networks equipment over the last 3 years. Why did we dump 3Com? Because of the core switches. 3Com killed support for the models that we and every other college in the state had, and did not offer the options that Extreme and Cisco offered in their products. Several of the colleges actually needed the expandability a Black Diamond offers. 3Com just did not have the products to do the job for all of us.

I agree that reliability is a key component of the cost/value equation. One addition I'd make to your comments is that these licensing policies drive up the total cost of owning the hardware so if their is a competitive product out their... it's worth taking a good hard look at. Of course this is a moot point if the you need a Cisco feature that the competition doesn't provide.

The 6500 is a rock solid core switch platform (and you pay for it). Since we're talking about Cisco though it's worth mentioning that Foundry Networks and Extreme Networks both have good quality equipment and in some cases can be used in place of the 6500.

Seems like a good idea

The long answer is that if you haven't done this before you better get some consulting help. Chances are you are talking about a ring topology and are going to be linking sites with different networks.

Perhaps you have telco supplied networks for each office hooked up over DSL or T1. If that's the case, have fun getting routing working without having the telco people disconnect you. Good luck reconfiguring the telco routers for that matter.

Routing complex networks is tough. Do you already have VPN's interconnecting these sites? Are you going to be introducing redundant routes? How are you going to manage these routes? IOS can suck if you've never seen how to configure routing processes. Routing software is also complex. Ripv2 is about as simple as it gets and it doesn't offer much control over which route you take -- the only metric is hop count. OSPF has design guides as big as phone books.

If you have a bunch of nats at your different locations, do the networks overlap? Are you going to have to renumber your networks?

If you are just playing in your spare time, you won't be able to do this for under a few grand. Fibre connections are generally not cheap. If you're lucky you could put a few fibre nics into a couple of linux boxes, but I don't foresee those nics being under $400 each.

If this is to be a business network, do it right from the start or you'll make yourself look stupid. People expect the stuff they don't understand to just work. There will be very little tolerance if services are going up and down and your fibre links are to blame.

I have not seen a 48-port GigE switch either, though Extreme does make a 32-port GigE switch (total of 32 10/100/1000BaseTX copper or 1000BaseSX fiber ports).

http://www.extremenetworks.com/products/datasheets /summit7i.asp

64 Gbps switching fabric and 48 Mpps switching performance... you're not going to find this at CompUSA and I think the list price would scare me!

Given the fact that you want no video, it seems you need servers on the cheap. If you're actually buying thousands (or really more than ten or so) of machines, I suggest you go with an integrator like XRam or Fnord. They'll build whatever you want for you, configure everything EXACTLY the same, test everything, install & do a basic config (i.e. network config and root password) for any OS you want, and install your machines on site. I'm sure the'll give you a nice quantity discount if you buy in the numbers you're talking about.

Don't even THINK about deploying that quantity of machines without racking them in a proper datacenter type environment (cooling, ample redundant AC power feeds, generator, decent physical security, etc.); FORGET about normal PC cases on Ikea shelves in your basement/office. Whatever cash you would save doing it the ghetto way is absolutely not worth the headache of blowing breakers, having your ambient temp at 35-40C and grilling PC parts if your cheap-o electro-cool chiller dies or spills its bin of water all over the place, your local power company decides you're not important, etc.

That being said, here are a few links for what I'd build if I had to do it myself on the cheap (try googlegear.com for good qty. 1 prices on this stuff:

Elite K7S5AL mobo (integrated lan)
1.2 GHz AMD Duron with a really good fan (i.e. Tai-Sol or similar overclocker freak fan)
at least 512M of brand-name CAS2 ECC DDR SDRAM
Western Digital JB series hard drive (WD800JB or WD1200JB)
Netgear FA311 NIC

The cheapest 2u rack case on the net seems to be the Electroseller IPC-2025 at $118 without power supply and fans. It takes nomal ATX size CDRom, floppy, power supply, fans, etc. pricewatch is your friend (-:.

This should net you a pretty sweet 2u rackable server for about $500. I envy not the man that has to assemble more than about 10 of these things by himself. Maybe those chainmail gloves that people that shuck clams for a living use would help.

If you want to do something with the data on these machines, you'll probably want to stick a pair (yes, a pair) of big ethernet switches in front of them. I suggest Foundry or Extreme. You can buy these "certified used" from BizInt.

"Imagine a beow..." (-:

I can't remember but if this was ever true, but it isn't now.

Extreme Networks and Cisco both have long haul GBICs available, and have done for a little while.

I don't have info on the Cisco models available right now (Ciscos website is BIG!), but Extreme have a 1000BaseLX-70, that will do 70 km on singlemode dark fibre. On top if this they do what I could best call a gigabit fibre line driver in the form of a "SummitGbX"[tm]. They claim it will definately do 80 km, and possibly up to 100 km, I have heard one claim that these units managed to reach 120 km. Basically, you hook these up to your 1000BaseSX interfaces at each end, and it does some wibbly-bits to bridge the 80 km or whatever length of fibre :)

I would love to verify all this independently :) but I never even dared ask the prices :/

Of course, all this doesn't really help your cause much... you might be able to find a friendly telco that will blow fibre in for you. Apart from that, your realistic options might not include wireless.
At that range, you will have to go for some sort of microwave transmission, and even then you will probably have problems due to the Fresnel effect, which bends and scatters photons at the earths surface. The maths escapes me at this moment, but to reach 45 miles in one hop you would probably need to have each end nearly 200 foot up in the air in order to clear inconvenient obstacles in between, like buildings and trees (how careless to put them there!). To do it in several smaller hops might be easier, but then you have to rent or buy locations to put your repeater stations on.
Another possible thing to do would be to link all the schools locally to a central point using some easily available method. Microwave links might be suitable here, as well as optical wireless links, T1 leased lines, or whatever you chose. The most usefull central point would be a telco CO, which means that you will only have to rent backhaul bandwidth on the telco's network without having to pay for an expensive tail to anywhere else. Almost all CO's will be served by fibre now, which makes renting a fast connection very much cheaper and easier to provision. As you well know, it is that last mile that makes it expensive.

I wish you good luck :) maybe you'll let us know how it turns out?

I can't remember but if this was ever true, but it isn't now.

Extreme Networks and Cisco both have long haul GBICs available, and have done for a little while.

I don't have info on the Cisco models available right now (Ciscos website is BIG!), but Extreme have a 1000BaseLX-70, that will do 70 km on singlemode dark fibre. On top if this they do what I could best call a gigabit fibre line driver in the form of a "SummitGbX"[tm]. They claim it will definately do 80 km, and possibly up to 100 km, I have heard one claim that these units managed to reach 120 km. Basically, you hook these up to your 1000BaseSX interfaces at each end, and it does some wibbly-bits to bridge the 80 km or whatever length of fibre :)

I would love to verify all this independently :) but I never even dared ask the prices :/

Of course, all this doesn't really help your cause much... you might be able to find a friendly telco that will blow fibre in for you. Apart from that, your realistic options might not include wireless.
At that range, you will have to go for some sort of microwave transmission, and even then you will probably have problems due to the Fresnel effect, which bends and scatters photons at the earths surface. The maths escapes me at this moment, but to reach 45 miles in one hop you would probably need to have each end nearly 200 foot up in the air in order to clear inconvenient obstacles in between, like buildings and trees (how careless to put them there!). To do it in several smaller hops might be easier, but then you have to rent or buy locations to put your repeater stations on.
Another possible thing to do would be to link all the schools locally to a central point using some easily available method. Microwave links might be suitable here, as well as optical wireless links, T1 leased lines, or whatever you chose. The most usefull central point would be a telco CO, which means that you will only have to rent backhaul bandwidth on the telco's network without having to pay for an expensive tail to anywhere else. Almost all CO's will be served by fibre now, which makes renting a fast connection very much cheaper and easier to provision. As you well know, it is that last mile that makes it expensive.

I wish you good luck :) maybe you'll let us know how it turns out?

I would have to say that the most exciting startup (at least for me) would be Extreme Networks (http://www.extremenetworks.com).

Having been associated with company since the beginning, I would have to say that competant employees that are willing to work their asses off are the most important thing. Also, make sure you know what you are doing before hand, PLAN EVERYTHING!!

"If we don't anticipate the unforcene, we are at the mercy of anyone or anything that cannot be programmed, catagorized, or easily referenced."

Also, look at other companies as examples. See who goofed up, and where people at the most trobules. Learning from other people's mistakes is a lot nicer than learning from your own!