Is the Distribution Layer Still Needed?

arnie_apesacrappin wonders: "I'm in the process of designing the network for a new building in what I would consider a small to medium sized company. It is on the scale of tens of access layer switches, not hundreds. There is a ongoing argument about the need for a distribution layer. My position is that with today's layer 2/3 switches in the core, the distribution layer is outdated for a network of this size. The layer 2/3 core can provide all the aggregation services of the old distribution layer and the routing/filtering functionality of the core with better price and performance. My opponents can only argue that having a distribution layer is the standard. So, are there good reasons for having a distribution layer in a small to medium network? If you were going to argue against the distribution layer, what points would you make?"

Comments

Glad you don't work here.

[grub]

Quit trying to be clever. Proper use of L3 equipment around the LAN and judicious use of VLANs is smart. Current equipment will let you design in redundencies for failed hardware so trying to aggregate all your networking smarts to a central point of failure is not cool. Frankly it sounds like you're trying to impress management without thinking of the ramifications.

Re:Glad you don't work here.

[Anonymous Coward]

haha "Flamebait"? you must have pissed off an MCSE with mod points.

What?

[Bootle]

All your technical mumbo jumbo is leaving me bamboozilified. Could ya tone it down a tad?

Re:What?

[Anonymous Coward]

Mr. President, what have we told you about posting on Slashdot?

It can be done.

[FreeLinux]

Removing the distribution layer is perfectly possible. The main requirement though, is having sufficient processing power and redundancy on the core to handle the access layer's connections.

Basically, if you eliminate distribution, you have to have a lot more processing power and lots more ports in the core. Depending on the network's size and distribution it will probably be more costly to build such a robust core. Also, don't forget that this thing is certain to grow. Can it scale easily and cost effectively with the more robust core? There will come a point that it will not scale effectively and the distribution layer will have to be introduced.

Re:It can be done.

[Kaamoss]

That's the real key, if the network can't be scalable then you're not setting your self up to do further work for the company. When you give someone a solution it should have the ability to grow with them. In the end it's almost allways cheaper to go with the more complete solution than the simple one.

Spoken like a true CCNA

[Schezar]

The very concept was never spoken of at university (Rochester Institute of Technology), nor has it ever come up in work (IBM).

Those three "layers" are abstractions, nothing more. The "distribution" layer is simply a term for traffic shaping and optimization. It's very useful in eliminating excess resource use on beleagured routers. Eliminating the layer is nothing more than simplifying your backbone architecture. There is no "layer" to eliminate except the theoretical one.

It always amazes me how Cisco-certified (not making any acusations here) network techs speak an entirely different language from university-educated ones. They talk about Cisco-specific concepts like they're set in stone universally, and use Cisco jargon for common and/or basic concepts.

There are other options besides Cisco, and not every network fits within the nomenclature of Cisco Jargon. You'd do yourself an immense favour to lean more about generic architecture concepts.

I don't want to sound mean, but a Cisco cert is about as useful as an MSCE.

Re:Spoken like a true CCNA

[brunson]

I love the fact that the very next posting after this one starts with "To preface, I am a CCIE, so I know a little about these things."

To the CCIE's defense, he gave a balanced and reasoned opinion that was not rife with Cisco jargon.

I, on the other had, find it interesting to talk to university educated CS majors (I graduated with a BA in Math, but have worked as an SA and programmer for 17 years) who use lingo (especially pattern nomenclature) and discuss concepts (stateless session beans, oooooooooh) on a level that makes it clear that they a) have never written code in the real world and b) don't have a clue about the mechanics of how things actually work (what do you mean I can telnet to an SMTP port and send mail)?

Re:Spoken like a true CCNA

[dpilot]

Is the real purpose of the "Distribution Layer" to distribute revenue to Cisco?

Reading TFA and the other posts, I can see the point of reaching for that degree of networking control in a big enterprise. At the other end of the scale - the home LAN level, network bandwidth is practically where nuclear-generated electricity once promised to be - too cheap to measure. I went from 10Mb to 100Mb because it was cheap and available, not because of need, and any future migrations will likely be the same.

You're obviously in the middle, between the home LAN and the big enterprise. I suspect the degree of network control you need depends more on your usage than simply the size of your network. For instance, do you have automated tools that periodically dump a Gig of data to a fileserver? Do you feel it necessary to detect and/or prohibit employee net activity? Do you have multiple sites, or other circumstances that form weak links that need extra control? How is your backup architected, and does that constitute a weak link in the network? Or your fileservers, for that matter?

Finally can you just deploy the architecture you're proposing, but make sure the equipment you buy can fit into a 3-layer with a little reconfiguration. That may become necessary as the company grows, too.

Re:Spoken like a true CCNA

[NoMoreNicksLeft]

Yeh, but did your fancy university classes teach you how to route appletalk with EIGRP? Huh? Huh? Did you learn how to bridge CDP across legacy switches? Did you even learn why open standards like OSPF may not be the best choice in a modern high-powered network, you savage? I think not.

Re:Spoken like a true CCNA

[benjamindees]

I've had experiences that negate both of these presuppisitions. At the small University where I went, I literally watched the ethernet equipment being installed. I then used it to do most of my assignments via a remote X session to the lab computers, from my dorm room. Professors looked at me dumbfounded when I told them why I wasn't attending labs any more. They wondered what I had "hacked" in order to be able to do that.

When I applied for a job at the same University as Network-something-or-another years later, they wanted someone with Cisco certs. I'm not sure if they even had more than a couple of pieces of Cisco equipment. I wrote a nice cover letter detailing my experience, and how it was not Cisco-specific, but was isomorphic to Cisco-specific concepts. I assumed that they, working for a *University*, would understand the difference between branded jargon and universal concepts. I made a point to ask for quite a bit less than what any self-respecting Cisco certified tech would ask for. I made it clear that I was a quick learner and quite flexible in my capabilities. What I got was a call back asking if I had a Cisco cert :)

On the other hand, though, try asking a professor how encryption works someday. You'll get a basic explanation of how data can be represented as ones and zeros, and by adding a key, you can obfuscate the message, and the receiver can subtract the key to get the original back. Anyone with a tenth grade education can understand that. Hell, most tenth graders could *implement* that. Now, I defy anyone to explain to the average person how to implement this simple concept with, say, OpenSSL, without using a lot of buzzword-laden crap.

There are idiots who know nothing but repeating technical jargon almost everywhere nowadays.

Re:Spoken like a true CCNA

[prof.morbius]

I think you're underrating the value of a CCNA, but you're right that the program doesn't present alternatives and uses different terminology than the Real World. That's what bugged me about it while I was taking the program; the education was OK, the indoctrination was a pain. Of course, the real problem is folks who (like those the original poster is arguing against) accept the Cisco Gospel at the expense of actual needs analysis.

Re:Spoken like a true CCNA

[WgT2]

I love the irony you bring to light about university settings.

One would think those working for and under university level expectations, and job applications requiring EVERY-SINGLE-JOB-YOU-EVER-HAD to be listed with it, to be somewhat on the ball about how to do things, at least efficiently.

But, no.

What is often forgotten is that universities are rarely anything less than a bureaucracies. Therefore you, as I and my classmates, might get a "Unix Administrator" who is unaware of the web interface to their server's email service (squirrelmail) and instead give a crash course on Pine to a group mostly Windows raised neophytes without giving them the basic, yet amazingly helpful, courtesy tip on using "tab completion" on BASH's command line! (That last point would make a good interview question or grounds for dismisal, in my book, for an 'Admin' titled position.)

I guess the ultimate irony is that the skills universities 'intend' to teach are actually the ones that, hopefully for me, helped the very people who implimented such things as 'tab completion' and using telnet to port 25 to send email. So, all is not lost for attending such institutions. It's the bureaucracy on the backend that slows 'em down.

You don't need it

[Anonymous Coward]

To preface, I am a CCIE, so I know a little about these things.

You are correct that the layer 3 switches offer a different perspective on how networks can be drawn today.

It used to be that big switches would sit in the computer room, with clunky slow routers sitting on top of them, acting as Routers-On-a-Stick, with some sort of trunk connecting them to the core switch.

I think the easiest design that will give you the most benefit would be to just trunk a link to whatever closet, and use a cheap layer 3 switch (perhaps Extreme or a similar variety) in the data closet, for end user hookups.

Have gateways set up on the switch, use a default route pointing back to the core, and divide up the ports to whatever VLANs you ported over--I prefer to have a management VLAN and a few ports set up for that, maybe an extra one for SPAN/Mirroring if necessary.

The end user traffic would likely never be routed until it reached the core, unless you'd like to trunk the core traffic over to the closet. Then the access layer switch could route to the core subnet if necessary and save the core switch(es) the effort of doing such routing. If you have a small business, it wouldn't make much difference either way--many chassis based layer 3 switches do 64Gb per second routing with their fabric, and it is unlikely anyone would notice a delay from the routing in the closet or in the core.

Again, it depends on how you want it to look and how you want trouble shooting to be. But you are absolutely correct--a distribution layer is no longer necessary. I would consider it, really, to be the Core/Distribution and then Access Layers, or the Core and Distribution/Access Layers.

You still are using the concept of the distribution layer, but it has merged with another layer, depending on your design.

Oh, and don't forget about spanning tree :) You still need that.

No clear choice.

[redelm]

Yes, with a fully switched network the major driver for a distribution layer (traffic congestion & collision domain size) has gone away. However, other reasons like expandibility, damage isolation and traffic isolation still remain. For a price. Pick your poison.

It's usually not needed in a network of that size

[jsailor]

You didn't state the size of your network other than to say small-to-mid size, but most small to mid-size networks can run fine without a distribution layer. You're also correct that it is an artifact of 1996-1999 switching technology limitations and large vendor propaganda that sells ports. You need to be careful about:

1. how you link your merged core/distribution switches: if your access uplinks are layer 2, you then have to span VLAN across core/distribution switches. If you plan on having your access switches perform layer 3 routing look into the costs your vendor may charge for that functionality. Some charge as much as $10,000 for the license.

2. Be careful you grow your VLANs and spanning trees. Definitely use per-VLAN spanning trees. Also seriously consider rapid spanning tree or vendor specific hacks (uplinkfast, backbone fast, etc.)

3. Use server access switches. Seriously consider redundant control processors in these.

4. Seriously consider redundant control and switch fabrics for the the core/distribution switches. In the three-layer model, this was not as much of a requirement. Also seriously consider the failover time associated with the redundancy you bought. Times ranges from stateful/1 second failover to 90 second reboots to the redundant processor.

5. If you do layer 3 routing and the access layer be very careful with your routing protocol design and avoid black-holes. Run through all failure scenarios and make sure you're covered.

6. Consider where you want to perform filtering for security, QoS, etc. By eliminating the distribution layer, you're forcing this the access layer. (arguably it belongs there, but think about how many places you'll be configuring and monitoring)

7. Most importantly, consider the costs after you've considered the above. You may find out that you're not saving much. Most of my clients do save, but some find out that after they've added redundancy and possibly upgraded switch models they are close the same cost.

8. Consider your support group. What are they used to? Can they adapt? Can they handle the added functionality that's been pushed to the core or access switches.

Again, I have clients with 1500 nodes running fine with a combined core/distribution. I also have a clients with 200 nodes that mandated three layers. IMHO the break point is somewhere around 1000-1500. As always every place is different, be careful, plan and you'll be fine.

Re:Layer 3 Switch?

[Lars T.]

Layer 2 and Layer 3 Switch Evolution - Volume 1, Issue 2, September 1998

Layer 3 switching is a relatively new term, which has been ?extended? by a numerous vendors to describe their products. For example, one school uses this term to describe fast IP routing via hardware, while another school uses it to describe Multi Protocol Over ATM (MPOA). For the purpose of this discussion, Layer 3 switches are superfast rout-ers that do Layer 3 forwarding in hardware. In this article, we will mainly discuss Layer 3 switching in the context of fast IP routing, with a brief discussion of the other areas of application.

Depends on how you define "distribution layer"

[dtfinch]

We just have a stack of 24 port gigabit switches. 4 ports on each switch is set up as a trunk to connect them together, effectively turning them into one giant, fast, very cheap gigabit switch. Looking at the Cisco diagram, this might be considered our distribution layer.

We normally have one port on the switches for each system, with the exception that in some locations we have smaller switches to allow them to share a line, so that we don't have to rewire the building. We also use some smaller switches as repeaters to parts of the building too far away to connect directly to the central switches. Those small switches outside of the server room, along with all our servers and systems, might be considered our access layer.

Then we have a tiny linksys router, intended for home use, connecting the entire building to the internet. I know, it sounds scary, and unprofessional, but it seems plenty capable of filling the bandwidth of a T1 and tracking as many simultaneous tcp connections as we use. We'll consider replacing it at the first sign of trouble. I guess this is our core layer.

I suppose that whatever you use at the top level to connect your systems to create a single network can be called your distribution layer. The switches may get cheaper over the years, but it's the same thing. If you just have a chain of 8 port switches running around the building, then your distribution layer is a bunch of 8 port switches.

However you design your distribution and access layers, your main goals should probably be to minimize line problems (mostly due to distance) and avoid bottlenecks. You seem concerned about price, so if you decide to use 100mbit switches to keep the price down, I recommend that get the kind that have gigabit uplinks and plug them into a gigabit switch, and plug your servers into the gigabit switch as well. Otherwise, your effective bandwidth will be 100mbit total rather than 100mbit per user.

I've designed just such a thing

[Jjeff1]

For a school, they have 5 buildings on a campus. Within each building was 1 to 5 wiring closets. A total of 900 ports or so. Their requirements were simple, they wanted speed, multicast support, and some access control between VLANs. IP only.

I'm a consultant and work with hardware from just about anyone, so it makes no difference who they bought. We were hired to design a network for this school using various vendors equipment. Primarily to compare costs.

In the end, they went with a solution from HP. A single 5300xl in each building connected to a bunch of 48 port edge switches in each closet. Their server room has a 5300xl with a couple Gig blades and a second 48 port Gigbit switch.

What really decided the issue was cost. They didn't need support for all the assorted protocols and features you get with cisco, and they didn't want to pay for it. With cisco, you had a 6500 series monster in the datacenter, then a distribution switch in each building, and a bunch of edge switches.

The HP solution was well under a third of the cost of the cisco solution, also free lifetime next day replacement warranty on hardware. For the money they saved, they can afford to have a shelf full of spares, including a spare core switch.

Personally, instead of looking at what model you want to use, look at what you need your network to do, then talk to your prefered vendors and see who can do it at the best price point.

Re:Layer 3 Switch?

[Zapman]

You are sort of right. A 'router' is capable of working with multiple subnets, but traditionally, only has a few interfaces. A 'switch' (or hub) is traditionally only able to deal with 1 subnet, but has lots of 'interfaces' (ports).

Switches have grown up, since the advent of VLAN's, they've been able to 'route' between vlans, and have expanded to OSPF, and other high end routing protocols, while keeping the port count. These higher end switches don't usually have WAN ports (T1, T3 type), or the ability to do super high end routing (OC-16, OC-192, Terabit), which is why Cisco and Juniper still sell routers. The two terms have become quite unclear over the past decade.

Your budget is too big

[fred fleenblat]

Count your blessings. You'd be amazed at how many small to medium sized companies (2000+ employees) have one Cisco router in a rack somewhere and use consumer grade linksys or d-link 10/100 switches everywhere else.

For sending email and word docs around, you really don't need the whole Cisco hierarchy. On the other hand, If you're sending uncompressed production video around, it's not enough.

Distribution layer exists only in the Ciscoland

[wikinerd]

The term "distribution layer" is defined by Cisco, which is just a corporation. There is no standard where you will encounter this term.

The most well-known networking standards are the OSI model and the TCP/IP model. Neither of these standard models include the term "distribution layer", which means nothing by itself: Is it about physical-electrical distribution, data distribution or information distribution?

I personally dislike "standards" or tech-speak set by corporations and I believe international bodies and computer scientists should be preferred when it comes to standards and technical jargon: Imagine two computer scientists, one using Cisco-speak and the other knowing only Microsoft-speak, how are they going to communicate? It's impossible! - unless they both adopt a common language like these proposed in the OSI or TCP/IP model.

I personally can communicate network concepts using the OSI model, and I am completely unaware of Cisco-speak. In an attempt to answer your question, I will assume that by "distribution" Cisco means "routing", which translates to "Internet layer" in TCP/IP-speak and is related to the Internet Protocol, while in ISO-speak it translates to "Network layer". If my understanding is correct, then the answer is that no matter how small your network is, you will want to use routing, for example for connecting your small network to the Internet. Even if the routing functionality is included in a device of another layer, or even when it is implemented in software, it will always be there, no matter whether the users or even the administrator can see it, especially if you are going to use the TCP/IP protocol suite.

Re:It's usually not needed in a network of that si

[CounterZer0]

I think the key reason to have that middle layer is for scalability these days.
Buying 15 smaller switches, and collapsing/trunking 200 switches onto 15/30 Gig/10Gig uplinks to a core means my core only needs 15/30 ports, instead of 200+. Sure, you don't *need* it, if you can afford the port density of that size on your core, but any decent sized network is going to be pressed for that kind of cash :)
It's significantly cheaper/easier to provided redundancy for 30 GigE ports than it is to provide redundancy for 150 GigE ports (both in cost and wiring complexity...).
But, if you've only got 15 switches, I'd just forgo the distribution layer, as it'd be cheaper and easier for you to manage a single core (or maybe 2) with a single 15 port GigE blade or something than to setup proper distribution switch layers.

But, as you grow, definitely make sure you investigate it - most of my sites have 3-4 distribution switches, serving 8-10 access switches each, but I've got one site where the previous designer decided a distrbution layer was unneccesary. He left me with 83!!! GigE fiber ports terminated on 2 core chassis ('cause the vendor didn't sell anything with 83 GigE ports in one chassis...that should have been a warning sign). So, when I want to upgrade that sites equipment, I'm kind of up a creek with no paddle, as it's INCREDIBLY disruptive and hard to move all that fiber, etc.

So, like the parent said, 'it depends', and make sure you are planning for realistic growth over 4-5 year period (at least)...

Absurd!

[wonkavader]

Nothing could be more useless than an MSCE.

Re:Your network is too small.

[Dr.Dubious DDQ]

And when the crappy consumer grade stuff fails they can swap in a new one for cheaper than two weeks maintenance on a cisco box.

And that, in a nutshell(tm), is what I absolutely hate about the "high end" stuff. The fact that the up-front cost to gain ownership of the physical device is one thing, but the hefty recurring fees to get ANY kind of support (including, as far as I can tell, bug-fixes, security updates, and so forth) get insane very quickly. Especially when you're presumably willing to pay the premium "ownership" price because the device should then not NEED much of anything in the way of "maintenance" to keep it running. I've come to think of this as just another "protection" racket - "Nice network you've got, and such an expensive router. it sure would be a shame if someone happened to find a security flaw and you didn't have access to updates, wouldn't it?" Taking that into account, it may often cost LESS to just replace the "consumer grade" stuff as it fails that it costs to keep paying "maintenance" fees on the expensive stuff.

I've gotten quite irritated with Cisco on this front - I picked up a Cisco 768 DSL router to replace the "Actiontec" piece of junk that the phone company was renting out. I went to Cisco's website to check for updated firmware and so forth, and got told "you have to register to see this". So, I went and gave out all the precious marketing information (name, address, phone number, blood type, shoe size, etc. etc. etc.), finally got to the end of the "registration" process, and got "Ha, ha, sucker, you STILL can't see this stuff because you're not a 'paid support' user or a 'Cisco partner'...". Thanks, Cisco, thanks a lot.

(On the upside, the router HAS been very reliable so far...and doesn't decide to just stop routing packets for no good reason until rebooted like the Actiontec modem did...)

Re:Your network is too small.

[dTb]

Please read a Cisco vulnerability announcement. You will see toward the base the procedure to get a free update that fixes the vulnerability if your equipment is not covered by smartnet. I quote:
Customers who purchase direct from Cisco but who do not hold a Cisco service contract and customers who purchase through third-party vendors but are unsuccessful at obtaining fixed software through their point of sale should get their upgrades by contacting the Cisco Technical Assistance Center (TAC). TAC contacts are as follows. * +1 800 553 2447 (toll free from within North America) * +1 408 526 7209 (toll call from anywhere in the world) * e-mail: tac@cisco.com Please have your product serial number available and give the URL of this notice as evidence of your entitlement to a free upgrade. Free upgrades for non-contract customers must be requested through the TAC.