Slashdot Mirror
Ask Slashdot: 802.11n Bake-Off Test Plans?
First time accepted submitter Richard_13 writes "I am seeking a bake-off test plan for an enterprise size deployment of 802.11n wireless. We are about to go to tender for a large scale deployment of 802.11n controllers and APs — and I need a bake-off (benchmarking) test plan that is focused on testing the *maximum number* of clients that an AP can handle before it falls over, in addition to the throughput for each client. We intend to test the latest products from the major vendors, Aruba, Cisco, HP, Xirrus, Ruckus, etc.; not consumer products like Linksys, D-Link or Netgear. Any bake-off test plans or useful links to multi-vendor wireless focused web sites would be greatly appreciated."
one more time~
The Kruger Dunning explains most post on
Just say "benchmarking" and you won't need to explain that "bake-off" means "benchmarking". Wait, what was the question?
Call a meeting of the competitors engineers. It's important you get them all in the room at one time with not too much advance warning of the topic.
Tell them what you think you want. Ask them as a group what you're missing. Then make them as a group come up with an eval plan and cook 'em off according to the plan they come up with.
If you need an independent judge, go to one of the labs that does independent third-party assurance and contract them to provide oversight.
Disclaimer: I've worked for one of those labs for the past 15 years.
Stand back and watch the fun......
Red
I run a couple of Cisco WLCs with about 15 LWAP APs spread across them and in my experience, each AP doesn't really handle any more clients than a good consumer grade access point. Once you get about 18-20 clients on an AP, performance drops to suckage. I tried to arrange the physical locations of the APs to try to keep about a 12:1 ratio of laptops to APs, but we're got one classroom with 24 seats and two identical APs at opposite sides of the room, and for some odd reason they never seem to automatically load balance all that well. I'll often see 18 clients lock onto one AP and only 6 on the other in the classroom and the user complaints come pouring in. I've come to the main conclusion that trying to use WiFi as a primary network connection is basically a big farce.
try Meru Networks we use those for 1400 users... good luck
We have a box that can emulate up to 128 stations, including wpa, wpa2, etc. It can do
DHCP or static IPs and generate Ethernet, udp, tcp, http, and other higher level protocols,
including IPv6. Multiple systems can be clustered together for additional throughput and
radios. Each system can run on only one channel at once, but can talk to multiple APs
on that channel.
One of our systems can saturate any of the consumer grade APs we have, and some folks have
used it to stress very big systems (conference centers, etc).
Runs on Linux of course!
http://www.candelatech.com/ct520-128_product.php
Thanks,
Ben Greear
Unless you're going to deploy like 100 APs or more i an skeptical that the vendors will work with you for such an effort.
Actually doing this correctly is going to be hard and expensive. Anyway, i'd read up on smalnetbuilder's methods and just run, say 10 or 20 concurrent client machines o a 3 or 4 AP set-up. make some of those clients mobile and walk around the space to see that hand-offs happen ok.
graph it all and look for major priods of drop-out etc. Again, though, unless you're doing a massive deployment or this is mission critical more than normal office lan this is not likely to be a cost effective exercise. I've previously had a good experience with Cisco APs
They are all going to do the same - which will exceed by a factor of 5 or 10 how many you will get in the normal course of a day.
The management tools are what you should be looking at.
Are you talking about 50 APs or 500 or 2000 ?
if someone tells me what the FUCK bake-off is
I will gladly accept all cookies created in the process.
If you're running a bake-off with access points, you're probably running just a bit too much power into the radios.
On-topic, I do like my redundant-controller, centrally-managed 160+ AP Aruba system.
Maybe you could hold a bake sale and offer free wifi for all your customers. Just give them places to sit.
Im not sure what you mean by no consumer stuff.but netgear has started offering small biz gear.including wifi setups supporting up to 150 APs. So not sure how big you thing big is, but they are one of the few mid size deployment shops. While my prior opinion of netgear was low they seem to be trying to break into the enterprise markets. Of course if you need mor APs than that, they are still too small.
Those who can, do.
The (not so big) secret is that most WiFi AP rolls over with 8 or so clients. Only a few manufacturers themselves test their products beyond that, and those work all the way to over 100.
The company selling the test equipment you need is called http://veriwave.com./ You can buy the equipment from them and test all the vendors, or even better, just ask them.
They do of course know, since that is how they test their own test equipment. Problem is that they can/will not tell you because then 1. you would not need to buy their product, and 2. AP mfg would fix their products, and Veriwave would not have a market for their products.
Maybe just do some social hacking to get it out of them.
don't cut it off www.mgmbill.org
Include ubiquiti as a hardware vendor.
Seriously good stuff, and way cheap.
If your needs aren't one of the top 5% situations, you can use their gear and save 50-75% over other vendors.
Look at Unifi - it's still not quite there yet on the software side - but what is there is really very good, even for enterprise use.
The only other stuff I'd consider would be something like Ruckus for really high end - totally deterministic, high client usage and density.
Good luck.
-Greg
They may not be selling the cheap one anymore, which would be good, because it was vastly inadequate for any real N-style usage.
Guns don't kill people, "with glowing hearts" kills people.
Don't forget about Juniper's new wireless solutions, from their Trapeze acquisition.
I've heard a lot of good things about Aruba and Xirrus.
Having actually done Cisco wireless support and new deployments, I would highly recommend against Cisco. They call it a "Cisco caveat" for a reason. Sure that feature works... you know... under the right conditions which will never be met.
I looked at equipment recently for wireless using the 'n' protocl - but noticed no mention of IPv\6.
No point of gertting new communications equipment, if it cannot be suicessfully usec with IPv6!
So make IPv6 part of the requirments.
Here are the results:
Vendors were tested with 30 then 60 wireless clients and 1 then 2 access points. So, 1 AP with 30 clients, 1 AP with 60 clients, 2 APs with 30 clients and 2 APs with 60 clients.
1: Cisco - Somewhat surprising. Great client density/bandwidth. Good load balancing between APs. Good management interface.
2: Trapeze (now Juniper) - Great client density/bandwidth (just a little slower (read less bandwidth to client, and just slightly less) than Cisco). Good load balancing between APs. Buy the extra Ringmaster management software.
3: Aruba - Significantly slower than Cisco and Trapeze. Good load balancing between APs. Good management interface.
4: Meru - Significantly slower than Cisco, Trapeze and Aruba. They did not have a network engineer available for the test to be present and we were unable to schedule another test before our purchase window closed.
We were going to test Xirrus but the rep we were working with left the company I believe in the middle of scheduling. We looked at Ruckus but were unable to schedule them.
These tests were performed in the spring of 2010. So products may have changed somewhat. You should be able to get demo hardware from any decent rep. We ultimately went with Trapeze after we put everything out to bid. Before that I was sure we were going to get Cisco gear but Cisco came in at twice the cost as Trapeze. We are deploying 128 APs without any issues. Client roaming and bandwidth are great (our primary requirements). All in all no complaints. Certainly liked the price point. Hope that helps!
In my somewhat limited(but rather painful) experience with attempting to use wifi as a serious connection, one of the issues that cropped up a lot was less with throughput, or with number of clients; but with client software behavior in the face of a glitch.
Dicking around at home and the wifi cuts out for a second? Reload the webpage and quit your whining.
Running your basic "enterprise" client configuration(documents directory is actually on a fileserver, authentication through AD, etc, etc.) and the wifi cuts out? Be prepared for frustratingly erratic appearances of apparently disappearing documents, authentication fails, not automatically reconnecting to the fileserver, Finder just twiddling its thumbs and thinking about infinity until that server either times out or comes back, etc, etc.
Even before any APs show up, you can start identifying the likely areas of sheer pain by using netem, switch jiggery-pokery, or just a $20 consumer AP and flicking your laptop's RF power switch: If your environment has client applications that don't play nicely if the network goes all to suck for a second or two from time to time, wifi deployment is going to be Fun.
Honestly, for most applications where wifi isn't a totally terrible idea(ie. heavily throughput dependent stuff), that would be the big focus of my testing(along with how useful the management tools and interfaces are). High throughput is far less valuable than stable connections.
Stop saying 'bake-off'. It makes you sound like a tool.
HTH
We had Cisco--their performance was less than desirable. Switched to Meru and all of our problems were solved. Went so well our entire university rolled them out. 10,000+ students, 1,000+ faculty/staff. Not sure about 100 connections per access point, but it sure is up there.
Find out what Starbucks uses or any other free highly saturated provider uses for their kit.
The only thing that matters is the bean counters and any ELA you have with existing vendors. Cisco might be good, or it might be crap, but if you have a pimp contract with them and good support, they're getting the contract. Live with it.
I want to delete my account but Slashdot doesn't allow it.
but i use ubiquiti products and they run nice... arounf 75 concurrent clients running inventory out of 3 AP on a big warehouse. most of ythe clients just upload a small txt file containing the items they were counting.
Also, a fee tech support guys browsing slashdot/facebook/espn while we wait for the inventory crew to finish..
This is possibly the worst place to ask this kind of question. The fact that your asking it here, makes me wonder how you got your job.
here you go:
OFFICIAL 45th PILLSBURY BAKE-OFF® CONTEST RULES
WHAT CAN I WIN?
100 finalists will win trips to Orlando, FL, March 25-27, 2012, to compete for cash and merchandise prizes totaling $1,054,000. Trip includes round-trip coach air transportation for finalist from major airport nearest finalist's home, hotel accommodations consisting of one double occupancy room for two nights at the Peabody Orlando Hotel, daily event-sponsored meals, a GE countertop microwave oven (estimated retail value of $200), and $125. Approximate retail value of each finalist package is $1,615.
RECIPE CATEGORIES
One overall Category Winner will be chosen in each of the following four Recipe Categories:
Breakfast & Brunches
Casual brunch or weekend family breakfast ideas. For example, sweet rolls, pull- aparts, pastries, breakfast breads, egg dishes or breakfast sandwiches.
Entertaining Appetizers
Appetizers and snacks to serve at casual gatherings with family and friends or for holiday entertaining. For example, bite size appetizers, tartlets, pinwheels, puffs, bruschetta or focaccia.
Dinner Made Easy
Easy to prepare main dishes that will delight your family. For example, pizzas, calzones, sandwiches, foldovers, casseroles, chilies, soups or savory pies.
Sweet Treats
Quick and easy treats for anytime celebrations. For example, cookies, pies, tarts, brownies or bars.
PRIZE STRUCTURE
One (1) Grand Prize Recipe Winner receives $1,000,000 and winner chooses $10,000 in GE kitchen appliances. Combined retail value of Grand Prize is approximately $1,010,000.
Three (3) Recipe Category Winners each receive $5,000 and winner chooses $3,000 in GE kitchen appliances. Combined retail value of each Category Prize is approximately $8,000.
Other Awards:
One (1) GE Imagination At Work Award (retail value is approximately $5,000)
One (1) Jif® Peanut Butter Award ($5,000)
One (1) Crisco is Cooking Award ($5,000)
One (1) Eagle Brand® Signature Recipe Award ($5,000)
Grand Prize Winner
At the Contest Finals, the judging panel will award one recipe winner for each Recipe Category. One of these overall Recipe Category Winners will be judged the $1 Million Grand Prize Winner. The $1 Million Grand Prize will be awarded as an annuity payable in installments of $50,000 each year for 20 consecutive years, no interest, with the first payment payable on or about May 1, 2012. Grand Prize Winner will also choose $10,000 in GE kitchen appliances. Combined retail value of Grand Prize is approximately $1,010,000.
Recipe Category Winners
The three remaining Recipe Category Winners will each receive $5,000 and will also choose $3,000 in GE kitchen appliances. Combined retail value of each Category Prize is approximately $8,000.
GE Imagination At Work Award
The judging panel will award the GE Imagination At Work Award to the finalist whose recipe the panel deems the most innovative. The GE Imagination At Work Award will be awarded to a finalist who has not won any other Prize or Award. The GE Imagination At Work Award Winner will choose $5,000 in GE kitchen appliances. Retail value of the GE Imagination At Work Award is approximately $5,000.
Jif® Peanut Butter Award
The judging panel will award the Jif® Peanut Butter Award to a finalist who is recognized as having the best recipe using at least ¼ cup of Jif® Peanut Butter. The Jif® Peanut Butter Award will be awarded to a finalist who has not won any other Prize or Award. The Jif® Peanut Butter Award Winner will receive $5,000. In the event that no qualifying recipes are selected as finalists, the prize will not be awarded.
Crisco is Cooking Award
The judging panel will award the Crisco is Cooking Award to a finalist who is recognized as having the best recipe using at least one (1) tablespoon of Crisco® Cooking Oil. The Crisco is Cooking Award will be awarded to a finalist who has not won any other Pr
Don't forget to put a damp towel at the bottom of the door. You don't want any RA's harshing your buzz. Then, no one leaves until the whole bag is gone...
All the major vendors should be aware of what is going on at www.bufferbloat.net and have something in place to ensure that their products will reflect new updates soonest when things get fixed. This is an ongoing problem that crept up on the internet tech community and there is work in progress to deal with it but it will take time.
See (for example) Bufferbloat - Dark Buffers in the Internet, 1/20/2011
Been there, done that, paid for the T-shirt
and didn't get it
Their APs are enterprise class and follow a new 'distributed' approach wherein you get better control over traffic as well as a number of features without the need for a local controller. Its the 'next generation' after the generation of thin-APs managed by controllers (Aruba, Trapeze models).
Also, when evaluating, max-clients should not be the only criteria, look for things like support for auto-rf (channel and power), support for voice (esp certifications from vendors like Spectralink, Vocera) , troubleshooting capabilities (packet capture, spectrum analysis), detailed statistics, rf-heat-maps, trending/reporting of longer term data, support for guest-access wlans.
How do you intend to do the benchmarking? In addition to just throughput tools (IXChariot, iperf) also do things like number of parallel voice calls, number of video streams in parallel etc. When testing throughput run both UDP and TCP. UDP will give you the best raw numbers, while TCP will give you more realistic data (& is less forgiving of the lossy nature of wireless so will typically drop back faster as packets are lost)
BTW when you have connectivity issues (which you almost surely will at some point) the vendor who can provide you with the best 'view' into the network is the one you should go for.
They probably belong in the list. It's at least worth checking them out. They'll give you some eval units at no charge, too.
Plenty of companies out there make tools for testing this sort of thing. Spirent, Ixxia, and Agilent, to name a few all have layer 4-7 traffic generation appliance type products for stress testing.
http://www.spirent.com/Devices-and-Equipment/Base_station_testing.aspx
I can say in our limited tests and real life use we continue to be impressed with Trapeze (now owned by Juniper).
test will rarely fit with real life, in our case Trapeze sent a sales engineer out and did a site survey asked us questions and gave us great advice... on how to move forward.
googled a little:
http://www.mobileindustryreview.com/2011/05/enterprise-wifi-best-practice-series-devin-akin-of-aerohive.html
http://www.tomshardware.com/reviews/wi-fi-performance,2985.html
http://www.ruckuswireless.com/press/releases/20110718-independent-test-reveals-ruckus-outperforms-others
Forget about 802.11n from "Aruba, Cisco, HP, Xirrus, Ruckus" and go with Mikrotik modification of 802.11n with timeslots. Job done cheap and much more effective! BTW I have read all answers so far and they look to me pretty childish and not much helpful. Strange for Slashdot :-/
I don't know if this thread is about baking or getting baked, but either way, i gots to get me something to eat.
Netgear makes enterprise grade equipment -- we have Netgear equipment all through our manufacturing facility and IT center. having said that... We used Xirrus wifi hwardware and it works quite well.
What does enterprise size mean?
What class of construction is the
building? Straw, sticks, bricks... this wolf wants to know
Are regions RF isolated from each other
do you have multiple floors and multiple
buildings? Can RF pass between these
odds and ends.
First you need to wire the building and also decide if
your WiFi boxes will get power from the ethernet wire
links or from the wall.
Do you need secure access for all or is this install
wide open and plan to let VPN do the security.
i.e. guests will always want WiFi service.
Have you done any site research. If a neighbor already
has WiFi deployed and all the channels occupied you
may be the last fool in the pool. There are some
Android applications that I would use to see what is already
transmitting.
Windows and floors do you want to secure the inside
from outside listeners? And do you want to secure the
inside from external access.
Since all WiFi is tested to and operates to public standards
there is going to be little difference from vendor to vendor
at first glance. Placement and wired infrastructure will make
as big a difference as anything.
You are going to need wired links lots of them. Plan on
a robust wired infrastructure to start. The cell phone companies
often have less trouble on their last mile... than you might expect.
Will Cell phones have the ability to connect their WiFi links
to your network? If the company provides phones the answer
will be yes... Cell phones + laptops + iTouch three times the
load you might expect. Will you have to put Femto cells in
for executives?
You may have to screen areas with hardware cloth or some
RF limiting wall covering to keep areas from interfering with
each other....
Net nannie... do you have legal issue to audit and manage
both incoming and outgoing? Privacy issues that make shared
pass words a tangle.
Central management? Can one person walk around and
check them all? Backup...
Truth is stranger than fiction, but it is because Fiction is obliged to stick to possibilities; Truth isn't. Mark Twain.
"We intend to test the latest products from the major vendors, Aruba, Cisco, HP, Xirrus, Ruckus, etc.; not consumer products like Linksys, D-Link or Netgear."
Cisco - > Linksys
You can call in all the experts and engineers to figure this out but never underestimate the importance of common sense and understanding of basics concepts involved. A couple of articles on Tom's explain it well. One story has some testing and benchmarking. May not be exactly what you need but may go a long way in ensuring good wi-fi. http://www.tomshardware.com/reviews/wi-fi-performance,2985.html http://www.tomshardware.com/picturestory/571-wi-fi-beamforming-networking.html
We run a thousand or so Aruba 125s at school here, covering all 600 or so acres of campus. Those are probably overkill for you (at about $750 a pop), but AFAIK even the lowest-end ones have the same essential features.
Basically, the network architecture puts the whole wireless network on a separate segment, all the way back to the aggregation points. They're gigabit wired into the building routers, but placed on a separate VLAN all the way back to one of the three aggregation points. Each AP is assigned to a controller, and will fail over to a second one if needed. The controllers pass the traffic to the rest of the network.
The controller architecture means you can do some pretty interesting things. Particularly, it means new APs are trivial to install - stick them into the controller's DB and plug it in to an Ethernet cable (it's PoE); it'll go and find the controller and pull down the config and any upgrades to the software. It also allows IP roaming between APs, even if they're in different netblocks. I can walk from one end of campus to the other (7 city blocks) while keeping the same IP and getting all my traffic, through about 150 different APs - much like a cell network. You can also do spectrum analysis through their management console - I once saw them find a broken microwave from all the interference they were seeing across the 10 APs in range a la Dark Knight.
The APs we have will band-steer clients over to the 5GHz spectrum if possible, which can support a huge number of clients, but you need the density for it to make a big difference. If you do, though, you can easily get 30 people per AP, with a few doing massive downloads/uploads and no hiccups. They don't recommend more than that, and in any case it's difficult to fit people densely enough that you wouldn't need a new one for signal purposes.
No, I don't work for them. I don't even work for the Networking department. I just really like the toys - though I suspect I might feel differently if I had to make the purchase! Quality isn't cheap...
I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
I was going to reference the same article from Tom's - they basically have already done what OP is asking for, using hardware including Cisco, Aruba, Meraki, Ruckus, Apple, and HP. The second and third pages "Hardware And Methodology, Explained" might especially be of interest.
If you spoof a controller. One of the first lessons I've learned is to never use dynamically configured devices on a campus. There will always be a geek that will find a way to tell your equipment what the best route for traffic is.
I was promised a flying car. Where is my flying car?
The result was Cisco but this was 3 years ago. The Aruba gear is poorly made and it took about 70% more APs for the same coverage. We used 900 Cisco APs as opposed to over 1500 Aruba. That being said we also like the Ruckus Gear which also feels flimsy compared to the Cisco 1042 APs but the beamflex technology is second to none especially if your trying to carry a voice or video signal.
Check out the Ubiquiti UniFI distributed wireless system, which uses a server-based controller. www.ubnt.com
Ceiling-mounted APs, MIMO-N 5Ghz, up to 4 SSIDs on separate VLANs, good feature set for corporate environments.
(satisfied Wireless ISP owner)
Be sure to add them to the list. Their virtual cell technology is pretty slick, and works well for busy environments. Just add more access points to a busy area or for an event, and the controller will take care of balancing the clients between the radios. No, I don't work for them, but I am a happy customer. We have over 250 access points installed across our campus.
Never ask for directions from a two-headed tourist! -Big Bird
For those fixated on the term bake-off, you obviously have never worked for a networking vendor or a major IT shop, or you would understand what the term means. And some of you demonstrated a level of intelligence that has me picturing you entering "snow white and the seven dwarfs" when prompted for an eight character password.
Actually, evaluation of networking equipment is easy. Just take the sale brochures and divide their performance numbers by ten.
You must be awesome at parties.
If you go.
(Unlikely).
See: http://www.tomshardware.com/reviews/wi-fi-performance,2985.html Great comparison.
Are you going to tell your boss that you're asking people on the Internet to do your job for you?
Don't forget to check out Aerohive, another decent option.
Call Veriwave, Beaverton Oregon
While I understand that in the world of Enterprise IT it is standard to use a package from one of the large vendors complete with controller, I very much disagree with philosophy of setting things up that way. Please note that my network is not to the scale of yours, only about 75 APs on a small campus, but I've had great luck using small-business/high-end consumer grade equipment, clever setups on "Fat" APs, and some powerful controller software. For example, my current design just uses off-the shelf EnGenius stuff because it was cheap and at least did 2.4N piped through a communications-isloated vLAN and uses pfSense as a gateway so I can do my shaping and captive portal. I mean at $90 per AP and zero licensing costs, I'm willing to double-up on density to offset for slightly lower quality parts and we still make out like bandits in our budget. Throw in some basic scripts to check on AP status and you really have 95% of what Cisco or Aruba does, but at 5% of the cost. I understand that beyond a certain scale this just isn't feasible, probably around the 150+ mark. But "large-scale" means different things to different people and the author didn't provide a rough AP count. Besides, I had Cisco drop off some gear for me to test out their 1142s and a controller a year or so back. I was less than impressed compared to what my system was already doing. Only hitch being that I had to manually configure each AP, but its all about balancing practial usage against budgets in my world. And my Aruba rep was creepy. Final note - please bear in mind that while we have opted for this solution for wireless, the actual infrastructure is top notch. No copper used in between buildings, 1Gbps minimum fiber and 10G fiber from our Core router the server room. I think a lot of the time a high-cost wireless solution is designed as such to offset the downsides of a poorly designed or outdated network. Spend money on your foundation first, you know?
herve leger bandage dress herve leger skirts herve leger dress on sale herve leger dress wholesale
http://www.ubnt.com/unifi
Cost/Performance ratio is unbeatable as these guys make all their own gear / software.
Been there, done that.
My advice - get the Aruba solution. It's far superior to Cisco from a management and intrusion handling standpoint, and the AP density is quite high and assigns it's channels automatically as long as you're using the centralized management facilities. The AirWave product is also a must-buy - allows for heatmapping, client mapping and tons of other reporting options.
The VPN tunneling facilities are tremendously powerful - my company can open a retail outlet by simply writing down an AP's Mac address and having the folks remotely boot it up and direct it to one URL - and whether they're on wired or cellular wireless, it works. Add in the RAP-2's they have, and you have an enterprise level VPN setup that can deploy to mission critical personnel's houses (and exec's) and provide seamless network access 24/7.
Sorry, but Cisco's just not there, IMHO.
Also, though this isn't as great a plug as it used to be, Microsoft uses Aruba - and they generally don't mess around with this stuff without doing their due diligence as well. I wonder what Google uses... Apple uses a combo of their own stuff and Cisco. The new Amigopod stuff is pretty powerful too - we've deployed full port auth throughout the network, plus have secured all the wifi access - and guests have to be sponsored for access and receive a temporary password via SMS.
You do the math.
Cheers.