Slashdot Mirror
10 Tips For Boosting Network Performance
snydeq writes "InfoWorld's Paul Venezia and Matt Prigge provide hands-on insights for increasing the efficiency of your organization's network. From losing the leased lines, to building a monster IT test lab on the cheap, to knowing how best to accelerate backups, each tip targets a typical, often overlooked IT bottleneck."
Unplug wires in network closet.
I learned from the BOFH that the fastest backups are written to /dev/null.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
Plug wires in again in a more colorful way.
Switching from Ts to Cable Internet service at work would get you fired within a week, since within that amount of time you will see downtime.
Stop your IT Department from visitting Slashdot
Seriously, does anyone backup to tape anymore?
I Am My Own Worst Enemy
Don't trust articles that have:
Created 2010-06-01 03:00AM
before the "well thought out" advice.
We should start a new Slashdot and return control to the geeks. It actually wouldn't be that hard to get some users to
#1 tip
Pull the ethernet cord that runs to your bastard roommate's computer. You know, the one who is always downloading porn when you are trying to frag noobs. That guy is a prick.
Like 'know your apps' means anything in the corporate world, especially when apps are custom built, what are you going to do, replace a custom built app with something else? If it was easy like that then why was it custom built in the first place? Sure, some custom apps can be replaced with out of box stuff, but seriously speaking, most cannot, and then your administrator is in the hands of the geniuses in the management, business, marketing, and software development departments :)
You can't handle the truth.
Just give Eric Schmidt a call, tell him you have nothing to hide from his company or the government and they will replace all your machines with shiny new Google Chrome OS based "Net tops", put all your data on their servers, give you a brand new direct fibre optic connection to their nearest office and all they want in return is the ability to meticulously sift through your data in order to find the best way to bombard you with text-based ads.
Everything is more shiny with Google.
What reason is there to run T1/T3 anymore? I know, by definition, the regulation over T1/T3 guarantees reliability. I have dumped T1's and switch to 2Base-TL (aka Metro Ethernet) and it is extremely reliable. For me, the "more reliable" argument doesn't hold much. The latency is very, very good -- often below 10ms. Even if the network goes down, I can afford some sort of backup link. I'm paying under $1,000/month for 10mbit (symmetrical). The footprint for 2Base-TL is pretty good because it is based on DSL technology. It doesn't have the reach that T1's have, but it isn't bad. The big difference is that is spreads the signal over multiple pairs of wire (in my case, 8 pairs) instead of a single pair.
If your company has T1's, shed yourself of the "regulated" links and check out 2Base-TL. You will be glad you did.
Get rid of fat clients, that will do wonders to reduce your network bandwidth needs out to the customer. Then beef up the datacenter network.
---- Booth was a patriot ----
Slow down your internet connection to a single 56k line... then people will stop trying to use it to look for porn and all the useless crap searches they do on google... You'll also save some money with the monthly bills!
once I told a coworker about emule. He downloaded and installed it. The next morning the CFO comes to me.... "Have you ever heard about emule"...the infastructure was screwed up, but instead of fix it they waited for p2p to bring the network to it's knees. The best way to test a network is to see how many simultaneous p2p connections it can handle before crapping out. Needless to say there were some consequenced for that employee.
Doesn't really help me improve my network performance at home.
It frightens me to think that there are people getting paid to take care of enterprise systems that would not already know everything in this article. Mostly, it reads like a thinly veiled ad for VMWare products.
Ask Slashdot: Where bad ideas meet poor googling skills.
The article suggests things that people worth their IT salt should have already implemented, or at least investigated. Really baseline stuff there.
However one big oversight I see a lot w.r.t. backups and local networks which toss large amounts of data around are configuring jumbo frames. This is often forgotten about when throughput is getting tight.
Ban ALL Microsoft products. The worst is WORD.
Yours In Tashkent,
Kilgore T.
run ntop off a span port or tap. You'll see the majority of your network traffic is from users idling away on things not quite work related. Separate egres traffic on port 80 and 443 with linux htb, tcng or equivalent profiling. Saves you bandwidth that exchange will immediately suck up.
boycott slashdot February 10th - 17th check out: altSlashdot.org
Write your HTML in notepad, just like the linked article :)
Seriously, I was almost shocked to see such a barebones site. Its been that long.
Run remote desktops. Bandwidth consumption to the desktop drops dramatically.
Run your heavy network I/O over the switch stacking fabric, where you've got shit loads of bandwidth. Channel bond.
Separate access ports/switches and storage network ports/switches. Use jumbo frames on the storage network, but don't route them.
Prefer shared memory first, then unix domain sockets over TCP/IP/LAN over WAN. Microsecond (or better) latency vs milliseconds or seconds.
Dedicate servers to applications, take advantage of copy on write & modern memory management.
Let your VM management hold a significant proportion of dirty pages. WTF is the point of loads of RAM if you insist on running at disk speed? But do use a logged filesystem.
Use a load management system. Grid Engine, Condor etc.
Deleted
Do the employees keep checking the value of their stock or why do you want to ban MSFT shares?
What does it cost to power your disks when you are not using them, Tapes cost zero. Disk farms for backup are a power and heat nightmare.
... and if you think it is about latency you are mildly retarded, as are the writers of this general knowledge article.
Leased lines in general have better SLAs but that isn't even much of a point anymore as they cheaper products "claim" to have similar ones. The difference here is how good is that business class dsl/fiber support at 2am? What are the odds they are actually going to be willing to send someone out to the telco closet right away if there is an issue? You buy leased lines because you need *real* support of the SLAs... not this, "well we were down for 5 hours, so how about we credit you a day off!" bullshit.
It's really scary for what passes for "good advice" these days.
--- I do not moderate.
A better article would be one that identifies HOW to "know your apps" rather than just telling you that you should.
What tools are available. How to use them. What to look for in the most common circumstances.
http://www.google.ca/search?hl=en&source=hp&ie=ISO-8859-1&q=site%3Amicrosoft.com+%22james+hamilton%22+%22internet+scale%22&btnG=Google+Search
I'm in Links right now, so figuring out which links are the best is beyond this browser, but some of those ROCK.
Interrupt Moderation = Disable
Here's a real tip, disable Interrupt Moderation on your Network Adapter Cards to achieve greater bandwidth, as much as 100%+, and lower latency (the two measures of network performance) at the expense of processor utilization due to more hardware interrupts that have to be handled.
Instructions: In Windows open up Control Panel, Network and Sharing Center, click on Change Adapter Settings, open Properties on your Local Area Connection (sometimes #2, #3, or something if you have more network cards), click on the Configure button, then the Advanced tab, select Interrupt Moderation, change the value to Disabled, while there look for any settings with the word Offload and enable them all, and then click the OK button to make the changes. This will restart your network card driver and make the settings effective.
Most network cards from popular manufacturers such as Intel, Broadcom, Realtek, etc. hold network packets in a buffer until enough time goes by before raising a hardware interrupt and telling the processor, operating system, and network driver that there are packets waiting to be serviced. By disabling Interrupt Moderation you instruct the network driver and card to raise the interrupt every single time a packet comes in, thus making your processor service the network card much faster thus decreasing latency on the packets held in the buffer and also increasing bandwidth by allowing more packets to flow through faster. This increases your processor utilization by a significant amount 10-30% but if you have a recent dual, quad, hex, octo-core processor and recent network drivers that are multi-threaded with multi-core support and have Receive Side Scaling support then the increased processor utilization is negligible to your computer and if you are running a network server then network performance should be a priority anyway.
I have personally seen and tested corporate and home LAN environments using Fast Ethernet 100 Mbit/s (~11 MByte/s) go from slow 6-7 MByte/s to 10-11 MByte/s throughput, and Gigabit 1,000 Mbit/s (~100 MByte/s) go from ~30 MByte/s to 95-98 MByte/s speeds due to these changes. No other network driver setting had as much performance impact as Interrupt Moderation.
IEEE 802.1AX (aka 802.3ad, Cisco EtherChannel)
For advanced network performance improvement look at link aggregation (channel trunking, link bonding, etc.) using the IEEE 802.1AX (aka 802.3ad, Cisco EtherChannel) protocol support in your Intel and Broadcom network adapters using their Advanced Configuration Utilities on your servers to bundle from 2-8 Ethernet network adapters into one trunk to increase your performance. Just tell your network administrators to enable those features on your ports and find out if they are able to do it if your links are going to the same switch or if they have virtual switching enabled in case your links span switches. Just think about 4 x Gigabit performance if you bundle all 4 NICs on most servers.
NetCPS
You can test your own network performance with this simple but great utility called NetCPS. Just be sure to disable Interrupt Moderation on both of the computers on your LAN that you will be using for the performance testing otherwise you won't be able to achieve these numbers if one of the computers can't handle the data as fast as the other one. Try it with your laptop and desktop for example.
NetCPS - is a handy utility to measure the effective performance on a TCP/IP network.
Just execute "netcps.exe -s" on the listening system and then do "netcps.exe computername " on the other computer to use the utility to test the throughput bandwidth. For Gigabit you can use the "-m1000" switch to increase the transferred amount to 1,000 MBytes instead of the default 100. Below is an example.
o Uninstall un-needed software reduce traffic
o Linux to avoid WindowsUpdate and AntiBadwareUpdates reduce traffic
o Network settings reduce traffic (NoTimeStamp etc)
o Firewall reduce traffic
o Squid reduce traffic
o Privoxy reduce traffic
o Reduce size of files that may use the network
= less traffic also means smaller power bills
Tells you go spend 1500 on 12 cores and then explain that you need spindles...
My office is located in what I am led to believe is the armpit of connectivity in the Seattle area. We have no options available to us besides T1s (which we have two of right now, and are getting a couple more installed soon). We're too far from the CO to get DSL (really?), no cable service (despite it being available about half a mile away), no reasonably-priced fiber (FiOS), and we tried Clearwire, which was an absolute joke. I must have called a dozen places trying to see if there was a way around any of it, but apparently not. It's really, really awful. I cannot believe it's 2010 and there is nothing besides T1s available to us. Our IT department encourages us to go home to download files > 50MB, which is a little insulting until you realize you can take about a two hour lunch under the guise of "going home to download."
When /dev/null starts giving access to all the files it has gobbled up over the years I imagine would be like the gates of hell opening. Dennis Ritchie as pestilence will ride a black horse made of swarming bits astride with other famous Unix dudes (imagine your own!). Sysadmins who have been practicing the arcane arts of administrating access to Hell's one and only 9600 BAUD BBS running Minix will rise hungry for bandwidth, porn access and hot pockets.
An Education is the Font of All Liberty
Banning Powerpoint usually shrinks cost by 50% over 5 years by preventing vaporware projects. If they can't present it as a flat text checklist in one page, possibly with hyperlinks, it's much harder to sell vaporware. But the policy is often hard to implement to sell due to entrenched fools idiots who've gotten used to buying and *selling* vaporware and "re-organization" plans that cost far more and waste everyone's time. I just did my best to shoot one down today. It was.... pretty awkward trying to poke holes in the excited "this will fix everyting!" presentation a manager had bought into without any actual engineers who do the work being aware of the presentation having happened, but I did my best to pull in hard experience to explain why the "re-organization" would cost far, far more than it would benefit anyone by citing the costs of the last 3 times I'd seen it tried and fail, and could give references to call.
I'm now on that middle manager's hit list for the next round of setting people up to fire them, disguised as promotions to programs that are guaranteed to fail, but fortunately, he's not in my direct chain of command, and I can those coming a mile off. It's amazing how easy to prevent such a project such a project by tracking down the fool's former secretaries and arranging that a sexeual harassment subpoena be served 30 minutes before their presentation, especially when their former secretaries are male and have grounds to serve the subpoenas. It's even funnier when you can arrange for their presentations to fail because they didn't virus protect their laptop and you can get it blocked from the network by verifying that it's been rootkitted, 20 minutes before their presentation. And yes, I did that last year.
Another INCREDIBLY useful article by kdawson /sarchasm
RFC 1925
Cheers,
Dave
They that can give up essential liberty to obtain a little temporary safety deserve neither safety nor liberty.
Ben
The author mentions the disk access for deduped primary storage (he points out (rightfully so) that deduped primary storage will perform slower than non-deduped primary storage), but he failed to mention what I think is an important point when discussing deduplication and network performance/bottlenecks.
If you dedupe your backups (the author mentions, for example, a VTL solution), you then gain the ability to replicate only the unique data to your DR site. In terms of saving bandwidth, this can be an absolutely huge savings. Imagine if you backup to a VTL, and with dedupe you get an average 25:1 ratio; that means that, for the purposes of DR, you can replicate 25x more data through your pipe than you would have been able to, without dedupe.
Nemilar http://www.techthrob.com - Visit Me!
Rebuttals to TFA's points:
2. See "SLA" section in number 1. If none, losing the leased lines is probably retarded. At least your boss will get a refund check the month after he fires you for choosing "alternatives".
3. Obviously the writer has never seen the opposite, never settling down with any particular system because it's replaced every year with the "next big thing" some CEO read about in a trade rag while drunk in 1st Class doing the globetrotter thing. Or the middle-ground death by attrition where a system is touted as the "best possible solution" tons of meetings to "customize" it for every department's whim, and when it's released it's so complex it slows down productivity, while no one's measuring.*
4. Just makes me laugh my ass off. If the boss were willing to spend $1500 on a single computer, we'd already be doing this with virtualization. We're not stupid. We're using seven or eight year old cast-off desktops from the IT department just to even HAVE a lab. You're hilarious.
6. Bwahahaha... we MAKE apps. Really big important apps for things we do for a living. Well more than 50% of the company doesn't know how OURS work. You think we can figure out how a vendor's app works? Golden.
(Better tactic, start with the damn thing completely firewalled and open enough just to get it working. Note any possible security issues from vendor even though you've already been told by the boss this thing's purchased and you have no choice in the matter, like... "We need TELNET turned on for this thing to work?! WTF is wrong with you people?!" Sigh, and continue to load said app that the boss says has to be loaded, note that boss is on the Board of Directors of the company that produces that software, and their CEO on yours. Realize you're out of your league, and try to figure out how to make good backups so when the thing gets the smackdown by a 14 year old in their basement, you can at least put it back online rapidly a few times in a row, overnight, while you finally have enough ammo to tell the boss the thing's insecure and the other vendor needs to fix it. Hopefully by then one or the other CEO will have been replaced, and no more golf games to discuss who will buy each-other's crap software this year.)
7. Did you read number 4? We're running everything on some old 20 MB SCSI disks and Ultra 5's from eBay. Seriously.
8. Again, number 4. We hit "over-virtualized" the day we loaded the virtual server. Who are you kidding?
9. ROFLMAO... you assume that we have a SAN or data that's backed up?! Oh, you're rich, my friend. Rich.
10. Backups. Again with the backups.
This might get modded "Funny" if folks think it's sad enough. Or they might realize it's true...
*"In order to open a ticket you must click the serial number field, type a 20 digit serial number, if it's not found the number will be deleted from your screen, so make sure you type it into Notepad first, after that, type the customer's last name, wait 15 seconds for all customers by that last name worldwide to pop up (can't select company name first), re-query using customers last and first name (not an option on the main screen), find one of six instances of the same customer (other people got lazy and just typed them in), pick the most accurate one, now back to the main screen to type in the company name, pick one of the ten of those, or if you like, one of their divisions that was put in with the full business name, all with different account numbers, now click "check entitlement" and see if they have a service contract."
See any possible problems with that? Okay now do it in 1 minute or less so the guy on the phone doesn't get pissed that you're putting him through the gauntlet (because you talk to him regularly, and already know his personal information), and stay nice and friendly and chit chat while you're doing it. Yes, I just described the REAL system I work with daily. It cost $2 million to deploy. Based on Siebel, which was a nice system when it was a desktop app, but is a god-awful web interface that "saves deployment time and money". Awesome ain't it?
Such fun reading this BS from writers who've never done any of it anyplace with a real budget. BOFH is closer to reality.
+++OK ATH
Tip #6 is the key but why not to design your applications and infrastructure for performance instead of "knowing" it? It's correct that infrastructure performance monitoring only gets so far - why even let it to go there? It's always less expensive to design upfront than trying to tune it later. Of course, if you or you company has already made bad decisions, it is more difficult but late is better than never. Trying to fix performance problems with vendor / manufacturer magical tools and toys is always doomed to failure even if it in short term it may look like "a miracle"!
Yes, especially tips #9 and #10, dedupe and fast backups are useful but doesn't everyone do that? For example dedube in nothing new - a long, long time ago the big systems only saved the changed information, be it backups or transaction logs - fast restores, less and faster to backup, etc!
#1 and maybe others can come later - if you are not yet desperate! #1, faster communications, etc is actually kind of worse - you give performance and often the effort is stopped there until next crises, usually twice or more worse! That's just normal corporate thinking!
I read your comment and it's pretty damn useless as well, all of our servers run Linux. I've never heard of "interrupt moderation", but it does support ethernet bonding.
If you ban CIFS and NFS, what's left? Sneakernet has great bandwidth, but the latency sucks and it's a bitch to search.
SSHFS :)
Not if there is any possibility that two users will write to the same directory (or file) at the same time. The underlying sftp will probably make a little mess if that happens...
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
insmod e1000.ko InterruptThrottleRate=1 (Mode 1)
insmod e1000.ko InterruptThrottleRate=0 (ITR off)
insmod e1000.ko InterruptThrottleRate=8000 (Fixed value for all I/O patterns)
Page 17
http://download.intel.com/design/network/applnots/ap450.pdf
Nice informative post, interrupt moderation sure sounds interesting. Link aggregation, however, is not as useful as it sounds for the following reasons:
1. Hardware link aggregation (link aggregation supported in silicon) works by hashing, not by distributing packets evenly across all links that are aggregated. If you can spare some time to ponder about this for a moment, you will be able to see why hashing is used. In real life situations, 4 x 1Gbps links aggregated together never equals 4Gbps throughput.
2. If link aggregation is handled by the software (which is most likely the case if aggregating multiple NICs on a server) then all it really provides is redundancy. It is very difficult for an average server to process 1Gbps of incoming traffic, let alone generate 1Gbps of worths of traffic. Not to mention the read/write speed of the storage device(s) used in the server.
(Unless it's using PCIE SSDs in RAID configuration, which would be very interesting and I am dying to find out the throughput of such a configuration!)
For once I actually know what I am talking about, so maybe I should have created an account before posting this one.
Brilliant, that'll be useful on our servers with Intel GbE NICs =]
Stop downloading pr0n.
Analytic & algebraic topology of locally Euclidean meterization of infinitely differentiable Riemmanian manifold
http://idle.slashdot.org/story/10/06/01/1324220/Snails-On-Methamphetamine
Whomever wrote this article only mentioned back-end data de-dup. Avamar (EMC) is client side data de-dup, hence lowering network traffic for backups and spreading the work load to the clients during backup windows.