Slashdot Mirror
Intel Gigabit NIC Packet of Death
An anonymous reader sends this quote from a blog post about a very odd technical issue and some clever debugging:
"Packets of death. I started calling them that because that’s exactly what they are. ... This customer location, for some reason or another, could predictably bring down the ethernet controller with voice traffic on their network. Let me elaborate on that for a second. When I say “bring down” an ethernet controller I mean BRING DOWN an ethernet controller. The system and ethernet interfaces would appear fine and then after a random amount of traffic the interface would report a hardware error (lost communication with PHY) and lose link. Literally the link lights on the switch and interface would go out. It was dead. Nothing but a power cycle would bring it back. ... While debugging with this very patient reseller I started stopping the packet captures as soon as the interface dropped. Eventually I caught on to a pattern: the last packet out of the interface was always a 100 Trying provisional response, and it was always a specific length. Not only that, I ended up tracing this (Asterisk) response to a specific phone manufacturer’s INVITE. ... With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death — and kill client machines behind firewalls!"
I think an actual summary would have been a vast improvement over TFS.
Whether it's your brand of switch, motherboard or even memory, never have the same across all machines if you can help it. The only time I'd recommend the same brand would be hard drives (due to concurrency issues), but then at least try go get them from different batches. If your lot of mobos will only handle one brand of memory for whatever reason even when cas latency is identical, then have two machines doing whatever it is you need to be doing.
One kind of anything makes it easier to kill you swiftly in the end, whether it's by a ping of death or a biological disease.
If computers were people, I'd be a misanthrope.
``Life is too short to be spent debugging Intel parts.''
-- Van Jacobson
http://www.cloud65.com/ just as Marcus answered I didnt know that a mother can profit $8765 in four weeks on the computer. did you read this webpage
I think the NIC packet of death might be just what you need.
Agreed. OP clearly has no experience managing large server installations.
Listen here my friend, has anyone really been far even as decided to use even go want to do look more like?
the drivers are certified to work
LOL this is a firmware bug, you can lock up the hardware even with no OS booted. Hilarious.
and you get real support
Yeah I love being told to reinstall windows on my linux boxes. Those guys sure are helpful !
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
I think the literal flying dagger of death might be just what he needs. And Marcus too. But before that, we'll teach him how to use capitalization and puncutation. Because it would be morally wrong to kill him before he understood these things.
One kind of thing makes it a zillion times easier to recognize a problem when it crops up, and makes it so you only ever have to troubleshoot an issue once.
How much more awful would it be if something similar happened next week on more computers, and he had to troubleshoot it all over again-- not even knowing whether the machines had NICs in common?
"Everything blew up" is a problem. "Everything blew up, I dont know why, and it will take 3 weeks to find a solution" is a huge problem. "Everything blew up AGAIN, and I it will take another 3 weeks because our environment is heterogenous" means you are out of a job.
oh I think this is at least slightly interesting. I remember the "ping of death" (and pissing off a few windows heads in my sights) back in 'th day.
This is basically a DoS attack on hardware. The fact that it can get through someone's firewall makes it a bit more effective. Having your ethernet port check out every five minutes (requiring a reboot to fix) just because someone down the hall (or in Bulgaria) wants to be an ass is definitely annoying and something I'd like to know is a possibility when troubleshooting screwy network problems.
I just got done swapping out a gigabit switch that was being wonky and slow for no obvious reason. I don't mind so much when hardware keels over and dies, but when it throws symptoms that don't immediately suggest where the problem is, those are the real time wasters. And we've come to rely on hardware generally being more reliable than software. So if my ethernet was going out when I VOIP'ed, I might have spent (wasted) a lot of my time troubleshooting the VOIP software.
I work for the Department of Redundancy Department.
It's actually a pretty good write up with a nice trace of his troubleshooting. If my customers gave me bug reports that included 10th of the level of detail he does in the article, i'd be over the moon.
Wasn't there an old program (Nuke 'em on the Mac I think), that would send out-of-band data (whatever that was), and it would crash the TCP/IP stack on Windows NT 3.51? There was another program on Linux called Pam Slam or something like that, that would also bring down NT servers... Very popular in the early days of the web to bring down your competitor's website.
If telephones are outlawed, then only outlaws will have telephones.
or just buy premade servers from dell or HP. they aren't that much more expensive, the drivers are certified to work and you get real support
...and you're guaranteed that every shipment will have radically different hardware, despite having identical model numbers.
Peter predicted that you would "deliberately forget" creation 2000 years ago...
if ($uid -ge 1000000) || ($uid == "Anonymous Coward"; then /dev/null
cat $foo >
else
cat $foo > $file
fi
---Up Up Down Down Left Right Left Right B A START
I'm guessing you didn't buy them with Linux on them... or prove it was a hardware issue. They have no reason to support something they didn't ship. Sure the support varies but their pro server support is actually decent if you get the right person on the other end. I had a case where teaming 2 nics caused windows to eat crap and die inexplicably and getting it back up was quite the ordeal. I couldn't even keep it stable long enough to unteam or remove the drivers (even in safe mode). Fortunately they did have documentation on the problem - a broadcom driver had a problem with a particular firmware set when teaming was used. I managed to flash the firmware update from a usb flash drive which got me to the point I could at least boot into safe mode and delete the drivers and then get a working older version of the driver from Dell's site up and running and teaming reconfigured. This was on an poweredge r610 btw. I feel bad for the poor sap who ran into this first and having dell support saved me unnecessary downtime, especially since there is no mention of this problem anywhere on broadcom's website. That said for 99% of the issues I've ever run into having on-site spares and a good internal KB has been far more effective than paying for Dell's support, but if it is free with the server why not use it...
Get a web developer
This hurt my brain.
There's a good reason a lot of our equipment is slightly older. No, we don't use ancient stuff, but they're not 100% top of the line made yesterday either. And that's because each time a new mobo, memory and storage combo that looks like its worth purchasing comes to market, the first thing we do is run a few sample sets under everything we can throw at it. Usually problems are narrowed down within the first couple of weeks or so, but that's why we have separate people just for testing equipment.
Now admittedly, it's getting harder with this economy so we have some people doing double duty on occasion (I've had to do a bit too when the flu came rolling in), but testing goes on for as long as we think is necessary before the combo goes live. We avoid a lot of the headaches that come with large deployments by keeping changes isolated to maybe 10-15 nodes at a time. It's a slow and steady rollout of mostly similar systems (maybe 3-4 identical) that helps us avoid down time.
We're not Google and we don't pretend to be, but common sense goes a long way to avoiding hiccups like "everything blew up". I think the biggest issue was when hurricane Sandy hit and we weren't sure if the backup generators would come online (this is a big problem with things that need fuel and oil, but stay off for a long time), so we brought in a generator truck for that too, just in case. Again, avoiding one of anything.
If computers were people, I'd be a misanthrope.
I would be curious to know if other versions like the Intel 82576 have the same vulnerability. Maybe we should crowd source this and people can post what they've tested with and received the same behavior.
Are We the Imperial We or the Editorial We?
Curious.
To ensure perfect aim, shoot first and call whatever you hit the target
I for one definitely appreciate the diligence of Kristian Kielhofner. Many years ago I was supporting a medium-sized hospital whose flat network kept having intermittent issues (and we all know intermittent issues are the worst to hunt down and resolve). Fortunately I was on-site that day and at the top of my game and after doing some ethereal sleuthing (what wireshark was called at the time), I happened to discover a NIC that was spitting out bad LLC frames. Doing some port tracking in the switches we were able to isolate which port it was on which happened to be at their campus across the street. Of all possible systems, the offending NIC was in their PACS. After pulling the PACS off the network for a while the problem went away and we had to get the vendor to replace the hardware.
Editorial, I assure you. :)
If computers were people, I'd be a misanthrope.
That's not the same bug. I'd explain, but that's what you get for saying "I wish this guy had done his homework."
I'm glad Mr. Kielhofner contacted Intel about this issue and had Intel confirm the bug.
Some years ago I had been diagnosing similar server NIC issues, and after many hours digging, Intel was able to determine the fault was due to the four-port server NIC being counterfeit. Damn good looking counterfeit part! I couldn't tell the difference between a real Intel NIC and counterfeit in front of me. Only with Intel's document specifying the very minor outward differences between a real and known counterfeit could I tell them apart.
Intel NIC debugging step #1 = verify it's a real Intel NIC!
or just buy premade servers from dell or HP. they aren't that much more expensive, the drivers are certified to work and you get real support
...and you're guaranteed that every shipment will have radically different hardware, despite having identical model numbers.
Sad but true. It makes it a PITA when dealing with disk images from one server to another.
Errrr, no. Have you ever tried to deal with replacements and/or issues within a large organization where everything is different? It's hellish.
Try tracking an issue across an enterprise of architecture when all the architecture is DIFFERENT. You also don't want to mix RAM, and drivers can be a real b**** for different motherboards. Oh, and RMA's things, not fun.
Different brands of RAM. Yeah, you try a rack full of servers playing mix'n'match and see how well that works.
Lastly... how many vendors/brands of enterprise gear do you think are out there, and for the ones that do exist how well do you think they talk together. Maybe you're happy mixing HP Procurves with your Cisco stuff but I don't recommend it, and for some stuff there aren't a lot of vendors to choose from anyhow.
They have no reason to support something they didn't ship.
They shipped you hardware. Therefore they need to support THE HARDWARE.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
line 2: -ge: command not found
with $uid set to 1000001:
line 2: 1000001: command not found
The condition is always false and the user never goes the /dev/null
I guess you need to use brackets, in bash at least...
Everything I write is lies, read between the lines.
Not to speak for OP, but there is a hint of logic in there. It wouldn't apply at farms where hegemony translates into resiliency, but it would apply in situations where resiliency results in the ability to withstand faults without replacing anything. Military and other tier one instances come to mind.
"Over specialize and you breed in weakness"
- Major Kusanagi Motoko
Intel NICs have (or at least had...) a very good reputation for performance and stability. Maybe this is a sign that their QA is starting to slip?
%$#@! auto-correct, I meant: homogeneous
You will have to update the kernel, though. The linux e1000 and e1000e drivers have a fuckload of hardware bug workarounds, and the ASPM thing did hit some people recently. You *must* have ASPM L0s and L1 disabled on the Intel NIC *and* its parent PCIe bridge, and the kernel driver usually will only be able to disable it on the NIC itself, if the BIOS is crap and leaves ASPM L0s or L1 enabled on the bridge or has a crap NIC eeprom image that causes issue with 128b/256b maximum PCIe packet (this one can be fixed by Linux, *if* you give it a specific parameter, no idea why it isn't automatic since it is major utter braindamage by the BIOS that is known to hang the box hard sometimes), the NIC can hang.
So by "bring down" you didn't just mean bring down, implying it was brought down, but you meant "BRING DOWN" (notice the caps), implying it was brought down (notice the italics). Such a critical distinction. If it was merely "brought down" this would hardly have been an issue. You could have simply ignored the dead router. As it stands, being brought down, this is a real problem, and you cannot ignore the dead router. Good job!
Whether it's your brand of switch, motherboard or even memory, never have the same across all machines if you can help it. The only time I'd recommend the same brand would be hard drives (due to concurrency issues), but then at least try go get them from different batches.
...and then along comes something like the Seagate 7200.11 firmware bug from a few years back, which caused all drives of several related models to self-brick after a period of time.
The ubuntu bug had to do with bad drivers and / or firmware; when the affected distro was installed on a computer with the affected NIC (which was the completely different e1000), it would render that NIC unusable, even afterreboots.
This bug appears to be triggered by receiving a crafted packet, remotely, and is fixable with a reboot. It also affects a different nic.
I see that beautiful 4chan meme lives.
Your post made my day.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
Too bad Intel gave a fix to them (a fix they ultimately couldn't use), but hasn't to anyone else.
Too bad Intel has also apparently known about the problem for months now.
"Intel has been aware of this issue for several months. They also have a fix. However, they haven't publicized it because they don't know how widespread it is."
Bullshit. I bet they were hoping to very quietly roll it into a driver update and have it all go away.
Please help metamoderate.
I ran into a tg3 bug where as the tg3 firmware took the byte value that it expected for a destination port number and redirected the udp packets with that value at that location to the BMC/SMDC/ipmi card (as designed). The issue was that the firmware did not appear to understand that a UDP datagram could be up to 64k so up to 40 1500byte packets and was always looks for the destination port on all packets (not just the first as it should have been) so if the data in the packet matched the expectations those packets never got to the OS.
This caused a client to have move their network port on the machine to the 2nd port (on 200 machines) that did not have the firmware bug in it, and this caused us to find another odd firmware bug...the bug being that if one uses jumbo frames and were to explicitly route to a certain set of nodes with smaller packets (to correct someone else's network bug where they sometimes report the wrong MTU size) then the firmware feature that puts packets together nicely helps you and puts the 6 1500's (the route explicitly broke up) together and attempts to send them on as the firmware does not have that complicated set of rules as the OS does around MTU size.
The broadcom guy I talked to (and he was definitely off-shored) was a ID10T, and claimed there was nothing wrong...even though we could generate a valid linux UDP NFS packet every time that would never get to the OS and completely stop NFS from working. The client found it because one of there data streams was running into this feature pretty consistently if the file offsets line up such that the data had the certain magic value that the fw expected at the right location.
And at the end of the day the real issue is the firmware is poorly documented and appears to be poorly tested and reviewed and is terribly important for stability.
You do realize both HP and Dell commonly come with the very Intel NICs being discussed here, don't you?
So for not much more expensive you get the same failures, the drivers are certified but still fail, and the real support clearly never detected or patched this problem. Yeay?
Intel, or possibly nation where the manufacturering happens, is that code was added into the chip to respond to a highly unlikely sequence. Then when you need to kill a large number of computers simply hit various web servers sending in the required packet. Now, if a nation is protected by a firewall, well, then this approach will not be that useful. However, if other nations do not have a centralized firewall/router, then it can be used to take down a nation.
I prefer the "u" in honour as it seems to be missing these days.
Or the Nintendo Wii?
rewriting history since 2109
I wish this guy had done his homework. This was fixed a long time ago:
http://blogs.computerworld.com/when_linux_does_well_the_e1000e_ethernet_bug_fixed
I am amazed that you got a patched e1000 driver working with a 82574L based piece of hardware... mighty impressive hacking! You should have written up a report on how you managed it for the rest of us to study as homework.
It's just an Intel support strategy. Release NICs with random and minor outward differences. When you have a support issue, say that it is counterfeit. Really cuts support costs!
There is a big difference between corporate support and customer support.
Apparently you never worked in corporate environment. If the problem can't be resolved by phone, HP, Lenovo, hell even Dell would send a technicien on site.
What you want is some homogeneity in sections, but heterogeneity between sections, so you're not brought completely to your knees when a bug like this is exploited, but you still have copies of hardware for part-swapping tests or frankensteining old servers.
tl;dr: monocultures suck.
Silence is a state of mime.
Sounds, like you've found the balance point between bleeding edge (things are broken/buggy) and outdated (no longer supported/available).
I wished more people would favor this approach. It would save money and time down the road. i.e. Planned Upgrade Path.
Also, cheaper with older ones. I also don't buy the latest stuff. I want the stable anc cheap ones. Also, older stuff have issues worked out and known. I stopped being in first in line unless I get paid to use and test. :P
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
A number of years ago I discovered that you can take down many routers, and Windows / Linux hosts by sending an ARP response that says "IP 0.0.0.0 is at MAC FF:FF:FF:FF:FF:FF". When you direct this packet to the access point in a wireless network, this makes the SSID broadcast disappear and the whole device go down. Never posted this until now, I wonder if this still works on modern devices.
They have no reason to support something not in an SLA. What they support is only loosely related to what they shipped you.
Maybe, just maybe, some frames could trigger an internal monitoring or debugging mode on the controller? Sometimes, manufacturers would want to remotely diagnose hardware, and that could be a way to do it. Of course, it could also be something else, much more sinister like, say, some obscure government backdoor. Not saying that this applies to this particular case, but since most silicon designs aren't open source, we can't be sure there's no such thing in there, lurking, waiting to be activated.
cpghost at Cordula's Web.
According to TFA (I know, WTF, I actually read TFA?), a reboot does not fix this problem, but a power cycle does.
The real "Libtards" are the Libertarians!
Eventually. After you run their diagnostic, which spends 6-10 hours checking every sector of every hard drive among other things, send them the resulting diagnostic file, wait until they decide that the bad memory you told them about was really the issue after all, THEN the clock start running on the premium "4 hour guaranteed" support.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
Hardware is just what you call something YOU don't configure/patch much even if someone else does :).
To a PHB everything might be hardware. To a HDD maker HDDs aren't hardware, same for CPU makers and their CPUs.
line 2: -ge: command not found
with $uid set to 1000001:
line 2: 1000001: command not found
The condition is always false and the user never goes the /dev/null
I guess you need to use brackets, in bash at least...
You're right, but bash doesn't even enter the picture: /usr/bin/[
$ ls -l `which [`
-rwxr-xr-x 1 root root 35264 Nov 20 06:25
The program is called [ and it complains if its last argument is not a ], so you need the square brackets no matter which shell you use.
And where do I get this mythical "firmware update" for the NIC CHIP? I'm sure the chip has code in it, but I've never even heard of a utility from Intel to update the in-chip code in a nic. (it's called "microcode", not firmware)
IIRC,. this is a known issue for certain chipsets, disabling power management for the PCI-E port the interface is a attached to in the BIOS is the known work-around.
say that it is counterfeit. Really cuts support costs!
Also cuts sales volume. If someone tells me there are lots of counterfeits that look almost like original, I'd stop buying.
No. In my old job we had the four hour gold support on our servers (all 5 of them, we weren't a huge customer).
It typically took less than five minutes on the phone with a knowledgeable techie before they escalated and sent us an engineer. Usually the parts and engineer would arrive within two hours.
Dell consumer support might be shite, but their business support is bloody good.
Same here with HP. Technician driving 250 km through the Black Forest and heavy traffic, there in 3 hours.
And we were an NGO with one server only.
The dangers of excessive individualism are nothing compared to the oppressiveness of excessive collectivism
Was just end of day, I'm totally checked out pseudo code- relax gents. But I should have known better if I was to be snarky in bash:
if [ $uid -ge 1000000 ] || [ $uid == "Anonymous Coward" ]; then /dev/null
cat $foo >
else
$foo > $file
fi
---Up Up Down Down Left Right Left Right B A START
the second condition should probably be another variable too like $uid_cn or such.
---Up Up Down Down Left Right Left Right B A START
You probably had a different issue than we did (random almost-daily blue screens) on one of 72 identically configured and imaged R510 servers we deployed as NVRs in half a dozen data centers. The tech that came took out all the RAM and reseated it (which I had already done) and told me the problem was gone. He'd put the DIMMs back in the wrong slots though, entailing a new set of diagnostics, another visit to put the same damn bad DIMM back in its original slot, three more days of blue screens, and finally another visit to replace it.
My current headache is a R210 that apparently had the wrong image put on it and blue screens out of the box. Took them 4 1/2 weeks to get us the replacement, and it has the same incorrect image. The techs on site shipped it to us, I put the correct image on it and it's happy. It's now back on its way to the customer site to get installed.
Still better than my previous experiences with Compaq business support, though.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
I started stopping the packet captures as soon as the interface dropped
Yes, that's usually when my packet captures stop, too.
Since no OS is grabbing the data from the buffer how do you plan on shifting the bytes through it in order to trigger the flaw? While it is not entirely outside the realm of possibility, it is unlikely this bug would be exposed without a driver on the host interacting with chipset and putting the firmware through its paces.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Rules for adblock plus:
ibtimes.co.uk##.ibt_con_artaux.f_rht
ibtimes.co.uk###bg_header
ibtimes.co.uk##.fb-like.fb_edge_widget_with_comment.fb_iframe_widget
ibtimes.co.uk##.twitter-follow-button.twitter-follow-button
ibtimes.co.uk##IMG[style="border:0;width:20px;height:20px; margin-top:-10px;"]
ibtimes.co.uk###scrollbox
ibtimes.co.uk###taboola-grid-3x2
ibtimes.co.uk##.f_lft.morebox
ibtimes.co.uk###wrap_bottom
ibtimes.co.uk##.bk_basic.bk_disqus
When the NIC powers up the first thing it does is load a bunch of default settings from a chunk of nonvolatile memory (also sometimes called the EEPROM).
You can reprogram the EEPROM using tools from the vendor, or if the driver supports it you can do it under Linux using ethtool.
I've worked a fair bit with the latest 1-Gig and 10-Gig parts (i350 and 82599). They seem pretty decent and stable, good enough for telecom use, though like all chips they do have a list of errata.
The developers are fairly active about updating the linux drivers in the core kernel as well as on sourceforge. The new chips (the 10-gig one especially) are very flexible but this means the drivers are getting a lot more complex than they used to be. (The programming manual for the 82599 is 900 pages.)
We posted the offending packet on CloudShark, with links to all of Kristian's articles. Check it out here: http://appliance.cloudshark.org/news/cloudshark-in-the-wild/intel-packet-of-death-capture/
Bought a lot of HP and Dell hardware, never seen this happen. What server models and what hardware ?
And while it's tempting to write ==, it should be just a =
I'm won't admit how many years of unix/linux it took for me to notice that, that few shells bother to complain about it.
1 Earth is warming, 2 It's us, 3 it's royally bad, 4 we need to take action NOW
Make sense, I always wondered why you needed spaces after the [ and before the ]. Make sense if they are program arguments. Good one !
But in realty, bracket support has been built into bash for eons mostly for optimization purposes. It is the case for other functionalities where the legacy executable is still present on the system but not needed.
~# which [ /usr/bin/[ /usr/bin/ttt /usr/bin/ttt: No such file or directory /usr/bin/[ /usr/bin/ttt /usr/bin/ttt /usr/bin/[
~# ls
ls: cannot access
~# mv
~# which [
~# if [ 1 = 1 ] ; then echo true; fi
true
~# mv
Everything I write is lies, read between the lines.