Slashdot Mirror
Dell Ships Infected Motherboards
An anonymous reader writes "Computer maker Dell is warning that some of its server motherboards have been delivered to customers carrying an unwanted extra: computer malware. It could be confirmation that the 'hardware trojans' long posited by some security experts are indeed a real threat."
That's some great QA you've got going on over there.
Check out my world simulator thingy.
pwned.
Blank until
The Pentagon is spending millions on research designed to ensure it can trust the microchips in critical systems, especially those made outside the US.
- I think the only true way to be sure is to manufacture the microchips yourself, of-course this costs much more than millions.
This comes down to the old question raised by Ken Thompson of Trusting Trust.
You can't handle the truth.
When will the world wake up and realize that lowest price is not a good reason to buy something. Fucking sheep. PAY FOR QUALITY!
It's firmware, meaning software in a ROM. It's only slightly unconventional.
And they say it's only on motherboards sent out as replacements. Interesting, you would think this would make it fairly easy to identify the source.
This malware code has been detected on the embedded server management firmware.
Firmware != Hardware It would have been impressive if it was a real hardware virus though e.g. some malicious chip that opens a backdoor on the network cards and allows remote code execution.
I used to have an IBM server with an IPMI module, that's basically a little computer that can piggyback on the network interfaces and which provides monitoring (on the eServer 325 you can see all of the ~10 fans' speeds, the voltages, and about eight to ten temperatures) and some limited remote management like immediate or scheduled shutdown and startup. It's actually an MSI mainboard IIRC, they went on to make nicer versions of the same stuff with more processor support for their own productization, all too different to use their BIOS on the IBM unit :) One of them may have become the eServer 326?
Anyway, way too much historical data. The point is that the IPMI module could be made by an OEM's OEM...
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Did they buy the parts from Sony? ;)
TFA didn't say how the malware got there.
My guess is that the computer that flashed the firmware was infected. Too bad Dell hates Linux; had the computer that flashed the firmware been running Linux or BSD (or even a Mac) this likely wouldn't have happened.
Free Martian Whores!
a feature.
Basically the entire computer's assembled in a sweatshop by barely literate people who are being paid jack-shit to assemble a "rich-boy toy" for some perceived fat cat in the US who sleeps on piles of money.
How the hell would they know if someone decided to pull a dick move like this?
And for what they're being *COUGH*paid*COUGH*, why the hell would they even care?
Chas - The one, the only.
THANK GOD!!!
I have not studied computer science, firmware trojans nor antivirus. Could someone explain to me:
1) How do firmware trojans work?
2) Are they OS independent?
3) What information can they send and/or damage can they do to a system?
I call it 'The Aristocrats'
I mean, if these motherboards from China are coming with malware... who knows how many of that stuff is already out there spying on us or using our PCs as bots without any way of finding out?
They might have it where there can be an extra chip on the board that simply copies the packets you're sending out and also sends them to some server over in China.
DoD has brought up this concern before, yet hundreds of Dell PCs are in secret military compounds and companies that do classified work, and a few years later China comes out with the same weapon/spy-tech.
How can you make such a claim?
Outsourcing to the cheapest bidder absolves them of responsibility?
I guess OJ really was innocent, and the lady that burned her own crotch by spilling coffee on herself really did deserve the million bucks from McDonalds..
No wonder the world is in shambles..
Please stop bringing up the McDonalds coffee case if you don't know the facts, and if you did know the facts you wouldn't have brought it up. Granted, even if you fully believe the decade old media misrepresentations of the case, I fail to understand how it's remotely relevant here.
many parts are sourced from china. would it not be distinctly possible for that government to experiment with such trojans? most likely the evidence trail would be hard to track.
Birth is the leading cause of death.
Imagine the repercussions for all other companies re branding these servers for their blackbox security appliances, which are basically
spray painted Dell servers with a custom OS and apps.
Who uses Dell for turnkey appliances? Can they vet that their products (some of them performing security related functions, coincidentally enough) are not vulnerable?
yikes.
**This call may be monitored for quality assurance purposes.**
Customer: Hi, my computer won't POST.
Steve (Samir): Okay, sir, first we must try a few things. Is the machine currently plugged in?
**3 hours later**
Steve: Sir, the problem appears to be a faulty motherboard. Unfortunately your system is out of warranty. Luckily, while the system was operational, our integrated key-logger was able to pull your shipping address and credit card numbers. We have billed you for a replacement system and it should be there in 3-5 business days. Someone will need to sign for it, perhaps your oldest daughter. Justine is turning into a fine looking young-lady, by the way.
A few of their SERVICE stock for a single motherboard showed signs of malware code on the embedded server management firmware. Dell reacted quickly and appropriately. You can read the forum posting that started this all here: http://en.community.dell.com/support-forums/servers/f/956/t/19339458.aspx
Of course this is disturbing, but it's quite a leap to say a 'hardware trojan' is 'shipping with Dell Servers'. Once again, a good example why you should never blindly trust "anonymous posters' on Slashdot... RTFA yourself.
The problem here is not necessarily one of QA, but it is one of rigorous procedures. The "hardware" is actually a flashable firmware image. The manufacturing process involves taking all the necessary programmable components and placing them onto the board in question. Now, I have no inside information on this current Dell problem, but perhaps the factory flashed the wrong image. Perhaps the ESM engineer was on his way out and decided to be malicious in a last-minute "oops" fix. Perhaps this issue only manifests itself in a very specific combination of hardware and OS that is so unlikely to be combined it didn't present itself in QA. I think the rush to assume that QA was inadequate or that foreign "sweat shops" are to blame is very naive. The overall complexities of todays x86 machine make all permutations impossible to test in a reasonable time frame. The real test for Dell is whether they plug this process hole and whether we see similar headlines in the future.
"I have no inside information on this current Dell problem .."
Perhaps it was malicious space aliens that compromised the manufacturing process.
Its not bad enough they ship with windows ?
#include bier;
There are some issues where malware winds up in places, and that is something beyond the vendor's control. However, having the motherboard's BIOS infected is just plain not excusable. How can people have any guarantee of security if a maker's QA process allows this stuff to happen? Even if they offshore it to another contractor, the buck stops at the company whose name is on the machine. How can we be sure that replacing the management software and/or a BIOS reflash will take care of the problem?
At least there are plenty of vendors to choose from in the x86 server market. IBM has some very good machines. HP always has had quality offerings. Oracle sells x86 and SPARC hardware, Cisco sells x86 servers that are decent. Even Apple has a top quality 1U server that can both work in a server room as well as a musician's rack.
"I just got a telephone call from a service scheduler informing me that the replacement R410 motherboard I received several weeks ago contains spyware in its embedded systems management firmware" peternli on July 20 2010 8:54 AM
"The service phone call you received was in fact legitimate. As part of Dell's quality process, we have identified a potential issue with our service mother board stock, like the one you received for your PowerEdge R410", DELL-Matt replied on Jul 20 2010 10:31 AM link
--
Imagine having to sit on discussion forums all day typing corporate bum-fluff©
I'd like them to ship Infected Mushrooms
The firmware is installed over seas / the system doing the install is the system that like has the infected code running it. It's just like the mp3 players and other stuff that has a usb disk / some kind of base code. SO THIS WHY TO UPDATE The bios on all new systems.
This why you need to install firmware bios updates on all new systems when you get them in as the first thing.
In other words, in an attempt to shut serious discussion down, piss all over the forum ;)
key words: climate, change, coercion, corepirate, deception, deforestation, destruction, ego, evile, extinction, fear, FraUD, frightening , greed, hired goons, hunting, illuminati, impoverished , nazi, nazis, Pleistocene, pollution, pyramid schemes, terminal damage, wars, wicked ways, wipe out
I have seen something like this before a friend of mine bought a new computer when we were in college and it kept having issues. One day in class we figured it it had a BIOS virus right from the factory. Easy enough to fix just flash the BIOS and everything is back to normal or pull the bios battery out over night. This is not the first time something like this has happened from computer manufactures. This is how ever the first time I have ever heard of server hardware having this issue.
http://www.thetechnologygeek.org
I didn't know that Dell owned a naval fleet.
Knowledge is power. Knowledge shared is power lost.
PROOF that unfettered markets WORK!
This isn't the first time a company shipped malicious hardware.
About 20 or 30 years ago Apple shipped some keyboards with malware on them.
Dell Boards Infected Motherships
perhaps I should take a break from Alien Swarm...
I have been a loyal Dell customer for many years. I am also a Dell Partner. Between this event (hardware malware), their bogus denial of system design and manufacturing faults on millions of Optiplex systems, battery failures on their laptops (I've had 2 fail in 18 months on my D630), and other design/manufacturing issues, I have finally decided that I will NEVER (never being a really long time) purchase another Dell, or recommend one to my clients. A reputation is hard to gain, but easy to lose. I've been patient with Dell, but this is the final straw. Sorry Dell, but you have caused what may be your own demise.
Sometimes, real fast is almost as good as real-time.
Dell could have kept development in-house and STILL keep costs down. However, management realized that if everything is done in house, when something fails it is DELL's fault, and heads must roll. By outsourcing, when a major screw-up is discovered, Dell management can blame someone like Foxconn, and not have to worry about any DELL manager taking the blame. They saw this coming, so they created a way to avoid the blame.
I need trepanation like I need a hole in the head.
I already pwned my Dell for 20 bucks and a rusty old lawnmower.
And how do you know the firmware binary you are installing is free of malware? How do you know the Windows/Linux binary application used to install the firmware is also free of malware? None of that software is open.
Elsewhere, researchers are also investigating the threat from would-be chip-plant saboteurs, who poison the chip-making processes to introduce a "kill switch" that makes the chip fail unexpectedly.
I wonder if these are the researchers trying to work around the Droid X kill switch?
Closed source firmware is not and never will be secure.
I really does not come as a surprise, that now many things at dell are broken, their leadership, support, now hardware comes broken, or compromised. I guess it might come as a surprise that most their hardware is made in china! We all know china wants to have the biggest botnet to control and censor the internet
How about all that crapware that computer manufactures send with a new computer, all the shareware and other programs that you'll never use for the life of that computer. Let me have a clean install, and crowd up the windows registry as I want. Two problems solved, my computer runs faster...and no chance in hell of malware.
Now, it is a rare week that I don't see a blurb about Chinese workers striking for higher pay. I for one would welcome a rising tide of labor costs in China to perhaps level the playing field a bit. Of course, China is now slowly expanding into the role of an African colonial power, so maybe that's were the newest dirt-cheap labor market moves to - assuming they can keep the warlords and dictators compliant.
Why should I be any more confident that a BIOS update is less likely to have malware than the OEM BIOS that ships with the hardware? I'm really asking.
i always thought of the Dell bios as a total as an unwanted extra....
I've been screaming for years that most silicon-based devices have inherent flaws.
Looks like one just got found. If you can access the registers, you can do almost anything.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
This is another reason why people need software freedom for the firmware in their computers. Apparently we need to be able to inspect, share, and modify this software. Coreboot is a great project along these lines.
Digital Citizen
Wow, that realization only took 17 years:
subversionhack:
http://subversionhack.livejournal.com/
~hylas
Let's face it, Dell is the Ryanair (or, if you're American, the Southwest Airlines) of server vendors. Anyone who's ordered a server from them knows the drill only too well.
You want a cheap server? No problem, sir.
Oh, you wanted hard disks with your server? They're an optional extra, sir. They cost more.
You wanted more than 512MB RAM? That'll be extra, sir.
You wanted a processor which wasn't discontinued 18 months ago yet somehow we've managed to find a whole warehouse full of the buggers? That'll be extra, Sir.
You want a 3 year warranty or are you happy with our standard 30 minute warranty? Three year warranty's extra, Sir.
You want to actually speak to a technician during the course of the three years? Or are you happy being routed to the office cheese plant? The technician's extra, Sir.
Now we know there's another question they'll ask.
You want a motherboard that hasn't been pre-infected with firmware level trojans? That'll be extra, Sir.
Did anyone read the problem before replying, of course not - this is /. after all - so, from Dell ( just the important points ):
3. The W32.Spybot worm was discovered in flash storage on the motherboard during Dell testing. The malware does not reside in the firmware.
4. All industry-standard antivirus programs on the market today have the ability to identify and prevent the code from infecting the customer’s operating system.
5. Systems running non-Microsoft Windows operating systems cannot be affected.
Doesn't seem very serious, of course it's Windows only so, of course, you are running antivirus AND, of course, after motherboard swap don't put it to production without testing - which would catch it?
Anyway, still wondering even without antivirus - home come that people let their systems communicate over network with unauthorized traffic? Just going back 20+ years designing network systems, some even Windows, my systems never allowed any unauthorized traffic in or out - this of course sometimes needed even building your own comm. stacks, traps, hooks, proxies, whatever but also guaranteed that all traffic was legitimate! Saves a lot headache - of course all attempts were logged, alerted and, in case of outbound, the sources were isolated - automatically! So - even Windows can be built that way (with pain!), just wondering why some don't do that?
The only way we can be safe is to have the computers design and build themselves!
I'm so glad we are putting essential processes of democracy inside of black boxes.
The advantage, nay benefit, of anonymous is the post must be read and debated on its own merits, not coloured in any way by preconception. /. could use a lot more of that.
... even if the version on the R410 was branded OpenManage(TM) and the firmware may have been a different code base.
Seems to me the only thing new here is that somebody pre-tweaked the code in the shipping firmware load so they, in addition to the authorized IT department, have the necessary keys to "remotely administer" your box, avoiding having to break the stock load's crypto.
Any bets on whether the NSA already has their own way in? Or the Chinese espionage aparatus ditto?
AMT ("Advanced Management Technology") is why I'm not buying Intel-based machines - and when my employer surplussed the old laptops I bought one that was three generations back - adequate, and the last model without a remote-administration "feature".
(I still don't understand why I see lots of Slashdot articles flaming DRM "features", but the remote administration "features" never rise above the noise level - despite being EXPLICITLY a mechanism whose sole purpose is to undetectably and unblockably take COMPLETE CONTROL of the box, spying and/or modifying to any extent desired, rather than just to hobble some of its apps.)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
I saw we seal each board in latex before we ship it to customers. That should take care of the problem!
It's not even really "firmware" unless it's the only thing on the server being booted.
There resides on pretty much every server class device these days some semblence of a panic boot or diagnostics/admin tool that resides on an on-board, USB, or SATA SSD on the system. This got zapped at the factory with a Windows Trojan that could zap the system under the wrong circumstances- and ONLY if you're running a WINDOWS OS on the system.
While it's an epic fail on Dell's part (talk about goofing something up there...)- it's even more of one for New Scientist since they either didn't wait to find out more details on things or didn't bother to read further down in the thread they reference to indicate that this was the case.
It's all about sensationalism, I suppose, these days.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
The true malware ala Dell is in removing anything that Dell installs automatically. The only way to shutdown Dell's bovine scatology is to shutdown Dell, permanently.
The mind conceives, the body achieves, the spirit manifests.
The alternative:
You want a cheap server? Sorry we don't have that!
No but we only sell servers with this much harddisk space.
Of course you need 4GB of RAM for a web server that no one will use.
The processor makes all the difference sir. It's 20Ghz with 50 cores or bust. You can't run {insert outdated crap here} on anything slower.
Yeah it costs $100000 but you'll get a 10 year warranty with a clueless idiot to stand by and not help you fix any problems you won't have.
What do you mean you're going to dell? They only sell cheap stuff!
When i hear the line "hardware trojans' long posited by some security experts are indeed a real threat" all i can think is no shit, it was already done years ago.
http://en.wikipedia.org/wiki/CIH_(computer_virus)
CIH spread and infected the BIOS itself rather than just the filesystem. It was shipped out on a bunch of Yamaha CD drives and the IBM Aptivas had it.
I don't even know absolutely for certain that the Linux binaries that I apt-get install aren't trojaned. Even if I had the time to audit the source and make sure it compiles to the binary I'm getting, I don't really have the ability to do that, especially if I the bar is set to "Never miss one."
However, I'm more confident in those binaries than I am in the proprietary binaries I install on my Mac. At least the .debs are signed and there are some people out there checking.
No, she deserved getting her skin back which wasn't going to happen but more importantly McDonalds needed a legal whack on the nose for selling an unsafe product. It's not kept constantly at boiling point anymore.
I followed all the links in the story and worked my way to the dell forums: http://en.community.dell.com/support-forums/servers/f/956/t/19339458.aspx The "warning" was posted by 'Dell - Matt M'...not a Dell employee.
Only for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for Windows
Please, tell the system the virus/malware/trojan is for. Maybe then we could "get the facts"* right. *http://www.microsoft.com/windowsserver/facts/default.mspx?R=cf
http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
Ken Thompson would show you how you'd fail in this anyway.
The Trusting Trust attack as Ken Thompson described it can be worked around using "diverse double compilation". To defeat this, a compiler virus would have to know how to infect GCC, TCC, Clang, and every other popular Free compiler for a given language, including non-self-hosting compilers (those written in another language entirely). Bruce Schneier explains, as does David A. Wheeler. Likewise, in the case of writing firmware to a flash memory, the would have to know how to infect a Willem programmer, a Wellon programmer, and every other popular flash programmer.
Can you trust compiler of a compiler?
Yes, because one can bootstrap from different independent compiler implementations. I explain why in another comment.
Don't diss Southwest Airlines. They may have a cheap image, but one thing they don't do is nickel-and-dime. They are one of the few remaining airlines that have a two-piece luggage allowance included in the price of a ticket. And they serve free non-alcoholic drinks on board.