Slashdot Mirror
Dell Ships Infected Motherboards
An anonymous reader writes "Computer maker Dell is warning that some of its server motherboards have been delivered to customers carrying an unwanted extra: computer malware. It could be confirmation that the 'hardware trojans' long posited by some security experts are indeed a real threat."
That's some great QA you've got going on over there.
Check out my world simulator thingy.
pwned.
Blank until
The Pentagon is spending millions on research designed to ensure it can trust the microchips in critical systems, especially those made outside the US.
- I think the only true way to be sure is to manufacture the microchips yourself, of-course this costs much more than millions.
This comes down to the old question raised by Ken Thompson of Trusting Trust.
You can't handle the truth.
It's firmware, meaning software in a ROM. It's only slightly unconventional.
And they say it's only on motherboards sent out as replacements. Interesting, you would think this would make it fairly easy to identify the source.
This malware code has been detected on the embedded server management firmware.
Firmware != Hardware It would have been impressive if it was a real hardware virus though e.g. some malicious chip that opens a backdoor on the network cards and allows remote code execution.
I used to have an IBM server with an IPMI module, that's basically a little computer that can piggyback on the network interfaces and which provides monitoring (on the eServer 325 you can see all of the ~10 fans' speeds, the voltages, and about eight to ten temperatures) and some limited remote management like immediate or scheduled shutdown and startup. It's actually an MSI mainboard IIRC, they went on to make nicer versions of the same stuff with more processor support for their own productization, all too different to use their BIOS on the IBM unit :) One of them may have become the eServer 326?
Anyway, way too much historical data. The point is that the IPMI module could be made by an OEM's OEM...
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
a feature.
Basically the entire computer's assembled in a sweatshop by barely literate people who are being paid jack-shit to assemble a "rich-boy toy" for some perceived fat cat in the US who sleeps on piles of money.
How the hell would they know if someone decided to pull a dick move like this?
And for what they're being *COUGH*paid*COUGH*, why the hell would they even care?
Chas - The one, the only.
THANK GOD!!!
I have not studied computer science, firmware trojans nor antivirus. Could someone explain to me:
1) How do firmware trojans work?
2) Are they OS independent?
3) What information can they send and/or damage can they do to a system?
I call it 'The Aristocrats'
Yes, because Windows malware that's designed to infect server system board firmware is so widespread these days and we all know it's impossible to do anything bad from a Linux machine.
It's worse than that. Even those of us that do realize it are kind of stuck. The model that saw out sourcing to China as the solution to pretty much everything more or less obliterated the midrange category for many items. It's really hard to find things these days that are midranged in price and quality. I don't generally need to go top of the line on things, but thanks to the outsourcing there isn't a whole lot of choice, I can cheap out which usually isn't a good idea or buy high end.
The free market really doesn't handle the situation where there's a nascent market for something which investors are ignoring.
It's also possible that the malware was actually dropped from a *nix or Windows system that wasn't itself infected, but where the user wanted to drag Dell through the muck. Doesn't need to be any of these Advanced Persistent Threats you keep reading about, just a terminated employee on his last day. I doubt that embedded hardware is connected to the internet while it's being assembled, so it seems unlikely that they got a chance infection - someone had to subvert their production process. That's most likely to be an insider.
How can you make such a claim?
Outsourcing to the cheapest bidder absolves them of responsibility?
I guess OJ really was innocent, and the lady that burned her own crotch by spilling coffee on herself really did deserve the million bucks from McDonalds..
No wonder the world is in shambles..
If you are implying that if the servers are made in the U. S. of A. this will never happen?
Think again.
And by the way, how much are you really willing to pay extra for stuffs made in the U. S. of A. ?
Muchas Gracias, Señor Edward Snowden !
Please stop bringing up the McDonalds coffee case if you don't know the facts, and if you did know the facts you wouldn't have brought it up. Granted, even if you fully believe the decade old media misrepresentations of the case, I fail to understand how it's remotely relevant here.
many parts are sourced from china. would it not be distinctly possible for that government to experiment with such trojans? most likely the evidence trail would be hard to track.
Birth is the leading cause of death.
Even if all of the computer systems are compromised, I'm pretty sure that China would still need insiders to explain the stolen information or to tell them which data are important.
**This call may be monitored for quality assurance purposes.**
Customer: Hi, my computer won't POST.
Steve (Samir): Okay, sir, first we must try a few things. Is the machine currently plugged in?
**3 hours later**
Steve: Sir, the problem appears to be a faulty motherboard. Unfortunately your system is out of warranty. Luckily, while the system was operational, our integrated key-logger was able to pull your shipping address and credit card numbers. We have billed you for a replacement system and it should be there in 3-5 business days. Someone will need to sign for it, perhaps your oldest daughter. Justine is turning into a fine looking young-lady, by the way.
A few of their SERVICE stock for a single motherboard showed signs of malware code on the embedded server management firmware. Dell reacted quickly and appropriately. You can read the forum posting that started this all here: http://en.community.dell.com/support-forums/servers/f/956/t/19339458.aspx
Of course this is disturbing, but it's quite a leap to say a 'hardware trojan' is 'shipping with Dell Servers'. Once again, a good example why you should never blindly trust "anonymous posters' on Slashdot... RTFA yourself.
Trust air gap!
Anything that millions of tax dollars build in secret should not be stored on a machine that accesses the Internet.
And this is how it's done. I've fixed complex 100+ PC networks that intentionally have no Internet connection.
No Antenna + No wire == safe.
Science & open-source build trust from peer review. Learn systems you can trust.
Its not bad enough they ship with windows ?
#include bier;
There are some issues where malware winds up in places, and that is something beyond the vendor's control. However, having the motherboard's BIOS infected is just plain not excusable. How can people have any guarantee of security if a maker's QA process allows this stuff to happen? Even if they offshore it to another contractor, the buck stops at the company whose name is on the machine. How can we be sure that replacing the management software and/or a BIOS reflash will take care of the problem?
At least there are plenty of vendors to choose from in the x86 server market. IBM has some very good machines. HP always has had quality offerings. Oracle sells x86 and SPARC hardware, Cisco sells x86 servers that are decent. Even Apple has a top quality 1U server that can both work in a server room as well as a musician's rack.
"I just got a telephone call from a service scheduler informing me that the replacement R410 motherboard I received several weeks ago contains spyware in its embedded systems management firmware" peternli on July 20 2010 8:54 AM
"The service phone call you received was in fact legitimate. As part of Dell's quality process, we have identified a potential issue with our service mother board stock, like the one you received for your PowerEdge R410", DELL-Matt replied on Jul 20 2010 10:31 AM link
--
Imagine having to sit on discussion forums all day typing corporate bum-fluff©
wow - i just don't think you have an idea of what is going on here.
this isn't some mass produced USB stick.
in fact the firmware they are infecting on the RAC is just that firmware - akin to the bios on your desktop but different in that some of the RAC's are actually small Linux boxes them selves..
for someone to sneak malware into the firmware of the RAC in a manner that would be useful - this person had to know what they where doing - they had to get it into some build of the firm ware and then either flash them each on their own or into the process and get it to pass QA..
the fact that the malware only effects windows installations on these boxes is a short sightedness of the the person who wrote it. from a hacking stand point.. the value of being able to get custom code to run in the RAC of servers destined for larger companies, its a gold mine.
'...if only "Jumping to a Conclusion" was an event in the Olympics.'
The firmware is installed over seas / the system doing the install is the system that like has the infected code running it. It's just like the mp3 players and other stuff that has a usb disk / some kind of base code. SO THIS WHY TO UPDATE The bios on all new systems.
This why you need to install firmware bios updates on all new systems when you get them in as the first thing.
I have seen something like this before a friend of mine bought a new computer when we were in college and it kept having issues. One day in class we figured it it had a BIOS virus right from the factory. Easy enough to fix just flash the BIOS and everything is back to normal or pull the bios battery out over night. This is not the first time something like this has happened from computer manufactures. This is how ever the first time I have ever heard of server hardware having this issue.
http://www.thetechnologygeek.org
I didn't know that Dell owned a naval fleet.
Knowledge is power. Knowledge shared is power lost.
Dell Boards Infected Motherships
perhaps I should take a break from Alien Swarm...
I have been a loyal Dell customer for many years. I am also a Dell Partner. Between this event (hardware malware), their bogus denial of system design and manufacturing faults on millions of Optiplex systems, battery failures on their laptops (I've had 2 fail in 18 months on my D630), and other design/manufacturing issues, I have finally decided that I will NEVER (never being a really long time) purchase another Dell, or recommend one to my clients. A reputation is hard to gain, but easy to lose. I've been patient with Dell, but this is the final straw. Sorry Dell, but you have caused what may be your own demise.
Sometimes, real fast is almost as good as real-time.
Dell could have kept development in-house and STILL keep costs down. However, management realized that if everything is done in house, when something fails it is DELL's fault, and heads must roll. By outsourcing, when a major screw-up is discovered, Dell management can blame someone like Foxconn, and not have to worry about any DELL manager taking the blame. They saw this coming, so they created a way to avoid the blame.
I need trepanation like I need a hole in the head.
Are you SURE that there is no antenna? Pretty much anything made of metal can act as an antenna.
"His name was James Damore."
Since when does dell hate linux?
I have a server room full of redhat dell boxes that say otherwise. They even sold me the redhat licenses on some of those.
These are servers kiddo, not desktops.
And how do you know the firmware binary you are installing is free of malware? How do you know the Windows/Linux binary application used to install the firmware is also free of malware? None of that software is open.
Google uses Dell for its Google Search Appliance. IIRC, they ship directly from Dell these days.
the growth in cynicism and rebellion has not been without cause
I really does not come as a surprise, that now many things at dell are broken, their leadership, support, now hardware comes broken, or compromised. I guess it might come as a surprise that most their hardware is made in china! We all know china wants to have the biggest botnet to control and censor the internet
Now, it is a rare week that I don't see a blurb about Chinese workers striking for higher pay. I for one would welcome a rising tide of labor costs in China to perhaps level the playing field a bit. Of course, China is now slowly expanding into the role of an African colonial power, so maybe that's were the newest dirt-cheap labor market moves to - assuming they can keep the warlords and dictators compliant.
Why should I be any more confident that a BIOS update is less likely to have malware than the OEM BIOS that ships with the hardware? I'm really asking.
i always thought of the Dell bios as a total as an unwanted extra....
I've been screaming for years that most silicon-based devices have inherent flaws.
Looks like one just got found. If you can access the registers, you can do almost anything.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
This is another reason why people need software freedom for the firmware in their computers. Apparently we need to be able to inspect, share, and modify this software. Coreboot is a great project along these lines.
Digital Citizen
Citation needed.
Since the Apple Macintosh was introduced just about "20 to 30 years ago" (==1984) and the amount of memony and CPU-power in keyboards was very small, then, I guess that your posting is just trollish nonsense.
Wow, that realization only took 17 years:
subversionhack:
http://subversionhack.livejournal.com/
~hylas
Let's face it, Dell is the Ryanair (or, if you're American, the Southwest Airlines) of server vendors. Anyone who's ordered a server from them knows the drill only too well.
You want a cheap server? No problem, sir.
Oh, you wanted hard disks with your server? They're an optional extra, sir. They cost more.
You wanted more than 512MB RAM? That'll be extra, sir.
You wanted a processor which wasn't discontinued 18 months ago yet somehow we've managed to find a whole warehouse full of the buggers? That'll be extra, Sir.
You want a 3 year warranty or are you happy with our standard 30 minute warranty? Three year warranty's extra, Sir.
You want to actually speak to a technician during the course of the three years? Or are you happy being routed to the office cheese plant? The technician's extra, Sir.
Now we know there's another question they'll ask.
You want a motherboard that hasn't been pre-infected with firmware level trojans? That'll be extra, Sir.
welcome datacomp
Did anyone read the problem before replying, of course not - this is /. after all - so, from Dell ( just the important points ):
3. The W32.Spybot worm was discovered in flash storage on the motherboard during Dell testing. The malware does not reside in the firmware.
4. All industry-standard antivirus programs on the market today have the ability to identify and prevent the code from infecting the customer’s operating system.
5. Systems running non-Microsoft Windows operating systems cannot be affected.
Doesn't seem very serious, of course it's Windows only so, of course, you are running antivirus AND, of course, after motherboard swap don't put it to production without testing - which would catch it?
Anyway, still wondering even without antivirus - home come that people let their systems communicate over network with unauthorized traffic? Just going back 20+ years designing network systems, some even Windows, my systems never allowed any unauthorized traffic in or out - this of course sometimes needed even building your own comm. stacks, traps, hooks, proxies, whatever but also guaranteed that all traffic was legitimate! Saves a lot headache - of course all attempts were logged, alerted and, in case of outbound, the sources were isolated - automatically! So - even Windows can be built that way (with pain!), just wondering why some don't do that?
The only way we can be safe is to have the computers design and build themselves!
I hope you don't think the US govt. is much more interested in protecting citizen's rights. The media are doing a very good job of keeping us misinformed. (I checked out the aftermath of another news story last week.)
P.S.: I'm not saying that the style in which the US power structure abuses citizen's rights are the same as that of the Chinese govt. (I *think* the Chinese system is simpler, but that may just be because I've never looked at it close up.) In the US the desire appears to be to function more through intimidation than through actions that leave readily visible permanent injuries. (Among it's citizens. For foreigners the US appears to be more violent and capricious than the Chinese.)
P.P.S.: This can be exemplified by the taser. It rarely leaves permanently visible damage. But it seems, by report, to be used freely and with little cause. I expect that quite soon it will become considered (among oppressed groups within the US) praiseworthy to snipe at cops. Threatening and intimidating is one thing, but when you start widespread torture, you are breeding a case for retaliation.
I think we've pushed this "anyone can grow up to be president" thing too far.
I'm so glad we are putting essential processes of democracy inside of black boxes.
Since when does dell hate linux?
Free Martian Whores!
... even if the version on the R410 was branded OpenManage(TM) and the firmware may have been a different code base.
Seems to me the only thing new here is that somebody pre-tweaked the code in the shipping firmware load so they, in addition to the authorized IT department, have the necessary keys to "remotely administer" your box, avoiding having to break the stock load's crypto.
Any bets on whether the NSA already has their own way in? Or the Chinese espionage aparatus ditto?
AMT ("Advanced Management Technology") is why I'm not buying Intel-based machines - and when my employer surplussed the old laptops I bought one that was three generations back - adequate, and the last model without a remote-administration "feature".
(I still don't understand why I see lots of Slashdot articles flaming DRM "features", but the remote administration "features" never rise above the noise level - despite being EXPLICITLY a mechanism whose sole purpose is to undetectably and unblockably take COMPLETE CONTROL of the box, spying and/or modifying to any extent desired, rather than just to hobble some of its apps.)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
The processor running the spyware is not an x86 and isn't a Windows machine. It didn't just catch something from a workstation.
I saw we seal each board in latex before we ship it to customers. That should take care of the problem!
It's not even really "firmware" unless it's the only thing on the server being booted.
There resides on pretty much every server class device these days some semblence of a panic boot or diagnostics/admin tool that resides on an on-board, USB, or SATA SSD on the system. This got zapped at the factory with a Windows Trojan that could zap the system under the wrong circumstances- and ONLY if you're running a WINDOWS OS on the system.
While it's an epic fail on Dell's part (talk about goofing something up there...)- it's even more of one for New Scientist since they either didn't wait to find out more details on things or didn't bother to read further down in the thread they reference to indicate that this was the case.
It's all about sensationalism, I suppose, these days.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
The true malware ala Dell is in removing anything that Dell installs automatically. The only way to shutdown Dell's bovine scatology is to shutdown Dell, permanently.
The mind conceives, the body achieves, the spirit manifests.
I'll buy that it was an insider (it almost had to be), but not that they did it just for spite. That's a LOT of work for spite considering that the server management board isn't even x86 and it was embedded in the firmware. That's WAY more work than just downloading some crap off the net and slamming it in there.
The alternative:
You want a cheap server? Sorry we don't have that!
No but we only sell servers with this much harddisk space.
Of course you need 4GB of RAM for a web server that no one will use.
The processor makes all the difference sir. It's 20Ghz with 50 cores or bust. You can't run {insert outdated crap here} on anything slower.
Yeah it costs $100000 but you'll get a 10 year warranty with a clueless idiot to stand by and not help you fix any problems you won't have.
What do you mean you're going to dell? They only sell cheap stuff!
When i hear the line "hardware trojans' long posited by some security experts are indeed a real threat" all i can think is no shit, it was already done years ago.
http://en.wikipedia.org/wiki/CIH_(computer_virus)
CIH spread and infected the BIOS itself rather than just the filesystem. It was shipped out on a bunch of Yamaha CD drives and the IBM Aptivas had it.
Yeah. Its kind of interesting that most of our major, critical electronics are being manufactured in china, which is a communist country and not exactly an ally at the moment other than financially. They are already looking at starting another arms race. Imagine if in the 1970s, our computers were being manufactured in the USSR. I don't really see why we should be looking at goods manufactured in china any differently. Clearly, if anything, they have no regard for our health and safety standards and keep churning out crap intended for kids laced with all sorts of deadly chemicals. Who knows. Maybe they are doing it on purpose. It almost seems like it would be fairly trivial for them to slip crap like this in. I mean how many times have laptops and netbooks recently been released with key loggers and everything else because "Ooops, some employee used an infected flash disk to set up the system"
zosxavius photography
I don't even know absolutely for certain that the Linux binaries that I apt-get install aren't trojaned. Even if I had the time to audit the source and make sure it compiles to the binary I'm getting, I don't really have the ability to do that, especially if I the bar is set to "Never miss one."
However, I'm more confident in those binaries than I am in the proprietary binaries I install on my Mac. At least the .debs are signed and there are some people out there checking.
No, she deserved getting her skin back which wasn't going to happen but more importantly McDonalds needed a legal whack on the nose for selling an unsafe product. It's not kept constantly at boiling point anymore.
The free market really doesn't handle the situation where there's a nascent market for something which investors are ignoring.
That's because investors (the bigger ones that matter) are acting as a separate self-interested class that has little in common anymore with the rest of society. This cancels their ability to recognize the finer (and even basic) aspects of what consumers need and they begin to look to the police as the way deal with the results of industrial shortcomings (cybercrime)... lobby for ever more police, more prisons, and invest more in police-related industries.
And to think, all they need to do is put a read-only toggle switch on motherboards. It'll never happen at this point though because it doesn't have anything to do with iPhones or re-purposing military 'innovations' for use against the masses here at home.
I followed all the links in the story and worked my way to the dell forums: http://en.community.dell.com/support-forums/servers/f/956/t/19339458.aspx The "warning" was posted by 'Dell - Matt M'...not a Dell employee.
Only for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for WindowsOnly for Windows
Please, tell the system the virus/malware/trojan is for. Maybe then we could "get the facts"* right. *http://www.microsoft.com/windowsserver/facts/default.mspx?R=cf
http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
Ken Thompson would show you how you'd fail in this anyway.
The Trusting Trust attack as Ken Thompson described it can be worked around using "diverse double compilation". To defeat this, a compiler virus would have to know how to infect GCC, TCC, Clang, and every other popular Free compiler for a given language, including non-self-hosting compilers (those written in another language entirely). Bruce Schneier explains, as does David A. Wheeler. Likewise, in the case of writing firmware to a flash memory, the would have to know how to infect a Willem programmer, a Wellon programmer, and every other popular flash programmer.
Can you trust compiler of a compiler?
Yes, because one can bootstrap from different independent compiler implementations. I explain why in another comment.
Don't diss Southwest Airlines. They may have a cheap image, but one thing they don't do is nickel-and-dime. They are one of the few remaining airlines that have a two-piece luggage allowance included in the price of a ticket. And they serve free non-alcoholic drinks on board.