Slashdot Mirror
China Infiltrated Apple, Amazon and Other US Companies Using Spy Chips on Servers, According To Bloomberg; Apple, and Amazon, Among Others Refute the Report (bloomberg.com)
Data center equipment run by Amazon Web Services and Apple were subject to surveillance from the Chinese government via a tiny microchip inserted during the equipment manufacturing process, Bloomberg BusinessWeek reported Thursday, citing 17 people at Apple, Amazon, and U.S. government security officials, among others. The compromised chips in question came from a server company called Supermicro that assembled machines used in the centers, the report added. The scrutiny of these chips, which were used for gathering intellectual property and trade secrets from American companies, have also been the subject of an ongoing top secret U.S. government investigation, which started in 2015, the news outlet reported. Amazon, which runs AWS, Apple, and Supermicro have disputed summaries of Bloomberg BusinessWeek's reporting.
The report states that Amazon became aware of a Supermicro's tiny microchip nested on the server motherboards of Elemental Technologies, a Portland, Oregon based company, as part of a due diligence ahead of acquiring the company in 2015. Amazon acquired Elemental as it prepared to use its technologies for what is now known as Prime Video, its video streaming service. The report adds that Amazon informed the FBI of its findings. From the report: One official says investigators found that it eventually affected almost 30 companies, including a major bank, government contractors, and the world's most valuable company, Apple. Apple was an important Supermicro customer and had planned to order more than 30,000 of its servers in two years for a new global network of data centers. Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons. [...] [Update: Some counterpoint: According to an earlier report by The Information, security concerns were indeed a reason why Apple and Supermicro parted ways.] A U.S. official says the government's probe is still examining whether spies were planted inside Supermicro or other American companies to aid the attack. Some background on Supermicro, courtesy of Bloomberg: Today, Supermicro sells more server motherboards than almost anyone else. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order server setups at banks, hedge funds, cloud computing providers, and web-hosting services, among other places. Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards -- its core product -- are nearly all manufactured by contractors in China. The company's pitch to customers hinges on unmatched customization, made possible by hundreds of full-time engineers and a catalog encompassing more than 600 designs. Further reading: Amazon Offloaded Its Chinese Server Business Because it Was Compromised, Report Says.
The report states that Amazon became aware of a Supermicro's tiny microchip nested on the server motherboards of Elemental Technologies, a Portland, Oregon based company, as part of a due diligence ahead of acquiring the company in 2015. Amazon acquired Elemental as it prepared to use its technologies for what is now known as Prime Video, its video streaming service. The report adds that Amazon informed the FBI of its findings. From the report: One official says investigators found that it eventually affected almost 30 companies, including a major bank, government contractors, and the world's most valuable company, Apple. Apple was an important Supermicro customer and had planned to order more than 30,000 of its servers in two years for a new global network of data centers. Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons. [...] [Update: Some counterpoint: According to an earlier report by The Information, security concerns were indeed a reason why Apple and Supermicro parted ways.] A U.S. official says the government's probe is still examining whether spies were planted inside Supermicro or other American companies to aid the attack. Some background on Supermicro, courtesy of Bloomberg: Today, Supermicro sells more server motherboards than almost anyone else. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order server setups at banks, hedge funds, cloud computing providers, and web-hosting services, among other places. Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards -- its core product -- are nearly all manufactured by contractors in China. The company's pitch to customers hinges on unmatched customization, made possible by hundreds of full-time engineers and a catalog encompassing more than 600 designs. Further reading: Amazon Offloaded Its Chinese Server Business Because it Was Compromised, Report Says.
Chinese market poison as baby food. Nobody should be doing business with them.
Apple and other companies have responded. It would seem Bloomberg has done little to provide any evidence over the past year, while these companies have investigated and found nothing of substance to the claims. Apple's response in particular is strongly worded and makes it clear that they find these claims to be baseless. https://www.bloomberg.com/news...
China been doing this for years and it's only just coming out.
Everyone involved on both sides has come out publicly to say Bloomberg is wrong. Why are we still talking about it?
All parties involved have it in their vested interest to deny this.
"That's the way to do it" - Punch
Why did a Supermicro get kicked off the NASDAQ ?
If you read the article instead of looking at the pictures, you'd know.
But I'll be kind to the handicapped today.
The device interacted with the BMC, which has lowest-level access to everything. The device would use the BMC to inject code into memory, allowing remote exploits, and phone home.
"Prediction: within 10 years, Windows will be a Linux distribution." Me, 7-6-2016
Auditing revealling financial irregularities that led to delayed SEC filings that predated even the earliest claims made in the Bloomberg article - ultimately it was about breaching SEC filing requirements, rather than the underlying financial issues, that led to the delisting.
UNIX? They're not even circumcised! Savages!
Where'd the chips come from? They are physical things that exist. Do you think Bloomberg faked the paper trail all the way up the supply chain (..)
Bloomberg says A, Apple, Amazon etc say B. That's where you need to back up your claim.
If Bloomberg did its job, it should have some expert(s) on call that can tell you what motherboard, what chip / where on the board, what pinout, what it does, and how they arrived at those findings. That's the core of their story after all.
If Bloomberg does, just publish those technical details & call it a day. If Bloomberg doesn't, then yes they are talking out of their nose and Apple, Amazon & co have every right to criticize them.
Remember when the USA did the same thing?
What? You're just now learning about it and act all surprised...please.
There was never any question what price U.S. manufacturer's were willing to pay outsourcing to Asia. It was just a question how long.
Apple et. al. are not stupid clucks, they went over motherboards with a microscope. They saw exactly how true to their design finished goods matched. Amazon paid a 3rd party due diligence and its public. SO, we have the answer now.
...let's hear more from people whinging about Trump's 'trade war' with China.
China's been a shitty actor on the world stage since they bred themselves out of irrelevancy.
Foreign companies have to establish a Chinese business, owned 51% by Chinese who almost always end up being a front for the PLA. ...and yet we should curry their favor so we can keep buying $9 folding chairs?
Draconian censorship laws. No free speech. No freedom of religion.
Currency manipulation and disregard for norms of international economic (and other) reporting.
Military occupation and absorption of neighbors it deems "were *actually* China anyway".
Sorry Hong Kongers, I guess you don't get to keep democracy and nobody cares...
An arbitrary, dangerously confrontational foreign policy including sweeping territorial claims.
Environmental destruction with impunity.
I don't like Donald Trump for a number of reasons, but the US confrontation with China is LONG past due; waiting any longer would likely make it military when China finally gets brazen enough to try to grab Taiwan.
-Styopa
I don't know to what degree "China" (it's government, it's people, or it's corporations, state owned or otherwise) are spying, but I do know it's not 0, not even close to 0. I have been close to accusations and convictions, they are absolutely spying using any available means. That's not surprising. If it made any sense to do it, adding stray hardware/software to a PC is definitely a viable approach to compromising it.
The real issue is technical. How do we create a secure compute environment? Apple has taken the route on its phones of building a very effective and secure trust chain. It is pretty hard for an unauthorized user to slip in stray firmware on their phones, I don't want to say impossible because there are some known and pretty exotic exploits. But very hard. Their design is such that even their MFGs cannot sneak in stray code to spy on you. The weakest point is still the single authorized user, and their ability to protect their passwords and biometrics. Apple's route also makes you, the owner, a perpetual customer rather than an owner. If they choose to lock you out, there's nothing you can do about it, your $1k phone is a paperweight.
PCs (I'm including desktops, servers and laptops) on the other hand are pretty much a free for all. The MFG can sneak on just about anything in their BIOS/EFI implementation, and anyone up and down the chain can do so without much oversight. It's a pretty open and competitive market, with many small players of little to no account, all trying to make the sale. Each of them provides their own hardware, and some EFI implementation they probably bought and then tailored to their implementation. Someone could also have added backdoors. That in turn hands off to my choice of OSes, which themselves could easily be compromised and I wouldn't know better until something happened. I am unquestionably the owner of this system, and can do anything I would like, but I also cannot rely on anything up and down the system. I'm the owner of a very leaky boat.
What we need is a system that can both be trustworthy and robust to middle-man attackers who may, at times, have direct hardware access, but still allows me to be the absolute owner of my hardware. I may make bad choices, those bad choices may compromise my system, but I need a foolproof way of knowing when I'm making a bad choice. It's not that easy of a problem in the current ecosystem, and we're waiting for someone to get caught doing something bad that forces our hand.
This is only a small part of the issues I have about the report. What is the chip monitoring or able to monitor? How is it programmed?
It's not impossible to envisage something that, say, could monitor Ethernet for a string and use that to program itself, but something that can both see an incoming Ethernet packet and see what the CPU is doing is harder to conceptualize.
I know this is Slashdot but... did you read the article? Supposedly this chip was put on the BMC lines that allow it to modify basically anything going to the CPU. They could have even tweaked the firmware on the board through the BMC. The chip does nothing but detect the loading of the OS and insert instructions that it downloads off of a known host. There was no data exfiltrated as far as anyone can tell. It was just lying dormant or used as a vector to penetrate other areas of the network. They were able to identify the 30 companies affected by monitoring traffic and/or hacking the C&C server used. But it was not detected because, as far as they can tell, the compromised systems themselves were never used to exfiltrate data.
Bloomberg published responses from the companies involved. Here are some excerpts that give you a sense of how they responded...
Amazon:
It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental. It’s also untrue that AWS knew about servers containing malicious chips or modifications in data centers based in China, or that AWS worked with the FBI to investigate or provide data about malicious hardware. [...]
And they go on to say a lot more that categorically denies Bloomberg's claims while making a mention of an unrelated firmware incident from 2016.
Apple:
Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.
On this we can be very clear: Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement. [...]
And they go on to say a lot more that categorically denies Bloomberg's claims while suggesting that Bloomberg may be confused about the 2016 firmware incident.
Super Micro:
While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard. We are not aware of any customer dropping Supermicro as a supplier for this type of issue.
And they go on to say a lot more that categorically denies Bloomberg's claims, including denying that they even make the chips that were allegedly compromised and that these companies supposedly purchased from them.
Meanwhile, here's a complete list of Bloomberg's sources who were willing to speak on the record:
*crickets*
Assuming the article is correct:
1. They were connected to the baseboard management controller (BMC) - so they were basically opening up the IPMI
2. My takeaway would be that you could use small command and control which would be very hard to spot, then make other changes which could exfiltrate only the data you were interested in.
Seriously, I expect that trump will push major changes in the west over this.
He’s already tweeted that affected companies should pick up and move their manufacturing to Russia.
#DeleteChrome
Yes I did, and it doesn't really answer my question, like I said it would have to be sitting on an externally accessible bus, like the Ethernet bus, in order to receive the instructions on what to do. Being able to monitor the operating system loading is next to useless, unless the OS itself is compromised, in which case you have far bigger problems than a 6502 sitting somewhere it shouldn't.
Which is why I asked where exactly it was. Saying it's on the "BMC lines" is... not an answer.
You are not alone. This is not normal. None of this is normal.
How exactly does "slow, negotiated processes" fit with the military occupation of the South China Sea or Tibet?
Those cards turn up on eBay for peanuts, and TFA identifies the location of the chip. It should be possible to get one.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
Yeah, well, nobody should be doing business with the US either, it's not like they don't do stuff like this.
In the US we are more likely to see an employee inform the press if an employer is doing stuff like this. In China, not so much.
And if there is no lawsuit, what does that tell you?
(That was the rhetorical, but here's the answer: Somebody would prefer to keep the details out of a courtroom.)
Yes I did, and it doesn't really answer my question, like I said it would have to be sitting on an externally accessible bus, like the Ethernet bus, in order to receive the instructions on what to do. Being able to monitor the operating system loading is next to useless, unless the OS itself is compromised, in which case you have far bigger problems than a 6502 sitting somewhere it shouldn't.
Which is why I asked where exactly it was. Saying it's on the "BMC lines" is... not an answer.
Do you know what a BMC does? The lines it is sitting on allows it to modify instructions on the CPU. You can actually use those exact same lines to perform remote hardware debugging through the BMC. And by hardware debugging, I mean anything that happens in the board initialization process after SEC finishes. So PEI onward in a UEFI environment. The BMC also has its own connection to the LAN controller(s) on the PCH. It can be used to power on / off, flash firmware over the SPI bus, interact with the server CPU directly, etc.
Buying chips offshore is a national security risk and always has been. If you're stupid enough to think that the Chinese military won't exploit chips/software/tech products bound for the USA for their own benefit, I have a bridge I can sell you.
Of course, as always, profits before country. Can't restrict Northrop Grumman, ya know. And you can bet the current crop of republican technopeasants don't have this on their radar.
Please do not read this sig. Thank you.
Everyone involved on both sides has come out publicly to say Bloomberg is wrong. Why are we still talking about it?
All parties involved have it in their vested interest to deny this.
All parties are required by law to deny this. It's a classified investigation which Bloomberg says is still open. According to Bloomberg's reporting, they don't just want to deny it—they have to deny it. With the Supermicro boards in question in use by the DOD and the CIA, it's quite literally a matter of national security.
They told us we were too dumb to make our own stuff.
Then they told us that people are too expensive to make our own stuff.
Then they told us after automating the factory floor, making labor costs insignificant we have to have a monopoly or we can't compete.
I wonder what their excuse will be now why we can't make our own stuff?
The excuse, which isn't an excuse, is that we don't know how. It's quite literally true. Building a high frequency mainboard correctly is nontrivial, and while we know how to design them, and know how to set up automated tests for them, we don't know all the little tricks that actual manufacturers have learned by doing the job for decades.
Sparkfun has been finding that out, and documenting some of it publicly. They bought a pick and place machine so they could fabricate their own boards for some of the stuff they design. Getting it to work reliably was a journey, and not an easy one. And that's for crappy little $20 low frequency parts that work even on a breadboard, not gigahertz boards worth $1000 before you even drop a CPU onto them.
Somebody will be learning how again. You can bet that now that it's public, the US government acquisitions process will start mandating US assembly for boards it buys for use in classified environments. Somebody will jump on that, because they'll be able to charge a huge premium for a while, since there will be no other option. Monopoly pricing always attracts the US business man.
I'd like to hear about mitigation. Would simply not configuring an IP address on the BMC be enough?
I generally configure whatever kind of BMC I have available on a server (such as HPE iLO or Dell iDRAC) because I like the idea of low-level remote access, but in truth I can't recall ever having used it to solve a problem.
The US, a few years ago, put chips in top end printers, under the assumption that when they were exported that foreign governments would be the typical purchaser. So if you were in Iraq and wondered why that smart bomb picked your chimney it was due to the printer sending the address. Sometimes what goes around comes straight down right at your noggin.
The NSA doesn't have access to most manufacturing plants. Chinese government does. My visit to China to see my friend recently who owns large swatches of buildings with some big name manufacturers, allowed me to waltz in anyone's plant despite "Intellectual Property" (Landlord has some huge privileges in China). Because of his government connections, no one dared question him or me why I was in there taking pictures. No one is going to dare report it happening to the affected companies either that I was in there. In fact, they were concerned more about my safety of anything happening to me than worrying about your IP. Anyone that thinks their data or product design is safe in China are either lying through their teeth or just completely oblivious to reality.
So, possibility of this happening in China to me is highly likely, because every employee there is easily bribed, manipulated or threatened. They could build an R&D lab and additional manufacturing line just for this purpose right in the plant without letting them know. Stuff the NSA could only dream of.
I have a few Supermicro motherboards. How can I check if they are compromised? Is there some audit tool available?
Not everyone. You keep using that word.
"First they came for the slanderers and i said nothing."