Slashdot Mirror


China Infiltrated Apple, Amazon and Other US Companies Using Spy Chips on Servers, According To Bloomberg; Apple, and Amazon, Among Others Refute the Report (bloomberg.com)

Data center equipment run by Amazon Web Services and Apple were subject to surveillance from the Chinese government via a tiny microchip inserted during the equipment manufacturing process, Bloomberg BusinessWeek reported Thursday, citing 17 people at Apple, Amazon, and U.S. government security officials, among others. The compromised chips in question came from a server company called Supermicro that assembled machines used in the centers, the report added. The scrutiny of these chips, which were used for gathering intellectual property and trade secrets from American companies, have also been the subject of an ongoing top secret U.S. government investigation, which started in 2015, the news outlet reported. Amazon, which runs AWS, Apple, and Supermicro have disputed summaries of Bloomberg BusinessWeek's reporting.

The report states that Amazon became aware of a Supermicro's tiny microchip nested on the server motherboards of Elemental Technologies, a Portland, Oregon based company, as part of a due diligence ahead of acquiring the company in 2015. Amazon acquired Elemental as it prepared to use its technologies for what is now known as Prime Video, its video streaming service. The report adds that Amazon informed the FBI of its findings. From the report: One official says investigators found that it eventually affected almost 30 companies, including a major bank, government contractors, and the world's most valuable company, Apple. Apple was an important Supermicro customer and had planned to order more than 30,000 of its servers in two years for a new global network of data centers. Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons. [...] [Update: Some counterpoint: According to an earlier report by The Information, security concerns were indeed a reason why Apple and Supermicro parted ways.] A U.S. official says the government's probe is still examining whether spies were planted inside Supermicro or other American companies to aid the attack. Some background on Supermicro, courtesy of Bloomberg: Today, Supermicro sells more server motherboards than almost anyone else. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order server setups at banks, hedge funds, cloud computing providers, and web-hosting services, among other places. Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards -- its core product -- are nearly all manufactured by contractors in China. The company's pitch to customers hinges on unmatched customization, made possible by hundreds of full-time engineers and a catalog encompassing more than 600 designs. Further reading: Amazon Offloaded Its Chinese Server Business Because it Was Compromised, Report Says.

29 of 369 comments (clear)

  1. Ever get tired of being Wong? by Anonymous Coward · · Score: 4, Insightful

    Chinese market poison as baby food. Nobody should be doing business with them.

    1. Re: Ever get tired of being Wong? by Anonymous Coward · · Score: 3, Informative

      Great news, it was both. Many, many different incidents to choose from. Many different products.

    2. Re: Ever get tired of being Wong? by Jahoda · · Score: 4, Informative

      The Chinese _executed_ quite a few people responsible for that. Say what you will,heads literally rolled. I know you're just here to stir shit up, so you don't care, however.

    3. Re: Ever get tired of being Wong? by tomxor · · Score: 3, Informative

      The Chinese manufacturer had replace glycerine with propylene glycol to save money. Lots of children died.

      I'm no toxicologist but I think you must mean "Diethylene glycol" not "Propylene Glycol"... if you look up the later on wikipedia in the human safety section [1] it states:

      The acute oral toxicity of propylene glycol (E1520) is very low, and large quantities are required to cause perceptible health damage in humans

      Where as Diethylene glycol (which is in the paper you reference at the very start of the toxological analysis section) and the wikipedia article [2] suggests it has high toxicity (albeit only empirically due to involvement in mass poisonings.):

      Despite the discovery of DEG’s toxicity in 1937 and its involvement in mass poisonings around the world, the information available regarding human toxicity is limited. Some authors suggest the minimum toxic dose is estimated at 0.14 mg/kg of body weight and the lethal dose is between 1.0 and 1.63 g/kg of body weight...

      [1] https://en.wikipedia.org/wiki/... [2] https://en.wikipedia.org/wiki/... Anyway it's nasty stuff... however it should be noted that most of these types of events on the Asian continent are more due to lack of strict regulation on food and medicine than malice. Fake medicine is a real problem over there due to the distribution channels, people but stuff in shops with no way to know how authentic it is... and we all know how good the Chinese are at making rip-offs, unfortunately when you swap out expensive components of a medicine without really knowing what you are doing the difference is death rather than a short lived knock-off.

    4. Re: Ever get tired of being Wong? by Anonymous Coward · · Score: 5, Interesting

      Heads rolled after the fact, yes. Mostly to save face, I think, and make a public message of "Look! See, we have laws too!" Every time it happens, it comes off looking more like PR and and an attempt to hobble further investigation. My question is always: what controls are you pitting in place to make sure this doesn't happen again?

      Whether it's adulterated baby formula, or adulterated medicine, or adulterated pork buns, t comes down to someone taking risks to make a fast profit. Plenty of that happens everywhere in the world, but it seems to be in China that the controls are lax enough and the people are desperate enough to actually KILL THEIR CUSTOMERS in order to make money.

  2. Apple and Others Respond by WankerWeasel · · Score: 4, Informative

    Apple and other companies have responded. It would seem Bloomberg has done little to provide any evidence over the past year, while these companies have investigated and found nothing of substance to the claims. Apple's response in particular is strongly worded and makes it clear that they find these claims to be baseless. https://www.bloomberg.com/news...

    1. Re:Apple and Others Respond by Anonymous Coward · · Score: 3, Informative

      Apple also says that they never intentionally slow down old phones and that police can't hack the iPhone.

      Who cares what Apple says? They lie all the time.

    2. Re:Apple and Others Respond by Rick+Zeman · · Score: 4, Insightful

      Sure, I bet it was strongly worded. With all of Apple's production in China, the Chinese could stop every iDevice from being made until Apple restaged manufacturing outside of China. While Apple has the cash reserves to weather the lack of product for over a year while that happens, the decline in market share during that interregnum would be near-fatal, if not fatal.

    3. Re:Apple and Others Respond by harrkev · · Score: 5, Interesting

      As a guy who DESIGNS hardware, I can confidently say this....

      Yes, it is possible to make a tiny chip that can disguise itself as a capacitor or a resistor. However, this part must be designed into the board for that purpose. There is such a thing as a "one wire interface." The part that it is talking to must know it is there and be intentionally taking to it.

      However, adding a chip like this (a two-terminal part as shown in the article) to an existing product not designed for it seems very problematic. I can immediately think of three options for such a ghost part:

      1) Pretends to be a signal filter capacitor. Possible, but it likely would not have the power to actively disrupt the signal flowing past it. This thing would only have access to ONE power rail and can get parasitic power off of the signal. But this kind of part would not have the power to actively disrupt the signal.

      2) Pretends to be a resistor. This is even worse, because usually low-value resistors are used, so the voltage drop would be minimal. I cannot imagine how this part would get its power.

      3) Pretend to be a pull-up or pull-down resistor. This might be useful in mis-configuring a part. It could alter its configuration to get the board into some sort of test mode. The problem is that this configuration would not allow the chip to receive any information from the outside world. So how do you control it?

      Of course, this assumes that the part really is just a two-terminal part (as shown in the article). If they replace an active device, something with three or more pins, then all of those limitations go away. Some sort of level converter in a signal path would be an ideal candidate. If you could drop a chip somewhere in the Ethernet interface path, then you can do anything you want... But those chips would look like chips and could not be mistaken for a passive component.

      --
      "-1 Troll" is the apparently the same as "-1 I disagree with you."
    4. Re:Apple and Others Respond by EvilSS · · Score: 4, Interesting

      So playing devil's advocate here: They could have modified the design, burying the extra traces in interior layers. After approval and the initial production run would you go poking around the boards being shipped out that closely to notice some small extra vias that had been masked over? Would you pull a board apart to view the inner layers if there were no problems? We aren't talking about a rogue employee here but a state sponsored program so you would expect it to have the engineering capability to modify a board design in a way as to not interfere with it's normal functionality. If they compromised the PCB manufacturer and assembly partner they could slip it in any time they wanted to.

      Now I'm not saying I'm buying this story. The very specific, very adamant denials from Amazon isn't the type of denial you would normally expect in a situation like this if they coudln't talk. But it is possible.

      --
      I browse on +1 so AC's need not respond, I won't see it.
  3. And the media blames russia by Kuruk · · Score: 5, Insightful

    China been doing this for years and it's only just coming out.

    1. Re:And the media blames russia by jythie · · Score: 4, Insightful

      On the other hand, knowingly having compromised servers like that would be a PR nightmare, so Apple and Amazon would also have an incentive to say 'everything is fine'. That is what makes stories like this so frustrating... unless the FBI chimes in, everyone is saying pretty much what you would expect to say regardless of if the story is accurate or not.

    2. Re:And the media blames russia by Anonymous Coward · · Score: 5, Informative

      If you read the article it says the chip was tied to the BMC, aka the IPMI implementation.

      So in short, if the machine is on the internet, it's susceptible to having a backdoor through it's own IPMI subsystem. Most legitimate data centers already knew about weaknesses in IPMI and put all the IPMI ports behind a VPN. I can't say the same for those who put bare servers on the internet.

      I'd like to know when this started though, because if it's as true as it sounds (nothing in the article really suggests anything far fetched) then ALL data centers need to be scrubbed. That means large gains for Dell and HP, but at the same time, THEY also make their boards in China as well, so we may in fact find the same kind of tampering on their server boards.

      So take the story with a bit of salt, because if this is really as bad as it sounds, then affected networks should see the spurious traffic on their firewalls (you are running a firewall to your corporate network right?)

  4. Re:Reporting? by Oswald+McWeany · · Score: 5, Insightful

    Everyone involved on both sides has come out publicly to say Bloomberg is wrong. Why are we still talking about it?

    All parties involved have it in their vested interest to deny this.

    --
    "That's the way to do it" - Punch
  5. Re:Function? Position? 6 Pins? by Killall+-9+Bash · · Score: 4, Informative

    If you read the article instead of looking at the pictures, you'd know.

    But I'll be kind to the handicapped today.

    The device interacted with the BMC, which has lowest-level access to everything. The device would use the BMC to inject code into memory, allowing remote exploits, and phone home.

    --
    "Prediction: within 10 years, Windows will be a Linux distribution." Me, 7-6-2016
  6. Back up your claims by Alwin+Henseler · · Score: 5, Insightful

    Where'd the chips come from? They are physical things that exist. Do you think Bloomberg faked the paper trail all the way up the supply chain (..)

    Bloomberg says A, Apple, Amazon etc say B. That's where you need to back up your claim.

    If Bloomberg did its job, it should have some expert(s) on call that can tell you what motherboard, what chip / where on the board, what pinout, what it does, and how they arrived at those findings. That's the core of their story after all.

    If Bloomberg does, just publish those technical details & call it a day. If Bloomberg doesn't, then yes they are talking out of their nose and Apple, Amazon & co have every right to criticize them.

  7. Turn About by Luthair · · Score: 5, Interesting

    Remember when the USA did the same thing?

    1. Re:Turn About by Actually,+I+do+RTFA · · Score: 4, Informative

      Not to electronics sold inside the US. And, since that's where I buy my electronics, that's what I care about.

      Also, you know, I'd rather the US have my data than the Chinese. I'd prefer neither, but between the two, definitely the US.

      --
      Your ad here. Ask me how!
    2. Re:Turn About by Holi · · Score: 4, Insightful

      Why? You live in the US, Te US has a lot more options on ways to misuse your data in ways that could have far more impact on your life. What exactly could China do to you, an American citizen?

      --
      Sorry, teleporters just kill you and then make a copy. A perfect, soul-less copy.
  8. So... by argStyopa · · Score: 5, Insightful

    ...let's hear more from people whinging about Trump's 'trade war' with China.

    China's been a shitty actor on the world stage since they bred themselves out of irrelevancy.

    Foreign companies have to establish a Chinese business, owned 51% by Chinese who almost always end up being a front for the PLA.
    Draconian censorship laws. No free speech. No freedom of religion.
    Currency manipulation and disregard for norms of international economic (and other) reporting.
    Military occupation and absorption of neighbors it deems "were *actually* China anyway".
    Sorry Hong Kongers, I guess you don't get to keep democracy and nobody cares...
    An arbitrary, dangerously confrontational foreign policy including sweeping territorial claims.
    Environmental destruction with impunity. ...and yet we should curry their favor so we can keep buying $9 folding chairs?

    I don't like Donald Trump for a number of reasons, but the US confrontation with China is LONG past due; waiting any longer would likely make it military when China finally gets brazen enough to try to grab Taiwan.

    --
    -Styopa
  9. The real problem... by Austerity+Empowers · · Score: 5, Insightful

    I don't know to what degree "China" (it's government, it's people, or it's corporations, state owned or otherwise) are spying, but I do know it's not 0, not even close to 0. I have been close to accusations and convictions, they are absolutely spying using any available means. That's not surprising. If it made any sense to do it, adding stray hardware/software to a PC is definitely a viable approach to compromising it.

    The real issue is technical. How do we create a secure compute environment? Apple has taken the route on its phones of building a very effective and secure trust chain. It is pretty hard for an unauthorized user to slip in stray firmware on their phones, I don't want to say impossible because there are some known and pretty exotic exploits. But very hard. Their design is such that even their MFGs cannot sneak in stray code to spy on you. The weakest point is still the single authorized user, and their ability to protect their passwords and biometrics. Apple's route also makes you, the owner, a perpetual customer rather than an owner. If they choose to lock you out, there's nothing you can do about it, your $1k phone is a paperweight.

    PCs (I'm including desktops, servers and laptops) on the other hand are pretty much a free for all. The MFG can sneak on just about anything in their BIOS/EFI implementation, and anyone up and down the chain can do so without much oversight. It's a pretty open and competitive market, with many small players of little to no account, all trying to make the sale. Each of them provides their own hardware, and some EFI implementation they probably bought and then tailored to their implementation. Someone could also have added backdoors. That in turn hands off to my choice of OSes, which themselves could easily be compromised and I wouldn't know better until something happened. I am unquestionably the owner of this system, and can do anything I would like, but I also cannot rely on anything up and down the system. I'm the owner of a very leaky boat.

    What we need is a system that can both be trustworthy and robust to middle-man attackers who may, at times, have direct hardware access, but still allows me to be the absolute owner of my hardware. I may make bad choices, those bad choices may compromise my system, but I need a foolproof way of knowing when I'm making a bad choice. It's not that easy of a problem in the current ecosystem, and we're waiting for someone to get caught doing something bad that forces our hand.

    1. Re:The real problem... by Anonymous Coward · · Score: 3, Informative

      Can't even do that - the first thing you do when the iPhone turns on is agree to a clickwrap license where you give up your right to sue and agree to binding arbitration with an arbiter of Apple's choosing. This same agreement also lets Apple remotely brick your phone with no recourse.

  10. Re:Stolen data has to be transmitted by jittles · · Score: 3, Informative

    This is only a small part of the issues I have about the report. What is the chip monitoring or able to monitor? How is it programmed?

    It's not impossible to envisage something that, say, could monitor Ethernet for a string and use that to program itself, but something that can both see an incoming Ethernet packet and see what the CPU is doing is harder to conceptualize.

    I know this is Slashdot but... did you read the article? Supposedly this chip was put on the BMC lines that allow it to modify basically anything going to the CPU. They could have even tweaked the firmware on the board through the BMC. The chip does nothing but detect the loading of the OS and insert instructions that it downloads off of a known host. There was no data exfiltrated as far as anyone can tell. It was just lying dormant or used as a vector to penetrate other areas of the network. They were able to identify the 30 companies affected by monitoring traffic and/or hacking the C&C server used. But it was not detected because, as far as they can tell, the compromised systems themselves were never used to exfiltrate data.

  11. What's that line about truth lacing its shoes? by Anubis+IV · · Score: 5, Interesting

    Bloomberg published responses from the companies involved. Here are some excerpts that give you a sense of how they responded...

    Amazon:

    It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental. It’s also untrue that AWS knew about servers containing malicious chips or modifications in data centers based in China, or that AWS worked with the FBI to investigate or provide data about malicious hardware. [...]

    And they go on to say a lot more that categorically denies Bloomberg's claims while making a mention of an unrelated firmware incident from 2016.

    Apple:

    Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.

    On this we can be very clear: Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement. [...]

    And they go on to say a lot more that categorically denies Bloomberg's claims while suggesting that Bloomberg may be confused about the 2016 firmware incident.

    Super Micro:

    While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard. We are not aware of any customer dropping Supermicro as a supplier for this type of issue.

    And they go on to say a lot more that categorically denies Bloomberg's claims, including denying that they even make the chips that were allegedly compromised and that these companies supposedly purchased from them.

    Meanwhile, here's a complete list of Bloomberg's sources who were willing to speak on the record:

    *crickets*

    1. Re: What's that line about truth lacing its shoes? by Anonymous Coward · · Score: 4, Funny

      Just remember everyone: cloud computing and giving large corporations all of everyone's secret business data in one place is totally secure.

      Keep repeating until you start to believe it.

  12. Re:Bloomberg's Banned Since I Arrived by Whorhay · · Score: 3, Insightful

    How exactly does "slow, negotiated processes" fit with the military occupation of the South China Sea or Tibet?

  13. Re:Would you trust the FBI? by AmiMoJo · · Score: 4, Insightful

    Those cards turn up on eBay for peanuts, and TFA identifies the location of the chip. It should be possible to get one.

    --
    const int one = 65536; (Silvermoon, Texture.cs)
    SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
  14. Offshore chips are an *obvious* security risk. by gestalt_n_pepper · · Score: 4, Insightful

    Buying chips offshore is a national security risk and always has been. If you're stupid enough to think that the Chinese military won't exploit chips/software/tech products bound for the USA for their own benefit, I have a bridge I can sell you.

    Of course, as always, profits before country. Can't restrict Northrop Grumman, ya know. And you can bet the current crop of republican technopeasants don't have this on their radar.

    --
    Please do not read this sig. Thank you.
  15. Re:Reporting? by Areyoukiddingme · · Score: 4, Insightful

    Everyone involved on both sides has come out publicly to say Bloomberg is wrong. Why are we still talking about it?

    All parties involved have it in their vested interest to deny this.

    All parties are required by law to deny this. It's a classified investigation which Bloomberg says is still open. According to Bloomberg's reporting, they don't just want to deny it—they have to deny it. With the Supermicro boards in question in use by the DOD and the CIA, it's quite literally a matter of national security.