Slashdot Mirror

← Back to Stories (view on slashdot.org)

New Evidence of Hacked Supermicro Hardware Found in US Telecom: Bloomberg (bloomberg.com)

Posted by msmash on from the doubling-down dept.

A major U.S. telecommunications company discovered manipulated hardware from Super Micro Computer in its network and removed it in August, fresh evidence of tampering in China of critical technology components bound for the U.S., Bloomberg reported Tuesday. From the report: The security expert, Yossi Appleboum, provided documents, analysis and other evidence of the discovery following the publication of an investigative report in Bloomberg Businessweek that detailed how China's intelligence services had ordered subcontractors to plant malicious chips in Supermicro server motherboards over a two-year period ending in 2015. Appleboum previously worked in the technology unit of the Israeli Army Intelligence Corps and is now co-chief executive officer of Sepio Systems in Gaithersburg, Maryland. His firm specializes in hardware security and was hired to scan several large data centers belonging to the telecommunications company. Bloomberg is not identifying the company due to Appleboum's nondisclosure agreement with the client. Unusual communications from a Supermicro server and a subsequent physical inspection revealed an implant built into the server's Ethernet connector, a component that's used to attach network cables to the computer, Appleboum said.

36 of 191 comments (clear)

  1. Bloomberg! Bloomberg! Bloomberg! by The+Original+CDR · · Score: 5, Interesting

    Has any other news media outfit independently verified the Bloomberg claims?

    1. Re:Bloomberg! Bloomberg! Bloomberg! by Sarten-X · · Score: 3, Insightful

      Nope, nor have enough details been released that somebody could even start. There's speculation, but Bloomberg hasn't published anything that would let someone verify it on their own.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    2. Re:Bloomberg! Bloomberg! Bloomberg! by Calydor · · Score: 3, Insightful

      Does Bloomberg?

      --
      -=This sig has nothing to do with my comment. Move along now=-
    3. Re: Bloomberg! Bloomberg! Bloomberg! by Anonymous Coward · · Score: 4, Interesting

      but I also wouldn't put it past Bloomberg to publish rumors for page hits.

      The Supermicro story is turning out to be a hoax.

      The only person actually named in the original Bloomberg story about the Supermicro servers was a "hardware expert" named Joe Fitzpatrick. As it turns out, he' s not all that much of an expert, and he has now done an interview where he says that he doubts the accuracy of the story:

      https://risky.biz/RB517_featur...

      He was communicating with one of the authors of the Bloomberg story for a couple of months before the story was published. Then, the story came out and things that he had described as being hypothetically possible were in the story, but presented as facts that they had gotten from various anonymous sources

      For example, the Bloomberg guy said to him "One of my sources said the chip might be a signal coupler. What does that look like?" So Joe Fitzpatrick sent him a link to a picture from a catalog. And, lo and behold, when the story was published it contained that exact picture, presented as "proof" of the chip that was implanted on the Supermicro motherboards.

    4. Re:Bloomberg! Bloomberg! Bloomberg! by rudy_wayne · · Score: 5, Interesting

      The authors of this most recent story were also the author of the original Supermicro story. They also wrote other pieces over the last couple of years were they have made lots of spectacular claims, with little or no evidence, and, there has never been any follow-up on the stories.

    5. Re:Bloomberg! Bloomberg! Bloomberg! by Sarten-X · · Score: 5, Insightful

      The public deserves the truth.

      Security is complicated. On the one hand, perfect security is impossible. Your servers can be hacked, your data can be stolen, and your users can be phished.

      However, there is another perspective that I think is equally important, if not moreso: It's not hopeless. The attackers are not omnipotent. They have 9-5 schedules, bureaucracies, budgets, and deadlines. If your system is protected well enough that your attackers' budget runs out, it will stay safe. From that perspective, security is just a matter of economics. Your security is bought by spending a little money and effort to drastically increase the effort the attackers need to spend.

      An attacker embedding a custom chip in server hardware, then processing thousands of phone-home results is expensive for them, and unlikely to get a result. However, replacing your whole data center to use non-Supermicro servers is also expensive. Frankly, the whole thing probably isn't worth anybody's time.

      Breaking into an internet-facing server with a default password is easy. There are lots of routers and firewalls out there with default credentials or hidden backdoor accounts. Exploiting one of those is ridiculously cheap for an attacker, and gets them far better results.

      The notion of "the attacker is almighty" doesn't help improve overall security, because it silences discussion about how to actually improve security posture. Instead, we should set aside hardware concerns for now, and ask "What's the easiest way we can be attacked, and how can we fix it?", then make the fix, and repeat until your own budget runs out.

      My skepticism is not about doubting China's ability. I'm sure China (or any nation or well-funded individual) could get hardware inserted into servers. What I'm skeptical of is whether China (or any nation or well-funded individual) would even bother with the expense and risk when they could send a phishing campaign instead.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    6. Re: Bloomberg! Bloomberg! Bloomberg! by MachineShedFred · · Score: 5, Interesting

      If there were supposedly thousands of these things sold to various customers all over the place, how is it that nobody kept one for forensic analysis?

      How is there not one live example if all these networks and servers were compromised?

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    7. Re: Bloomberg! Bloomberg! Bloomberg! by Anonymous Coward · · Score: 2

      the idea of bloomberg publishing sensational garbage is even easierto believe

    8. Re:Bloomberg! Bloomberg! Bloomberg! by infolation · · Score: 4, Interesting

      First the authors shorted supermicro stock ahead of the original claims, then they used the profits from that short to pull an even bigger leveraged short of supermicro stock ahead of the second batch of claims.

      I have no evidence of this but... if you were writing those stories, why wouldn't you?

      2018/10/04 US:SMCI $21.47 --> $8.55
      2018/10/09 US:SMCI $15.55 --> $10.80

    9. Re:Bloomberg! Bloomberg! Bloomberg! by Aighearach · · Score: 5, Insightful

      Does anybody think the Chinese government deserves the benefit of the doubt?

      Does Bloomberg?

      Yes. Bloomberg is a center-right media outlet, and almost all of their profitable business is related to selling financial information to professionals. They make an industry-leading software product called Bloomberg Terminal that they use to disseminate this information.

      I wouldn't trust them on political reporting, because they tend to give the perspective of a center-right business executive. But on general news that doesn't relate to their industry, they are nothing if not mainstream. They don't go for bombastic tabloid nonsense, it would tarnish their brand. Getting page views isn't the purpose of their public news service; enhancing their brand is the purpose.

      Therefore, I would give Bloomberg the benefit of the doubt that they believe this information to be true, and to be of great import to purchasing and IT managers, in addition to investors and financial services providers. This is big enough that the insurance community is probably taking a lot of interest, too. They would never intentionally publish a false report that purported to be of great interest to the industries where they make their bread-and-butter; it would be all downside for them.

      https://www.bloomberg.com/comp...
      Don't worry about the PR there, just look at the bottom of the page under "Products" and "Industry Products" and you can understand why they are a trusted source on this; they'd lose a lot by being wrong. And they have a lot to lose.

    10. Re:Bloomberg! Bloomberg! Bloomberg! by Sarten-X · · Score: 2

      On the contrary... I've been a government contractor, and money was often an issue, though mostly it was in terms of ROI more than actual dollars. Governments don't mind spending a lot of money as long as they know they're getting what they asked for.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    11. Re:Bloomberg! Bloomberg! Bloomberg! by ole_timer · · Score: 2

      exactly! phishing is nearly free, but we have to spend all this money - viola, hardware exploits, buying software zero days, etc. the ends (i. e., the data) justifies all means to get it. money is not the issue.

      --
      nothing to see here - move along
    12. Re: Bloomberg! Bloomberg! Bloomberg! by Junta · · Score: 3, Interesting

      It does certainly sound that the reporters behind the story are not particularly good at understanding the information they get, or else vetting their sources...

      The first story appears to be cobbled together out of misunderstandnigs spread across many sources (the number of sources then used to declare how valid it must be. Of course one of those 'sources' has come forth and said one source used a hypothetical and his role in corrobariting it was to include a picture of what a signal coupler is, showing how dodgy the story was assembled.

      This time, it's at least more straightforward, one named source with a more straightforward and more credible strategy. However it is entirely possible that the guy doesn't know what a BMC is and mistakes the errant traffic from a BMC trying to DHCP or somethnig as an overtly malicious thing. He may not recognize some component of the jack or phy or noted the NCSI lines from NIC to another chip and presumed that was snooping.

      Now it's one thing to put this out there for further investigation to get clarity, but the stories are emphatic and unambiguously making accusations which is causing the general tech market stock to move by billions of dollars and for customers to take the headlines at face value and decide things (moving from one company that was 'more chinese' than they realized to an american company with the same supply chain issues in all likelihood, even vendors making systems elsewhere generally ship circuitboards out of China). This could end up in a big defamation suit by many parties in the tech industry.

      --
      XML is like violence. If it doesn't solve the problem, use more.
    13. Re:Bloomberg! Bloomberg! Bloomberg! by barc0001 · · Score: 4, Interesting

      > how they could siphon off gigabytes of data and ship it to China, presumably over network connections and through firewalls,
      That's the interesting thing to me as well. If your network and firewalls are properly designed, it shouldn't matter if your servers have a rogue little chip wanting to call home - your network should shitcan any attempt regardless.

    14. Re:Bloomberg! Bloomberg! Bloomberg! by rtb61 · · Score: 3, Interesting

      Why would it be in any way shape or form outrageous, they would be logical and expected. The best supply line hack, high efficiency capacitors are smaller than low efficiency capacitors (much more expensive as well). So you can put a high efficiency capacitor in a low efficiency capacitor casing and have room left for a chip. What the chip does is check from a signal on the power flow, if it gets the right code, it shorts out the connection and the capacitor dies, taking out what ever product ie a computer it is in. So done on a broad scale you can collapse a country. The chaos, everything with that capacitor goes down, replacement computer motherboards are really difficult to order because the computers down. Get the motherboard, and if the signal is still going, once it is powered it fails.

      Get enough tainted capacitors into the infrastructure and that country goes down for months, everything goes down, it basically just lost a war it never knew it was fighting, would I trust capacitors out of China or the US in essential infrastructure, absolutely not, I would strive for all infrastructure components to be locally made. If I was supplying them to another country, I would bobby trap it all, just in case, so much fucking cheaper than a war machine and far more effective. Not happy with the supply of my tainted components, make your fucking own, or buy some other countries tainted components. US tech industries are so screwed and the US government did it to them.

      --
      Chaos - everything, everywhere, everywhen
    15. Re: Bloomberg! Bloomberg! Bloomberg! by Anonymous Coward · · Score: 2, Interesting

      The presence or absence of a defamation lawsuit will clarify the truth. Plus a defamation lawsuit would require the companies filing the suit to prove the accusations are false and allow outsiders access to the technology under review for independent analysis. And China seems to be pretty quite about the whole affair since a story like this will further limit their ability to sell their technology in the US. Maybe China is afraid to say anything because they are afraid of giving Trump a bigger bat to whack them upside their head with. Not to mention the US government are reactionary morons who have already enacted regulations against both Chinese and Russian technology companies. And Congress doesn't require any proof when China or Russia is involved. The only thing the two political parties in the US agree on is giving China and Russia a hard time. However, in this instance there is really is no reason not to stick it to the Chinese. It's not like the US cannot manufacture or get similar technology from others. SE Asia is full of countries who can undercut China when it comes to taken advantage of low labor costs.

    16. Re:Bloomberg! Bloomberg! Bloomberg! by Graymalkin · · Score: 2

      An attacker only needs to get through the defense once (for various values of once) to be successful while the defender needs to block every attack to be successful.

      A little spy chip spitting out a DNS query for an innocuous looking domain to exfiltrate data and/or grab C&C instructions from a TXT record or something might never be noticed.

      --
      I'm a loner Dottie, a Rebel.
  2. Where? by 110010001000 · · Score: 5, Interesting

    Where is the evidence? They keep saying they have it. Why don't they show it?

    1. Re:Where? by Aighearach · · Score: 4, Interesting

      Investigative reporting doesn't work that way in most cases. There are a lot of unknowns. Right now, they enhance their own research by not giving out too many details, and letting the companies involved say stupid things that might be refutable by that evidence.

      Evidence is good. Don't decide if it is actually true or not until you get it. But that doesn't imply that when you first hear about the issue, the evidence will be published, or that it is tactically wise to lead with the evidence instead of the accusation.

      If we get to the end of the story and Bloomberg says "that's all we have," that's when you can weigh the evidence they presented. If they haven't presented the evidence yet, then before you start to worry about that, you should simply check if the process has reached the end, of if the evidence is still waiting to be released. If it is still waiting to be released, there is nothing suspicious at all about the fact that you have not been given a personal viewing.

  3. Plenty of evendince of this is real by supercell · · Score: 5, Informative
    I had SMCI stock in 2017 and sold it after reports that Apple dropped them when they found serious security issues with their servers.
    Now Apple and others claim they have no idea what Bloomberg is talking about. Clearly something was installed on Supermicro servers to cause Apple and others to stop using them.

    Report from early 2017
    https://www.marketwatch.com/st...

    1. Re:Plenty of evendince of this is real by mujadaddy · · Score: 4, Interesting

      Correct: Bloomberg's reporting is lagging real events, but Apple & Amazon haven't come up with a better explanation of why they switched hardware at that time.

      --
      Populus vult decipi, ergo decipiatur...
      "Force shits upon Reason's back." - Poor Richard's Almanac
    2. Re:Plenty of evendince of this is real by Aighearach · · Score: 2

      And yet, they do offer an explanation they claim is not only better, but true! That it was only a software issue. They said that in response to the first Bloomberg story. So now Bloomberg is doubling-down on that part.

    3. Re:Plenty of evendince of this is real by Anubis+IV · · Score: 3, Interesting

      I had SMCI stock in 2017 and sold it after reports that Apple dropped them when they found serious security issues with their servers.

      Going by that, the timeline would be that these companies discovered malicious hardware in 2015, kept thousands of those servers in service for two or more years, and only then decommissioned them. Does that make any sense at all?

      Instead, if you read their initial responses to what Bloomberg published, they actually say more than that "they have no idea what Bloomberg is talking about". For instance, Apple provides an alternative explanation for Bloomberg's confusion:

      [...] Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.

      Apple dropped SuperMicro shortly after that incident, making it a much more likely cause for the falling out. Likewise, Amazon cites firmware issues with SuperMicro boards in their response, though you'll note that they were still using SuperMicro boards in 2018:

      Additionally, in June 2018, researchers made public reports of vulnerabilities in SuperMicro firmware. As part of our standard operating procedure, we notified affected customers promptly, and recommended they upgrade the firmware in their appliances.

      All of which is to say, nothing about Bloomberg's story makes any sense. The timeline makes no sense, none of the alleged victims has anything to gain by lying, one of their only named sources has come out saying he doubts the story, literally every company or agency allegedly involved has said it's untrue, and Apple has even gone so far as to formally inform Congress that inasmuch as the story pertains to them it's untrue, while additionally affirming via press releases that they are not under a gag order or anything else of the sort.

      Someone's credibility is going to take a nosedive after the dust settles from this, and I expect that it will be Bloomberg's.

  4. I have a load of SuperMicro gear by guruevi · · Score: 3, Interesting

    Also from that era that they say. I haven't seen anything anomalous. The fact is that some of their IPMI stuff is vulnerable and they're not updating the firmware (eg. old versions of Dropbear SSH), so if you leave it on the Internet, it may get compromised.

    On the other hand, I also don't leave that stuff on a routable VLAN. If it tries to connect to anything (and I haven't seen it reach out), I'd notice and it wouldn't work anyway. Sure the IPMI has some hooks into the rest of the hardware so it is potentially capable of doing 'weird stuff' to my Linux or Windows kernels (although it'd have to be pretty smart to intercept keyboard authentication, wait for someone to be away from the keyboard, automatically replay credentials, then load a workable kernel module to do that) and have the OS compromised do the dirty work, but then again, I haven't seen anything there either and we've used various integrity and antivirus systems from TripWire, Sophos and Cylance that probably would've noticed.

    --
    Custom electronics and digital signage for your business: www.evcircuits.com
    1. Re:I have a load of SuperMicro gear by ole_timer · · Score: 4, Insightful

      you have no ip worth stealing...why would they go after you?

      --
      nothing to see here - move along
    2. Re:I have a load of SuperMicro gear by ArchieBunker · · Score: 2

      Who in the hell exposes their management consoles to the outside world?

      --
      Only the State obtains its revenue by coercion. - Murray Rothbard
  5. Might not be just Supermicro by caffeinejolt · · Score: 5, Interesting
    The article states:

    The executive said he has seen similar manipulations of different vendors' computer hardware made by contractors in China, not just products from Supermicro. “Supermicro is a victim -- so is everyone else,” he said. Appleboum said his concern is that there are countless points in the supply chain in China where manipulations can be introduced, and deducing them can in many cases be impossible. “That's the problem with the Chinese supply chain,” he said.

    According to the original article - the alleged Chinese culprit chip exploited via the BMC. Aspeed is the company that makes 99% of the BMC controllers in Supermicro boards. If China really did go through the trouble to develop a chip to exploit via Aspeed controllers.... why limit themselves to Supermicro? I know at least Tyan and Lenovo also use Aspeed. From China's intelligence perspective, they would want a solution that could work across multiple board vendors.

    According to latest:

    Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer. "The module looks really innocent, high quality and 'original' but it was added as part of a supply chain attack," he said.

    Really wish they would give us more to go on than just that. Not sure about other Slashdotters, but I have Tyan/Supermicro/Insert-Taiwanese-Motherboard-Manufacturer boards in production, and would really appreciate more information on what to look for.

    1. Re:Might not be just Supermicro by TechyImmigrant · · Score: 2

      >Really wish they would give us more to go on than just that. Not sure about other Slashdotters, but I have Tyan/Supermicro/Insert-Taiwanese-Motherboard-Manufacturer boards in production, and would really appreciate more information on what to look for.

      It doesn't add up.

      Why would you put your trojan chip in the ethernet connector? It's away from the signals you want to get to - serial and/or usb wires to get at UEFI. Trying to cram and ethernet stack, phy and the drivers into a tiny package to get at a much harder but to crack is not how you would do it. It's also inconsistent with the first story.

      --
      I should use this sig to advertise my book ISBN-13 : 978-1501515132.
    2. Re:Might not be just Supermicro by jeff4747 · · Score: 2

      Why would you put your trojan chip in the ethernet connector? It's away from the signals you want to get to - serial and/or usb wires to get at UEFI

      Option 1: There's no requirement that there be no connection from this chip to another part of the computer. If you're planting rogue hardware in one thing there's little reason to believe you can't plant rogue hardware elsewhere.

      Option 2: PXE boot attack.

      Option 3: Their desired attack vector only uses this additional chip, the chip itself isn't the attack vector. After all, where do you get that BIOS image? Or the firmware on the rest of the motherboard's chips? Or perhaps you need the server to receive a malformed packet to implement an exploit of some other process, but that malformed packet can't survive routing.

      Option 4: This isn't for spying. Cut off all packets or scramble some packets and you've effectively disabled the server.

  6. #gifs by Anonymous Coward · · Score: 2, Funny

    Pics or it didn't happen.

  7. Completely incredible... by rthille · · Score: 5, Informative

    "Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones."

    Take a look at a google image search for "motherboard" and see if you can find an RJ-45 socket that doesn't have a metal shield around it for RF blocking.

    --
    Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
  8. Re:time to make stuff in the USA! by Sarten-X · · Score: 2

    Put in a tightly-configured firewall so your data doesn't get sent anywhere without your approval.

    Keep management systems isolated so the data-holding servers can't modify that firewall.

    Don't rely on tightly-integrated single-source solutions, so one vendor being compromised won't leave that firewall ineffectual.

    Maintain independent layers of security that protect in case of another layer's failure.

    That's defense in depth.

    --
    You do not have a moral or legal right to do absolutely anything you want.
  9. The CEO says keep it secret. by emil · · Score: 3, Interesting

    Do you think that your corporate security team wants to admit that you were infiltrated?

    The first dozen companies that admit this will likely see their stock price decline. Do you want your company to go first?

    1. Re:The CEO says keep it secret. by MachineShedFred · · Score: 2

      Yeah good plan, because it always goes better when they try to cover it up and instead get hauled in front of Congress, testifying under oath.

      What does that do to the stock price?

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
  10. Re:Silly Chinese by Ungrounded+Lightning · · Score: 2

    Why put the chip on the Ethernet connector? You know this doesn't decrypt encrypted traffic.

    To give it the ability to exchange command-and-control traffic with a remote controller while keeping it from the rest of the system (by "eating" the incoming packets for itself without handing them to the processor's stack, and sending outbound packets directly, again without processing them through the rest of the system.)

    This is both convenient, and lets the C&C communicate with the victim box even when the bulk of the victim is shut down.

    The Ethernet controller has lots of processing power to play with once it's subverted, control-channel access to the board management system, and already has power-when-the-system-is-down specifically so it can hear the wake-on-LAN packets and bring the machine up to full function - one less mechanism to build.

    That's exactly what Intel did when they first started doing Management engines. It was only later versions where they moved it in deeper.

    --
    Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
  11. that's directly from NSA's playbook by sxpert · · Score: 2

    implant in ethernet connector point to NSA's ANT catalog,
    either "COTTONMOUTHIII" https://nsa.gov1.info/dni/nsa-...
    or "FIREWALK" https://nsa.gov1.info/dni/nsa-...