Ask Slashdot: How Does One Verify Hard Drive Firmware?

An anonymous reader writes: In light of recent revelations from Kaspersky Labs about the Equation Group and persistent hard drive malware, I was curious about how easy it might be to verify my own system's drives to see if they were infected. I have no real reason to think they would be, but I was dismayed by the total lack of tools to independently verify such a thing. For instance, Seagate's firmware download pages provide files with no external hash, something Linux distributions do for all of their packages. Neither do they seem to provide a utility to read off the current firmware from a drive and verify its integrity.

Are there any utilities to do such a thing? Why don't these companies provide verification software to users? Has anyone compiled and posted a public list of known-good firmware hashes for the major hard drive vendors and models? This seems to be a critical hole in PC security. I did contact Seagate support asking for hashes of their latest firmware; I got a response stating, "...If you download the firmware directly from our website there is no risk on the file be tampered with." (Their phrasing, not mine.) Methinks somebody hasn't been keeping up with world events lately.

Comments

how ?

[itzly]

This is pointless without JTAG hardware to directly access the flash memory.

Normal users would read/update the firmware through the existing firmware, so if that's been tampered with there's no way you can be sure.

Re:how ?

[storkus]

And how many lay people even know what JTAG is?

You, the individual, can't hope to keep up with organizations that can out-spend you hundreds to thousands of times in terms both man-hours and money. How can you even know if the code you download off the manufacturers' web sites hasn't been tainted during production? Your only hope is to stay below their radar, or have enough trusted people around you or time on your hands to personally go through the code and verify it. I'm betting, even in their mom's basement, hardly anyone has time for that.

Re:how ?

[Anonymous Coward]

And how many lay people even know what JTAG is?

This doesn't matter as much as you think. If people who know what JTAG is each verify a few random devices they have access to and find nothing then we know that the majority of devices are okay. The whole general infrastructure is more or less sound. On the other hand if they find a few examples, then there will be proof that there's a problem and there can be a serious attempt to replace everything because there's a real demonstrated problem.

Re:how ?

[twitnutttt]

Of course, reading the memory from the computer you booted the hard drive from means you are potentially running a compromised machine if the hard drive is compromised.

But if you booted a different, known-good machine, then mounted the hard drive in question as a secondary drive, it seems feasible you should be able to read and verify the firmware.

Seagate's response here seems ridiculously out of touch, and I can only hope that their posture on this will adapt quickly as the news and newfound scrutiny of the hard drive firmware layer trickle through the organization's practices.

Re:how ?

[earthminion]

@AC, "some new trend" and "someone goes on blabbing about JTAG" ... No its not a new trend, people like you who don't really know about JTAG, have always trolled down comments they don't like (oh the irony). The fact is JTAG is about as near to a standard as we have with embedded CPUs and all developers of embedded systems know about and use JTAG. Also how do you think the initial bootloader firmware gets on the embedded systems during manufacturing? ... I'll give you a clue, JTAG.

Re:how ?

[Anonymous Coward]

No. The hard drive itself is a computer and is compromised. It doesn't matter if you boot off of it or connect it later. You can't trust anything coming out of the main interface.

Many hard drives have a secondary TTL level serial port that you can use to load new firmware on a bricked drive. It's possible that the serial connection is wired in such a way to be safe from corrupt firmware. So it might be possible to recover a compromised hard drive that way, but I wouldn't trust it without a lot more information about the serial port and how it works than is publicly available.

Re:how ?

[wvmarle]

How can you even know if the code you download off the manufacturers' web sites hasn't been tainted during production?

You can't, but you can be quite sure that the manufacturer will take serious measures to make sure this doesn't happen. This protection against tampering to compromise computers just piggybacks on more general protections to keep firmware sound, such as tests to make sure there are no bugs in the firmware that cause data loss, and that software published on the web site is the software the company intends to publish.

This for the simple reason that one mistake here may result in bankruptcy, as people may lose trust in the whole company. Without trust in its products by its customers, a company can't survive - especially when it's about storing valuable data.

Re:how ?

[wvmarle]

Copying some data is quite different from replacing data, and far easier to do unnoticed. The NSA copied existing SIM encryption keys; they did not attempt to replace them with their own keys or so.

It is pretty hard to detect an intrusion, access to data, and copying of that data. Especially if the attacker gets access through an authorised account by getting their hands on someone's login credentials.

It is much easier to detect the replacement of data: this can be done with e.g. automated cryptographic checksum tests against remotely stored known good checksums, or against a freshly compiled copy.

A lot of data will have to be replaced unnoticed (source code is being read by humans, who may detect changes if it happens to be the part they work with) to stand any chance of getting a compromised binary on someone else's site unnoticed.

Re:how ?

[ShanghaiBill]

But if you booted a different, known-good machine, then mounted the hard drive in question as a secondary drive, it seems feasible you should be able to read and verify the firmware.

No, you would be going through the processor on the HDD, that is running the supposedly compromised firmware. There would be nothing to stop it from lying to you about updating itself. Firmware malware would most likely be implemented as a stub that checks for a special key like "NSA_1234", and otherwise jumps to the "real" firmware, so there would be no way to test for its presence without knowing the key. They only way to be sure, would be to write directly to the flash via the JTAG port.

Moving the HDD to a different computer would make no difference, since the firmware is in the drive.

Re:how ?

[AK Marc]

You can't, but you can be quite sure that the manufacturer will take serious measures to make sure this doesn't happen.

I worked for HP when every computer re-imaged in their repair shop was sent out with a virus. Other makers have essentially recalled new PCs for the same thing. I can't give cites, because it's covered up. 10x the budget of the repair department was spent to hide the problem. To convince people it can't happen. Must trust the chain of trust. Even when it's broken.

Re:how ?

[ArmoredDragon]

Off hand I can already think of a technique I myself have deployed (not written by me) when hacking DirecTV smart cards, or what I know Xbox 360 mod chip users do: Use "stealthing" code that presents the data that is SUPPOSED to be there when asked by any existing commands that are used to read the firmware contents, but otherwise the hacked code is what gets executed during normal running operation.

It wouldn't surprise me if whoever wrote this went to such lengths to add this kind of feature to their firmware. I mean look how excruciatingly feature packed stuxnet was.

Re:how ?

[ckatko]

>But if you booted a different, known-good machine, then mounted the hard drive in question as a secondary drive, it seems feasible you should be able to read and verify the firmware.

No... no, you're missing it. The firmware isn't some magical OS. The firmware runs whenever it's connected. Not only when its booted from. Do you know what firmware is?

The firmware handles all requests. So clearly, you are requesting data from something that's tampered with, to see if it's tampered with. It's entirely possible that to make their firmware harder to catch, the firmware would only give you the "false" previous firmware data as you talk to it. Given the complexity of all of their groups viruses we've seen so far, and the fact they compress their payloads, this is not far fetched at all.

I mean, have you ever even used a microcontroller before? How do you think data gets off your hard drive?

Re:how ?

[wvmarle]

As many already pointed out: you can not trust the firmware image provided by the drive itself, for the simple reason that you have to talk to the very firmware you try to verify, and which may be compromised.

Think of the kid calling "are there any monsters under the bed?", and the monster under the bed answering "no!".

Re:how ?

[rs79]

I used to write hard drive firmware.

Here's what I would do: pull the controller board off a drive. Write a piece of software that emulates the physical drive to take it out of the equation.

You have to find a way to read the firmware. If you have to desolder a chop ad read it that's ok. Do this to many drives and eventually you'l find one of these that isn't like the other.

Now figure out what it's doing.
The firmware on those drives is not in the slightest bit complicated it's actually very straightforward.

Old SCSI drives circa late 80s is where I'd start looking.

Re:how ?

[Cramer]

Even with JTAG (hint: most drives don't have a JTAG header), all you'd get is the tiny bootloader. Almost everyone stores the full firmware on the plater. (because a 32K flash is cheaper than 8M)

(NOTE: This is a **MAJOR** cause of drive failure. The protected sections where the firmware is stored is never "refreshed"; when it becomes too weak to read, the drive is a paperweight.)

Hashes not useful

[IamTheRealMike]

Seagate is correct. Putting a hash on the website doesn't improve security at all because anyone who can change the download can also change the web page containing the hash.

  The fact that this practice is widespread in the Linux world originates from the usage of insecure FTP mirrors run by volunteer admins. There it's possible for a mirror to get hacked independently of the origin web page. A company like Seagate doesn't rely on volunteers at universities to distribute their binaries so the technique is pointless.

A tool to verify the firmware is poetically impossible to write. What code on the drive would provide the firmware in response to a tool query? Oh right ..... the firmware itself. To make it work you need an unflashable boot loader that acts as a root of trust and was designed to do this from the start. But such a thing is basically pointless unless you're trying to detect firmware reflashing malware and that's something that only cropped up as a threat very recently. So I doubt any hard disk has it.

BTW call a spade a spade. Equation Group == NSA TAO

Re:Hashes not useful

[itzly]

What they need to do is put a bootloader in there that can't be read or modified, and then sign the firmware binary with the bootloader's key.

Re: Hashes not useful

[Anonymous Coward]

Even if a hacked firmware isn't calling home, there are many things it can do. A simple example is detecting a read of /etc/shadow and replacing the hash for the root password with a known hash. Pwned.

Re:Hashes not useful

[amck]

Won't work. See the modification of Cisco hardware intercepted between manufacture and delivery.

Open-source the whole stack. Require access to reflash the firmware securely by independent means.

Previously I would have thought this a pipedream, but with China looking to deny access to its markets to insecure equipment, I'm hopeful that this will happen.

Re:Hashes not useful

[Splab]

If something in your computer is compromised, you are f*cked. At defcon they demonstrated the ability to infect multiple hardware devices on the PCI bus, so even if you manage to get rid of the malware from one device, it was still spreading from the rest.

Re:Hashes not useful

[bill_mcgonigle]

Seagate is correct. Putting a hash on the website doesn't improve security at all because anyone who can change the download can also change the web page containing the hash. ... A company like Seagate doesn't rely on volunteers at universities to distribute their binaries so the technique is pointless.

There are many possible attacks. A hash on a website is not invulnerable to a rogue employee at Seagate (or one "just following orders").

A hash protects against a rouge insertion at the endpoint. Like if your PC is compromised by an attacker and then you pull the hard drive and [assuming there's a way to get a hash from SMART/ATAPI) you can compare the hash of the firmware that the drive is running to the list of published firmwares at the vendor's site. If the attackers are only modifying a small subset of drives, this works fine - they can't also intercept the check to the vendor's site - not unless they've broken TLS and/or have malware on every possible machine.

A tool to verify the firmware is poetically impossible to write. What code on the drive would provide the firmware in response to a tool query? Oh right ..... the firmware itself.

Well, today you can pull the image from JTAG, or so the experts have said (you can verify the firmware directly from memory with a hash if you have moderate funding). There's all sorts of talk about how ATAPI is write-only for firmware because the vendors don't want their competition to get their code and decompile it. This appears to be nonsense, as any other drive vendor already has the debug tools to pull such things from memory, and extracting it from an update isn't that hard - if a 16K DOS update utility can extract it, so can a multi-billion dollar R&D company.

To make it work you need an unflashable boot loader that acts as a root of trust and was designed to do this from the start. But such a thing is basically pointless unless you're trying to detect firmware reflashing malware and that's something that only cropped up as a threat very recently. So I doubt any hard disk has it.

They most certainly do not. So, here we are at today and need a way forward. There are a few ways forward, a fistful of crypto protocols to choose from to ensure future usefulness of hard drives for security applications, and INCITS/SATA-IO ought to be having emergency meetings _right now_ because this (NSA/GCHQ) is a major threat to the industry. The vendors may need to move operations outside of five-eyes to remain commercially viable.

Re:Hashes not useful

[Anonymous Coward]

SIM cards use symmetric crypto, one copy of the key/shared-secret is kept on the SIM to be sent to subscribers and the other is sent to the PLMN operator and stored (in plain text) in it's Authentication Center. There are many possible ways to steal the shared secrets this way. In some instances those files were sent using plain old FTP, in others they were sent by snail mail and could easily be intercepted en-route by the likes of GCHQ and NSA. It's as low as a fruit may hang on a tree, any lower than this you may as well say it already dropped to the ground.

For a firmware they can use assymetric crypto and the private key can be stored on a HSM, so it can be much harder to steal the key. Sure, they can still probably change the firmware before it is ready for signing if they compromise the vendor's internal network or the firmware developer's computers, and depending if the HSM isn't air-gapped, someone may try using side channel attacks to extract they private keys. Depending on how the firmware is stored, an adversary may simply change the the public key on the drives before it reaches the target, but then it may fail if the user tries to upgrade to a genuine firmware later. Anyway, it's a bit more complicated than stealing shared secrets of SIM cards.

Actually, this is something that has always bothered me. Why are we still relying in old/broken crypto for cell phones? The use of symmetric crypto makes it difficult to make sure the shared secrets never leak and the cyphers used are a joke. A5/1 is a joke and can be broken on a PC. A5/3 has some known flaws too and it's not unconceivable that a high budget adversary may be able to brute force it.

Pretty pointless

[rainer_d]

I guess even if there was a way, the vendor would probably just get a NSL to put the backdoor in himself

I'm still waiting for the first CEO to go to jail for refusing this.
Either it's easy to say "No", or nobody bothers, because "war against terror etc.".

Re:Pretty pointless

[Anonymous Coward]

Obviously the under-reporting of what happened to Joseph Nacchio of Qwest Communications by the corporate media is working.

He refused to cooperate with the NSA because he believed (correctly) that their requests for blanket spying on customers were illegal. Keep in mind this was before Bush signed the law granting telcos retroactive unconstitutional immunity from breaking the law, because every other company apparently cooperated with this. The NSA could have gone to the usually rubber-stampy FISA Court, but apparently they were worried that even that normally useless body would rule against them.

Then Mr. Nacchio got conveniently arrested and charged with "insider trading" and his company got harassed with threats of not getting any more government contracts. He was prevented from bringing up any of the NSA strong arm tactics in his defense because "national security", a concept and government authority I conveniently can't find anywhere in the Constitution of course.

He's out of jail now, fortunately. At the time of course all the national security state types were out trolling that people who believed the NSA would do such things needed tinfoil hats, etc. and now of course thanks to another American hero we know the depths of contempt to which they hold the rule of law and the Constitution.

So one CEO did go to jail for protecting his customers. In fact, with all the dirty dealings, corporate spying, financial misdoings, and basically crashing the US economy in 2008, isn't it funny that the ONLY high profile CEO put in jail of late was somebody trying to do the right thing for average people? Everyone should think long and hard about that.

Re:Pretty pointless

[bill_mcgonigle]

I'm still waiting for the first CEO to go to jail for refusing this.

Dude, you're fourteen years behind the news. The technique is not to get you on the "refusing NSA" charge, but any of the other countless criminal acts you commit every day. This is the primary purpose of a hyper-criminalized environment - so that everybody can be easily bent to the whim of the power structure. See also: charge stacking and the de-facto abolishment of the Sixth Amendment through the plea-bargain process (or, if you're a corporation, the no-plea deal for really efficient fascism.

Re:Almost impossible

[itzly]

In the end, such an elaborate scheme is probably directed towards very high value targets. I don't think this is the kind of trojan that runs out in the wild. I could be mistaken, though.

Millions of people doing on-line banking are a high value target. The investment to design a trojan only needs to be done once.

We need hardware write-protect for firmware

[jonwil]

We need jumpers or physical switches that prevent firmware stored in flash (whether it be GPU firmware, BIOS, HDD firmware, network card firmware or whatever) from being overwritten unless you specifically flip that switch.

Re:We need hardware write-protect for firmware

[Anonymous Coward]

In the back of my head for several years now I was thinking the same about the operating system.

Total ignorance of the nut and bolts of OS interiors, but put a copy of the OS on a physically protected chip to verify the integrity of the OS. Yes some files do change but they can be handled as exceptions and stored else wise.

Re:We need hardware write-protect for firmware

[gnupun]

What good will physical switches do if a virus is waiting for you to flip that switch to write-enable so that it can now infect the HDD firmware? Switches would be useful if you never update the firmware. In which case, eliminate the switch and make the firmware permanently read-only. My point is, we need a more secure way to update firmware.

Re:We need hardware write-protect for firmware

[alphatel]

What good will physical switches do if a virus is waiting for you to flip that switch to write-enable so that it can now infect the HDD firmware? Switches would be useful if you never update the firmware. In which case, eliminate the switch and make the firmware permanently read-only. My point is, we need a more secure way to update firmware.

Unless the virus is resident in Bios, (which can also be protected in the same manner), it would be impossible to be infected if you are in a power off state, then enable your switch/jumper, power on, flash your firmware, then disable the switch/jumper after completion before booting into your OS.

In the old floppy days things were pretty much this way. Time to go back.

Re:We need hardware write-protect for firmware

[gnupun]

This could work. As long as the OS is not started during the update... only an uninfected BIOS running and a flash drive would be involved in updating whatever firmware requires updating.

Also, the BIOS should flag an error and stop booting to OS if any firmware switch is in the write-enable position.

Re:We need hardware write-protect for firmware

[hey!]

That's a bit like saying that having a portcullis in the castle gate doesn't help you if the enemy is already inside the walls, which is unquestionably true, but misses the point that having the portcullis makes it harder (although not impossible) for the enemy to do that.

I agree that a more secure way to update firmware, but we have to be realistic in that this would also tend to create new targets for malware writers (e.g. stealing signing keys).

I suspect what we really need is stuff that will occur *outside the box*, such as better vendor of firmware downloads and some kind of police agency tasked with discovering and investigating dodgy firmware. But of course the objection remains -- such an agency itself would be a potential source of problems.

Re:We need hardware write-protect for firmware

[kimvette]

If you've got HP blade servers and call in with something even as mundane as a hard drive or mezzanine card failure, they will often insist you upgrade the firmware before agreeing "yes the hard drive is fuxxored" and sending the replacement part. Even more ridiculous is depending on the tech they might actually ask you to update the motherboard firmware when a motherboard has failed (Um, yeah.), or the iLO firmware even though it is totally unrelated to the problem (fortunately on HP iLO/LOM updates usually don't interrupt services).

The problem with that is even though you might be able to keep the services patched (and even kernel if you use ksplice) and measure uptime in years, updating motherboard or even NIC firmware requires downtime. Even an active/active cluster can introduce down time for some users so downtime of a server is best avoided. Why update the motherboard firmware if there are no bugs blocking production or introducing security issues?

I understand why support reps go through the script and ask you to update firmware so they're dealing with what matches their one test system in their lab, but if it worked as deployed for months or years with the older firmware until the HDD or card croaked, why require a firmware update to a known-stable system before agreeing "Yup, $foo has failed, I'll dispatch a rep with the FRU within four hours" even when S.M.A.R.T reports a hugeassed list of errors, or it's simply not even powering up?

As far as drive firmware goes, I've had to update firmware only twice: once on an SSD, and once on Seagate drives which would freeze during recalibration, which would break arrays (I think it was the infamous 1.5TB drives but it's been a while).

How it's done: Link to SpritesMods.com article.

[Futurepower(R)]

Jumping to page 3 of the article in SpritesMods.com: Parts on the [hard disk] PCB: "My target was to try and compromise the security of a system by using hard disk firmware mods."

Re:How it's done: Link to SpritesMods.com article.

[Smallpond]

That's a great article. So to answer the question in the summary, the way to verify your firmware is:

        Buy an identical replacement drive
        Use the vendor tool to install the same firmware version
        Use the tool described in the article to read the firmware from each drive over JTAG
        Compare the two copies to see if the suspect drive has been modified.

Doable

[sshir]

Actually, writing a verification firmware is possible (assuming, that it was written after attacking code was written and writer was not NSLed)
Simply because attack code doesn't know what output verification code must produce. It either must execute new code (will be busted) or not (will be busted). Putting a full blown interpreter (or some trap mechanism on flash access) will screw timing - again it will be busted.

Re:Doable

[sshir]

Replying to myself: actually one can easily exploit embedded flash size limitation. Simply make new firmware huge and uncompressable. Attacker will ran out of place to hide (without creating timing side effects e.g. storing stuff on platters)

How much CPU power & storage in HDD controller

[swb]

How much CPU power is in HDD controllers and how big is the flash storage on the controller?

I'm mostly just curious, but I wonder how much "elbow room" there is to do something nefarious like blocking updates or protecting boot sectors without compromising drive performance significantly.

Is there a mechanism for running software on the drive controller -- passing input, getting output, etc?

Re:Do you trust the company?

[drinkypoo]

So you would likely flick the switch trusting it was Dell BIOS update yet its a BIOS update carrying spyware.

Yes, the need to have some kind of open bios (coreboot? pretty lumpy process, though) is real, but that doesn't decrease the need for write protect jumpers.

Secure Boot + Full disk encryption

[vojtech]

Actually, the much hated Secure Boot (with the shim loader, MOK, and GRUB2), combined with full disk encryption (for example using LUKS), and in filesystem compression (btrfs2) can quite nicely protect you from anything that a malicious firmware in a harddrive could do. The firmware will only ever see encrypted data passing through it, except for when loading the bootloader and the kernel, which will both be cryptographically verified by UEFI. The in-filesystem compression is there to compensate for the compression SSD drives normally do themselves to gain additional speed that will be impossible to do that on encrypted data.

Sure, this basically converts the problem to trusting the main BIOS (UEFI), but that's something you have to solve in any case.

Re:Secure Boot + Full disk encryption

[vojtech]

Of course the UEFI code cannot bypass the drive firmware. From the point of view of secure boot, the boot media is untrusted, and thus it doesn't care whether there is any malicious code in the drive firmware. It simply verifies that anything it loads from there is signed and thus uncompromised. If the bootloader or kernel were tampered with, Secure Boot will refuse to boot.

Re:How much CPU power & storage in HDD control

[digitalchinky]

Have a read of this: http://spritesmods.com/?art=hd...

There is a decent chunk of compute power in the controllers.

Re:How much CPU power & storage in HDD control

[GrangerX]

Well, if you're the drive controller firmware, storage space is a non-consideration for the most part. You can store stuff on the drive platters. A certain percentage of those are reserved for sector reallocation anyway, so you could use those without anyone really every being the wiser.

Seagate HDs

[BlackLotus89]

If it's about seagate hds you can take a look at seaget.With this you can dump the buffer and memory of your harddrive. Here is an explanation https://blacklotus89.wordpress... and here is the code https://github.com/BlackLotus/... Maybe this can be used to dump the firmware as well (somehow)

Re:Seagate HDs

[jeff4747]

With this you can dump what the firmware claims is the buffer and memory of your harddrive.

FTFY.

Security by Oscurity

[ramriot]

Here is the problem:
Manufacturers guard their intellectual property fiercely, and they guard their proprietary firmware fiercest of all. Thus the API for uploading drive firmware is Write Only (WO). Thus within the existing API and interface there is by design no way to validate the firmware. What that means is that, if you are able to build your own firmware (because you have a copy of the source, obtained deviously) then you can alter it to your own ends and even make it so that the (WO) overwrite API does nothing.

Outside of the existing interfaces though you can with sufficient skill get some knowledge. If the firmware is stored on a flash chip separate from the drive CPU you can get a copy of the microcode by probing the chip directly either during read cycles with the drive active or by controlling the chip fully with the drive off. Unfortunately you cannot do this so easily if the firmware is stored in flash within a drive micro-controller. As to JTAG, that may or may not work because in production a manufacturer may choose to disable that interface to prevent competitors doing exactly what you are wanting to do.

In Summary, you are SOL unless manufacturers rewrite their firmwares to add a secure means of proving firmware validity, and don't ask me how.

row:how ?

[ArcadeMan]

if that's been tampered with there's no way you can be sure.

Nuke it from orbit. It's the only way to be sure.

Re:How much CPU power & storage in HDD control

[thogard]

There is enough flash and ram to run Linux on the controller. I've seen it done at Ruxcon/Breakpoint where the hard drive booted up to the point where it couldn't find a root disk to mount.

It is trivial to make firmware that watches for things like /etc/shadow files and returns something else. You can have this code activate by searching for data that would be logged and hunting for the magic key and that is trivial since every system logs to disk.