Slashdot Mirror


Exploiting the DRAM Rowhammer Bug To Gain Kernel Privileges

New submitter netelder sends this excerpt from the Project Zero blog: 'Rowhammer' is a problem with some recent DRAM devices in which repeatedly accessing a row of memory can cause bit flips in adjacent rows. We tested a selection of laptops and found that a subset of them exhibited the problem. We built two working privilege escalation exploits that use this effect. One exploit uses rowhammer-induced bit flips to gain kernel privileges on x86-64 Linux when run as an unprivileged userland process. When run on a machine vulnerable to the rowhammer problem, the process was able to induce bit flips in page table entries (PTEs). It was able to use this to gain write access to its own page table, and hence gain read-write access (PDF) to all of physical memory.

37 of 180 comments (clear)

  1. Impressive by Anonymous Coward · · Score: 5, Insightful

    Don't have much more to say than that's an impressive exploit.

    1. Re:Impressive by twistedcubic · · Score: 3, Insightful

      Double bonus if this result gets manufacturers of laptops to FINALLY include ECC memory.

    2. Re:Impressive by Anonymous Coward · · Score: 2, Informative

      It's a hardware problem, not a software problem.

    3. Re:Impressive by Shinobi · · Score: 5, Informative

      And, if you had read the actual paper, you'd see that ECC isn't proof against it either

    4. Re:Impressive by MachineShedFred · · Score: 2

      so when they said:

      We also tested some desktop machines, but did not see any bit flips on those. That could be because they were all relatively high-end machines with ECC memory. The ECC could be hiding bit flips.

      they actually said that ECC doesn't matter?

      I guess we read differently.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    5. Re:Impressive by MachineShedFred · · Score: 3, Informative

      I see - you were looking at the PDF link, which is more theoretical. yes, if you can get two or more bits to shift inside a 64-bit chunk, then ECC doesn't help. There's got to be a low probability of that actually happening though - the Google Project Zero wasn't able to make it happen with ECC at all.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
  2. possible iOS Exploit? by muphin · · Score: 2, Interesting

    is this possible to exploit on an iPhone?

    --
    It's not a typo if you understood the meaning!
  3. ECC Memory by Frobnicator · · Score: 5, Interesting

    Yet another reason to push shared providers for ECC memory. The error correcting memory is so far not vulnerable to this attack, all the researchers that have tried it report that ECC memory identifies and corrects the corruptions. Of course some attackers may have found a way, but ECC minimizes the risk

    Amazon says it uses ECC in their AWS machines, but other big hosts like Equinix say that ECC memory is "available". Be careful about your hosting, folks.

    --
    //TODO: Think of witty sig statement
    1. Re: ECC Memory by Bruce+Perens · · Score: 2

      Yes, you beat me to it. A correctly-configured ECC motherboard with real ECC memory would defeat this. Watch out for fake ECC memory that just simulates the correction bits.

      Once memory starts being vulnerable to row interference, having a machine without ECC becomes much more dangerous, regardless of this exploit.

    2. Re: ECC Memory by Bruce+Perens · · Score: 4, Insightful

      It has yet to be established whether hammer techniques can result in a correct data+ECC pattern. If so, it should be possible to permute the memory in a way that defeats this, either on the memory module or the memory controller.

      That would make a good research paper for someone.

    3. Re: ECC Memory by thogard · · Score: 2

      ECC might be able to help the attack. If you know the state of memory and the associated ECC values you would like and can calculate a designed bit pattern with the same ECC that meets the requirements, you may be able to get the ECC hardware to flip the bit for you as you hammer bits that don't matter as much.

      Hammering memory to induce writes where they shouldn't happen has been done for decades. It was used back in the days when you needed high voltages to do writes in eeproms when people found out that you could use a 5V write power supply and sometimes get bits to change if you tried enough times.. Related techniques have been used with bubble memory and iron core as well.

    4. Re: ECC Memory by tlhIngan · · Score: 2

      I can't see how it would be possible to defeat ECC.

      The attacker would have to construct a write that affects the desired bits in the row-to-be-hammered and has check bits that affect the row-to-be-hammered's check bits such that the altered row is validated. This is probably nigh impossible to do in all but a select few constrained cases.

      It's possible, but very unlikely.

      Rowhammer is not new - it's been known since the 90s since it affects NAND flash memory as well (the same stuff in an SSD) - here there are two problems. It's called write disturb and read disturb. Because in NAND flash, all the storage transistors are wired in series - each transistor is a page of flash, and all the pages are wired in series, so reading one page requires the transistors in other pages to be activated to be able to read the desired page.

      So NAND flash manufacturers design their transistors sepecially so you minimize write disturbs (where writing to a page flips bits in other pages) as long as you write the pages in sequence. But there's also read disturbs where the act of reading a page can flip bits on other pages because you're still activating those series transistors.

      That's why NAND flash has the spare area after each page - it's for ECC data which is required to catch such errors.

    5. Re: ECC Memory by viperidaenz · · Score: 4, Informative

      When you write to a row of memory, it's ECC bits are written to too by the memory controller, which are next to the other ECC bits.

    6. Re: ECC Memory by Macman408 · · Score: 5, Interesting

      I hadn't heard of this either, but a quick google turned up a description of false parity RAM: http://en.wikipedia.org/wiki/R...
      TLazy;DR: To save cost where parity RAM was required by the hardware but not by the operator, modules existed that would calculate the parity bit upon reading the RAM, rather than storing the parity bit. I don't see any evidence that this type of module ever existed for ECC though.

      To make sure memory is ECC, it's probably sufficient to count the memory chips on a DIMM. If there are 9 or 18 (or even 36, if it's a particularly large DIMM) identically-marked chips, that's ECC. If there are 4, 8, 16, or 32 chips, then it's probably not. If one of the chips is marked differently than the others, it might be a little more complicated; it might be possible that it's a different memory chip (e.g. if there are 4 x16 memory chips, you'd only need one x8 to get a x72 ECC DIMM, so that last chip would be different). But it's also possible that it's buffered/registered memory, and the different chip is the buffer/register.

      And an aside on the topic of buying RAM for yourself:
      In general, I'm not a fan of cheaping out on memory. I did computer repair for a while, and it shocked me how many problems were caused by bad RAM - from the obvious ("my computer crashes every time I boot it") to less obvious ("every few days, an application crashes") to the rather insidious ("it was running fine, and now I can't mount my hard drive any more"). It got to the point where, when a computer came in with nonspecific symptoms like that, I'd open up the computer and peek at the RAM chips first. If they had no recognizable manufacturer, they were certainly garbage. If they were recognizable but not top-tier, they probably needed some stress testing on our RAM tester. And if they were the good stuff (Samsung always had my vote there, though it's hard to find because they don't sell directly to consumers), then it was probably something else.

      That's also where I learned that things like memtest86 or other software diagnostic tools were basically useless too. Only the absolute worst memory would fail a test, even a looped test run for days. Most bad RAM was marginal - after all, it probably passed some manufacturing tests. We had a rather expensive (~$4k-8k) box that would test memory, doing things like varying the supply voltage or self-heating the RAM. When RAM is installed in your PC, you're still limited by the hardware - i.e. the voltage regulator and the memory controller - which probably keep the memory as close to nominal conditions as possible. Obviously, those machines are rather hard to come by, so you have to make do with software tests instead - but a pass on those just means I can't prove it's bad; it doesn't mean the memory is good. Even if I pass all memory testing, I'll still swap/remove/replace DIMMs in an attempt to find which one is bad, because it's often not obvious.

    7. Re: ECC Memory by Shinobi · · Score: 3, Informative

      In the paper, they actually state that ECC isn't entirely proof either, because you can get multiple bit errors as per their testing. SECDED will be defeated by that. Chipkill might work.

    8. Re: ECC Memory by An+ominous+Cow+art · · Score: 5, Funny

      Now that's insideous!

      It takes 2 bit changes to change 'i' to 'e', so your problems are worse than you thought...

    9. Re: ECC Memory by bluefoxlucid · · Score: 3, Interesting

      Wouldn't it be better design to put ECC into the memory controller, and arrange the chips to support ECC? That is: physically wire the memory chips and the memory bus to write far from each other (64 chips = interleave every 1 bit across all chips; 32 chips = interleave 2 bits per chip, starting on high bits in addressing so you put them half a chip's distance away from each other), physically protecting against chip-local anomalies. Have the MMU perform a single rotation, logically reserving an amount of RAM on each slot to carry ECC for the previous slot. Write ECC bits to those areas.

      Doing it in this way provides zero-cost physical isolation of single-chip memory errors (it's just the wiring layout), while also isolating the ECC from its corresponding module (avoiding RAS/CAS thrashing, allowing simultaneous access to the ECC bits and the corresponding RAM). It lets you pop in an ECC chip and use ECC, or pop in a non-ECC chip and sacrifice 12.5% of your RAM to error correction. That's a lot of RAM: on an 8GB system, it's almost 1GB.

  4. Rowhammer in MemTest86 & on Slashdot by PassMark · · Score: 5, Informative

    It is worth noting that the row hammer issue isn't new. It as been known about for some time. Including this old Slashdot post
    http://hardware.slashdot.org/s...

    There has been an implementation of row hammer testing in MemTest86 V6.0 for over 6 months now as well. MemTest86 implements just the single sided hammer, whereas Google used a double sided hammer.
    http://www.memtest86.com/
    While the double hammer might produce more RAM errors, this pattern of memory accesses isn't very likely to occur in real life software. So is of limited use as a RAM reliability test.

    What is new in this report is the fact that they manipulated the RAM bit flips to turn them into an exploit. Something that was previously speculated on but considered too hard to implement.

    What they didn't show however is any results from desktop machines. All their testing was on laptops. In fact they state, "We also tested some desktop machines, but did not see any bit flips on those". So the problem isn't as grave as it might at first appear. They speculate that ECC RAM blocks the bit flips and this has also been the experience with MemTest86, most (but not all) of the flips are single bit flips, which ECC would correct.

    Disclaimer: I'm one of the MemTest86 developers.

    1. Re:Rowhammer in MemTest86 & on Slashdot by Anonymous Coward · · Score: 2, Interesting

      Similarly, this is a relatively well documented issue in the embedded world as well...

      ARM have erratum for half a dozen IP revisions over the past few decades, as do many IHVs that make SoCs based on them.
      Similar deals in the MIPS and PowerPC realms.

      Some architectures thankfully have a rather locked down MMU/MPU, where unprivileged code simply can't even attempt to rowhammer (some ARMs are like this) - alas, most do not - thus you're at the mercy of all the other hardware vulnerabilities in your system (not just your CPU, but your memory controller, interconnect bus(es), coprocessors, and memory itself - which is what this rowhammer exploit is really about, JEDEC (LP)DDR2/3 simply isn't strict enough at lower frequencies about signal integrity - which is what rowhammer exploits).

      I guess hardware validation isn't as big a deal in the x86 scene as it is in the embedded scene.

    2. Re:Rowhammer in MemTest86 & on Slashdot by phantomfive · · Score: 2

      What is new in this report is the fact that they manipulated the RAM bit flips to turn them into an exploit.

      That's bigger than being able to corrupt memory in the first place. What it means is that every computer (laptop I guess, without ECC) is vulnerable to a privilege escalation exploit, and the difference between root and a normal user is meaningless.

      Next all we need is a way to exploit this from javascript. :)

      --
      "First they came for the slanderers and i said nothing."
  5. Deja vu... by sotweed · · Score: 5, Interesting

    This problem is remarkably similar to a problem I encountered in the memory of a 7094 (old
    IBM computer) which had a core memory which stored 36-bit words. The memory was supposed
    to work by operating on 6 bits at a time at 200 nanosecond intervals. The reason for this was to avoid
    creating a magnetic field that was too strong. The problem occurred when the timing was off due
    to failure of a component and two of the intervals overlapped. This meant that when one attempted
    to store a word with 35 1s, the field created was strong enough to store 36 1s. We wrote a
    diagnostic to demo the problem, and with that the engineers were able to isolate and fix the problem
    in short order.

    1. Re:Deja vu... by ArcadeMan · · Score: 5, Funny

      Alright, alright! We're getting off your lawn!

    2. Re:Deja vu... by sotweed · · Score: 5, Interesting

      I was describing something that happened in a machine that was built before the world settled
      on 8-bit bytes. The machine had 36-bit words, and each word had an address. The 6-bit
      nibbles were not addressable. It was 32,768 (2**15) words of 36 bits. Equivalent
      to a little over 100K bytes!

    3. Re:Deja vu... by wonkey_monkey · · Score: 2

      We wrote a diagnostic to demo the problem, and with that the engineers were able to isolate and fix the problem
      in short order.

      That claim alone is enough to date your story back at least 30 years.

      --
      systemd is Roko's Basilisk.
    4. Re:Deja vu... by Anne+Thwacks · · Score: 2

      The IBM709x series were in use around about 1970. By 1974 the system 360 was all the rage, and bytes were in common use, so probably over 40 years ago.

      --
      Sent from my ASR33 using ASCII
  6. Re:Frist Psot!! by grcumb · · Score: 5, Funny

    I got First Post!!!!!!! Yippppppeeeeeeee!!!!!!!!

    And I don't even know what a DRAM rowhammer is!!!!!!!!!!

    Dude, guess what? We row-hammered you into second place. Now excuse me while I flip you the bit. :-)

    --
    Crumb's Corollary: Never bring a knife to a bun fight.
  7. Multiprocessing by Bruce+Perens · · Score: 2

    Multi-threaded programs really do need those cache flushes to implement their interprocessor communications, don't they? It seems to me that they would be the ones most likely to hit this problem.

    1. Re:Multiprocessing by Pelam · · Score: 2

      I'm not sure. The locked instructions, compare and exchange and mfence ensure cache coherency so in my experience the flushes are not necessary.

      Maybe driver code needs the flushes. Driver needs to know data is really in the RAM before hardware with DMA can get it.

      Cache flush instructions seem to be a late addition with SSE2.

    2. Re:Multiprocessing by Bruce+Perens · · Score: 2

      Compare-and-exchange and mfence would be doing cache flush all of the way to RAM and global cache line invalidation, wouldn't they? So, they can potentially be used to hammer too.

    3. Re:Multiprocessing by TheRaven64 · · Score: 3, Interesting
      They don't flush, no. They will add memory fences, which will generate cache coherency bus traffic, but won't trigger a write back to main memory (modern CPUs can snoop the cache of other cores, so the data will be sent cache to cache).

      The main reasons for flushing the cache are:

      • If you have some non-volatile DRAM and want to ensure consistency.
      • If you're doing DMA on anything other than the latest Intel chips, so that the DMA controller will see the data that you've flushed from the cache.
      • If you're writing a JIT compiler or some other form of self-modifying code (including a run-time linker) and need to ensure that i-cache and d-cache are consistent (I think x86 does this automatically, but I could be wrong).
      • If you're writing a crypto algorithm and want to make side-channel attacks via the cache difficult.
      --
      I am TheRaven on Soylent News
    4. Re:Multiprocessing by TheRaven64 · · Score: 3, Interesting

      Nope, no cache flush for compare and exchange. Modern CPUs use a modified version of the MESI protocol, where each cache line has a state associated with it (modified, exclusive, shared, invalid in MESI, a few more in modern variants). When you do a compare and exchange, you move your copy of the cache line into exclusive state and everyone else's into invalid. Before this, you must have the line in the shared state (where multiple caches can have read-only copies). When another core wants access to the memory, it will request the line in shared state. If another cache has it in its exclusive state, then the exclusive line will be downgraded to shared and a copy of its contents sent to the requesting site.

      If atomic operations had to go via main memory then they would be significantly slower than they are and would be a huge bottleneck for multicore systems.

      --
      I am TheRaven on Soylent News
  8. Difficult to exploit on servers by AaronW · · Score: 5, Informative

    In reality this would be difficult to exploit on a server since servers typically use ECC memory. ECC memory can typically detect two bits and correct one bit error and will likely catch this unless you can flip enough bits correctly so that the ECC remains correct. Doing this means that you would need to know the contents of those memory locations prior to flipping the bits.

    I don't know about X86, but on the CPUs my company makes we support hardware address randomization, so that the address lines going out to memory are randomized such that finding adjacent rows or columns can be very difficult to figure out.

    It's a shame that Intel only supports ECC memory with their XEON processors rather than all of their processors. ECC does not add much in terms of cost, only 12.5% more DRAM chips and a few more traces and a few other miscellaneous parts (resistors, capacitors). Even the lowest end processors my company makes (for things like small routers, wireless access points, etc) supports ECC and address line randomization.

    --
    This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
  9. Thanks to Wang by michaelmalak · · Score: 4, Informative

    All RAM on PCs used to be parity RAM until Wang started suing RAM manufacturers in the 90s over its patents on parity SIMMs.

    1. Re: Thanks to Wang by fuzzyfuzzyfungus · · Score: 5, Funny

      That sounds like a real dick move on their part.

    2. Re: Thanks to Wang by hottoh · · Score: 3, Informative

      All RAM on PCs used to be parity RAM until Wang started suing RAM manufacturers in the 90s over its patents on parity SIMMs.

      Not so.

      Many had ECC. AAPL computers skipped ECC, to save money and look stupid at the same time (I made the stupid part up).

      AAPL = Apple Inc., FYI

  10. Re:Escalation by mlts · · Score: 2

    I wonder if there is -any- way to mitigate this in software, similar to how the Linux kernel intercepted the instructions to prevent the FDIV bug from happening in early Pentium chips. The only way I see would be to use a Bochs style emulator, and deal with its immense performance hit that its style of emulation does (where hardware virtualization hooks are not used.)

  11. [citation needed] by Anonymous Coward · · Score: 2, Interesting

    I've followed the PC market since the 1970s. I was paying a good bit of attention to memory cost. As far as I could tell, people noticed that parity RAM hardly ever caught errors, and that non-parity RAM was cheaper than parity RAM. Parity became optional, then eventually went away entirely, simple because it was marginally more expensive.

    Patent issues came and went, but I don't think they were a major driver away from parity.