Hyperthreading Considered Harmful

cperciva writes "Hyper-Threading, as currently implemented on Intel Pentium Extreme Edition, Pentium 4, Mobile Pentium 4, and Xeon processors, suffers from a serious security flaw. This flaw permits local information disclosure, including allowing an unprivileged user to steal an RSA private key being used on the same machine. Administrators of multi-user systems are strongly advised to take action to disable Hyper-Threading immediately. I will be presenting this attack at BSDCan 2005 at 10:00 AM EDT on May 13th, and at the conclusion of my talk I will also releasing a paper describing the attack and possible mitigation strategies."

Comments

Sysadmins have been advised...

[Anonymous Coward]

to give their hyper-threading processors some Ritalin.

This ought to be interesting

[displague]

Doesn't Linux handle HT the same way it handles SMP? So even if there was a hole in HT, hardware-wise, software wise you would be just as protected as you would be on an SMP system?

Re:This ought to be interesting

[cahiha]

Well, I didn't see an explanation on the site... here is my guess what happened.

With HT, registers for multiple threads have to reside within the same CPU, whereas with SMP, they are physically in two different chips. Perhaps the registers used by HT are not properly protected against reading by other threads. The fact that SMP code was probably reused may be the cause of the problem: the code developed for SMP didn't have to deal with this situation before.

Guessing even a bit more, this may be hard to fix in software. If the hardware simply doesn't protect registers properly, then the kernel may have to clobber them to protect the information, but that may impose far too much overhead for HT.

However, a workaround might be to permit HT only for multiple threads within the same process. That would still give some speedups to compute-intensive processes that are written to take advantage of threading.

Re:This ought to be interesting

[uss_valiant]

Doesn't Linux handle HT the same way it handles SMP? So even if there was a hole in HT, hardware-wise, software wise you would be just as protected as you would be on an SMP system?

If you follow the article link you see that it only mentions *BSD and SCO unix variants. Not a word about linux or windows or any other x86 based OS.
I guess we will have to wait for his final paper which should be available in +12 hours or something like that.
Seems like he spent considerable time on this issue and he is unemployed. If you read his website, you probably come also to the conclusion that he sees this as an opportunity to get paid for his volunteer work. This /. announcement for his paper that will be available in a few hours can be seen as a great advertisement for his own work (no offence).

He could also have contacted AMD to get a little funding as they didn't jump on the HT train :)

Re:This ought to be interesting

[Chris Snook]

Unlike SMP, with HT you're interleaving two threads on the same physical execution unit. That means that there is data from another thread in registers at the same time that you're executing, without having enough instructions execute during a context switch to flush the pipeline. It also means that the other process's page table is in the MMU while you're executing. Even if their proof-of-concept attack doesn't work on some other operating systems, everyone needs to look over their code to make sure this isn't just an accidental effect that could change with increasing pipeline depths, different context switch logic, etc.

Re:This ought to be interesting

[gklyber]

Limiting hyperthreading to single processes is a good idea anyway. Since the multiple threading units share the same cache, you'll get better cache coherency by doing this.

Re:This ought to be interesting

[timster]

Regular desktop apps tend to have lots of threads, but the issue is not how many threads exist but rather how many of them attempt to use CPU at the same time.

For instance, your web browser might spawn a thread to do a DNS lookup, since you wouldn't want the GUI to block during DNS. That thread hardly uses any CPU though. When your Web browser does real work, like rendering, it will usually be confined to a single thread.

Re:This ought to be interesting

[Drakonian]

I just watched his talk, and you are on the right track. Your workaround is one he suggested too. It's actually a timing based attack based on watching the cache misses in a spy thread to try and reverse the RSA public key. The interesting thing is this isn't Hyper-Threading only - it's possible on normal procs too that don't flush the cache between context switches. It's just that with HT context switches can be far more common.

Whoosh!!!

[EmagGeek]

Shit, did anyone see that blur???

Yeah, I think that was Intel's server market going right out the window at Mach 10...

Re:Whoosh!!!

[paulhar]

Somehow I doubt it. If every time a securty issue was found with a product people binned it then we wouldn't be running Linux, Windows, Firefox, IE, WiFI, etc...

Re:Whoosh!!!

[mmkkbb]

Actually, Intel CPUs contain patchable microcode ROMs. You can see the option to enable it when you configure a Linux kernel.

Quick fix

[Junior J. Junior III]

I am counteracting the harmful effects of hyperthreading by eating a high-fiber diet. So far, I haven't had any problems.

Think!

[dybdahl]

Not all multi-user systems are designed to be secure against the best hackers around, and there is often bad cost/benefit at following all security recommendations as soon as you hear about them.

Give us some more facts, so that we can think for ourselves.

Where's the details?

[tulimulta]

Not much to read yet. Seems more like a publicity stunt by the author. This could have been posted *after* the details have been published.

Re:Where's the details?

[Anthony]

Maybe there are still unsold tickets for BSDCan05.

Re:Where's the details?

[Drakonian]

Sure. It's a timing based attack on based on watching cache misses. If you have a Spy thread running on an HT processor that is also running OpenSSL for example, you can get a picture of the frequency of cache usage and from that reverse engineer the exponents and multipliers used in the RSA exponentiation. Note: You'd definitely need some cryptographic experience for this. From this, you can get about 310 bits of the 512 bit exponent and brute force the rest, which can be done in polynomial time.

The reason HT is vulnerable is because both threads share the cache and context switches can happen at any time. It could on normal non-HT procs too but the context swithces are more likely to flush the cache or not happen as often.

Re:Where's the details?

[cperciva]

How about just not allow different UIDs on the core at the same time?

That would be the ideal solution (assuming that you also check for setuid/setgid programs). Unfortunately, it's really hard to do that correctly due to problems of kernel data locking.

FreeBSD's policy on security fixes is that they must never ever break anything -- so if necessary (as in this case) a simple but suboptimal fix will be used instead of a complicated fix which might have the inadvertent side-effect of causing machines to crash.

How to exploit

[jabagi]

I'm curious to see how an exploit can be made out of this. Is it possible to assign one of the virtual CPUs to a "sniffer" for a prolonged period?

Re:How to exploit

[mikael]

The Ars Technica page on hyperthreading with the Xeon might provide some clues. It lists which parts of the CPU are replicated, partitioned and shared.

One final bit of information that should be included in a discussion of partitioned resources is the fact that when the Xeon is executing only one thread, all of its partitioned resources can be combined so that the single thread can use them for maximum performance. When the Xeon is operating in single-threaded mode, the dynamically partitioned queues stop enforcing any limits on the number of entries that can belong to one thread, and the statically partitioned queues stop enforcing their boundaries as well.

...

The same can be said for the register file, another crucial shared resource. The Xeon's 128 microarchitectural general purpose registers (GPRs) and 128 microarchitectural floating-point registers (FPRs) have no idea that the data they're holding belongs to more than one thread--it's all just data to them, and they, like the execution units, remain unchanged from previous iterations of the Xeon core.

For a simultaneously multithreaded processor, the cache coherency problems associated with SMP all but disappear. Both logical processors on an SMT system share the same caches as well as the data in those caches. So if a thread from logical processor 0 wants to read some data that's cached by logical processor 1, it can grab that data directly from the cache without having to snoop another cache located some distance away in order to ensure that it has the most current copy.

...

You might think since the Xeon's two logical processors share a single cache, this means that the cache size is effectively halved for each logical processor. If you thought this, though, you'd be wrong: it's both much better and much worse. Let me explain.

Each of the Xeon's caches--the trace cache, L1, L2, and L3--is SMT-unaware, and each treats all loads and stores the same regardless of which logical processor issued the request. So none of the caches know the difference between one logical processor and another, or between code from one thread or another.

more info at KernelTrap

[Anonymous Coward]

I read about this last night here at KernelTrap. They offer more info, evidently having talked to Colin...

This is SERIOUS!!!

[AltGrendel]

SCO: This affects OpenServer 5.0.7 if an update pack is applied and SMP is installed; it also affects UnixWare 7.1.4 and 7.1.3 with hyperthreading enabled, but hyperthreading is disabled in UnixWare by default. This is covered by advisory SCOSA-2005.24.

Ooooo, I'm SCARED!

Simple Solution

[JerkyBoy]

Recompile your kernel with hyper-threading disabled. Simple question: Why do I have to wait until this guy does his conference presentation to find out what the exploit is, how it is implemented? I have to admit that this is one time when RTFA didn't work. Anyone have any more information?

May I be first to say...

[game kid]

...I'm glad I'm stuck with a 1-gig Pentium III.

Reminds me of a bug in Michigan Terminal Service

[jesup]

On MTS (IBM mainframe OS used at universities in the 70's/80's and probably into the 90's) there was a bug where when process switching, the FP registers of the last process to run were stored in a world-readable page of memory. The RPI ACM used this to create an inter-process communication program -- actually a 'chat' program (MTS had no inter-process communication other than files at the time).

Extreme Edition...

[Sialagogue]

With Moore's Law still holding up, isn't it a little early to be using up names like "Extreme Edition?" So, I'd like to propose my own corollary to Moore's Law:

"The microprocessor industry will run out of hyperbole long before they run out of transistors."

Re:Extreme Edition...

[grouse]

Am I the only one that noticed that the natural acronym for Intel Pentium Extreme Edition is "I PEE?"

Oh dear.

[Morky]

I guess I need to shut off hyperthreading on our app server before the users who can't sort an Excel spreadsheet have a chance to expliot the vulnerability.

Probably a Timing-Based Attack

[Sunlighter]

My guess is that this is a timing attack. While thread 1 generates an RSA key, thread 2 times itself performing various instructions. If thread 1 is using the FPU to do a multiply, the FPU won't be available for thread 2 right away, so there will be a measurable delay. Thread 2 can then determine when thread 1 is running multiplies.

If my hunch is correct, an OS could fix this by allowing a process to enter a "secure mode" which would force the other thread on the same CPU to be idle when that process was scheduled.

Re:Probably a Timing-Based Attack

[MoogMan]

It is more likely that there is some inherant cache leakage from Logical CPU 0 to Logical CPU 1. Therefore, a process running on CPU 1 could continuously snoop data that CPU 0 is read/writing as its doing its calculations. e.g. In-between an operation at a FPU, being stored in the cache ready to be processed in a different unit.

Re:Probably a Timing-Based Attack

[AtrN]

This got mentioned in comp.arch and Dan Bernstein pointed out others have mentioned similar things previously. The abstract mentioned reads,

Other People's Cache - HyperAttacks with HyperThreading - Dag Arne Osvik, Norway

We have investigated the use of memory caches of modern processors as side-channels for timing attacks against software implementations of cryptographic algorithms. In particular, we have successfully performed a new kind of attack where the attacker has no privileges other than being able to run on the same processor as the victim. That is, the attacker has no access to plaintext or ciphertext, and is not allowed by the operating system to communicate with the victim. In this scenario we have recovered 45 out of 128 key bits from AES encryption of English text in just one minute on an Intel processor with HyperThreading. Moreover, with regular known plaintext attacks we have achieved full key recovery.

In other news..

[peterprior]

...it appears Windows XP Starter Edition may be the most secure option after all...

opportunity to get paid for his volunteer work

[dpilot]

I'd sooner guess that by presenting a paper at a conference, he's hoping to turn this into a job offer. There are any number of stories about black-hats mending their ways, and getting security jobs. Here's someone trying to start out as a white-hat, doing things the right way to begin with. Seems to me that if he's on the mark, he's a better risk for a job offer than a reformed black-hat.

Same Guy?

[Bananatree3]

This is the same guy who calculated the 1 Quadrillionth hexadigit of Pi (no, not digit. It is in base 16). His project was called PiHex. According to his currently short but illustrious trackrecord, along with this current announcement, he is destined for being a big-name IT security guru.

Re:It is just an 'give me a job' attention grab

[Intrigued]

I don't see that.

If he can produce even a moderately effective proof-of-concept exploit (which apparently he has), someone with a little malicious creativity will find out a way to abuse it.

Also as a security professional, any gap, niche or irregularity in core security processes needs to be taken seriously even if nothing ever pans out in a real exploit.

As far as the attention grab, I don't begrudge the guy at all. If the exploit is bogus, he will have advertised to the world "I'm an idiot - don't hire me!". If it is valid, he has shown his worth and deserves some support.

Re:No details given...

[Ed Avis]

I think he is giving the world some warning that the security hole exists and that after the conference exploits for it may appear in the wild (if they don't exist already). Vulnerabilities are sometimes announced this way: for example the OpenSSH team issued a mysterious warning saying that everyone running sshd should turn on privilege separation. A little while later they disclosed an exploit in the current sshd and made a new release with a fix. It happened that if you had privsep turned on you weren't affected by the exploit. The warning gave most admins a chance to secure their systems, but didn't give any details of the exploit.

Now, you might not think this guy is credible and so wait for him to 'show you the code' before applying the suggested fix. That's up to you.

On the other hand

[MountainMan101]

I'd be willing to bet he's right. He is currently awaiting a doctorate from the University of Oxford, which is commonly held as the finest academic institution in the world.

(I'm not biased by having spent the past 7 years there)

Re:On the other hand

[babbage]

And this isn't the first time he has come up with some interesting research that has been mentioned on Slashdot before. Sure, he seems to be a little arrogant, but with his record so far, I think he's earned the benefit of the doubt here...

You can't win for losing

[FreeUser]

the reason comes tomorrow. Oh, and you should also give me all your cash today because it is obsolete, more details to come tomorrow.

Yes. While I am a "full-disclosure is better than not" guy, you (or others like you) would be screaming even louder about how "irresponsible" this guy would be if he had released the "reason" today (said "reason," BTW is a proof-of-concept exploit, one that malicious jerks will probably adapt to their desires after it's released).

Oh yes sysadmins, disable hyperthreading because some poster on slashdot said so. This is just too gay.

Not as asinine as clueless AC posts like yours, modded up as "insightful" by equally clueless people who happen to have moderation points today. The guy is awaiting his doctorate at one of the world's most prestigious universities, has an excellent track record, and has chosen a conservative but less-controviersial approach in disclosing this issue.

All of which you would have known, if you'd bother to read TFA rather than spouting off nonsense here.

SCO Unix variants...

[Per Abrahamsen]

As we all know, this includes Linux :-)

Security is a real-time embedded application

[gvc]

Some of the most effective hacks/espionage come from exploiting "secondary channels" for information.

For example, I know of one hack from the good old days that involved placing a password across a page boundary. The OS compared the password to a plain text version character-by-character, so faulted if the characters up to the page boundary were all correct. Observing the disk access light (or the time to reject the password) provided character-by-character cracking.

Of course, password checking is now more sophisticated, but so is cryptanalysis. I think people that use encryption for real are well aware that there's an exposure in doing so on any time-shared system, or any system that can be observed in any way by a potential cryptanalyst.

I would guess, based on the sparse information presented here, that this is the nature of the attack. If - and that's a big if - you can cause an adversary to be scheduled in just the right way, you may be able to capture part or all of a private key by observing timing artifacts of the hyperthreading implementation.

This may be good security research, but unless I were protecting state secrets, I'd wait and evaluate the risk relative to other security risks that we find acceptable. I would also guess that the exposure is minimal compared to other high-tech and low-tech potential information leaks.

Re: On the gripping hand

[Lodragandraoidh]

Alan Turing went to Cambridge and earned a fellowship there. That is also where he conceived the idea of the Turing machine - the basis of all programmable computers.

Where you get your education is immaterial. More important is what you do with it.

Re:So does anyone know...

[anno1602]

Hyper Transport has nothing to do with Hyper Threading. Hyper Threading means processor support for several (usually two) execution threads at once. Hyper Transport is a bus technology to interconnect pocessors, RAM, motherboard chips, PCI bus and the like.

AMD's Hyper Transport is similar to Intel's Hyper Threading, but in my books, superior.

That's like saying that the computers from Apple Computers are similar but superior to the computers from Apple Records. Notice how Apple Records makes no computers? Just because they start with the same word does not mean two things are the same.

Paper

[cperciva]

My paper is available here.

Have fun reading, I'm going back to the conference.

Re:Timeline - WTH?

[cperciva]

Why wasn't Intel notified over the past SEVEN MONTHS ?

They were. I've clarified the page somewhat now, but "Other security teams" includes Intel.

Re:2 or 3 months before notifying other vendors?

[cperciva]

Why notify FreeBSD and then wait 2 or 3 months before notifying other possibly affected vendors (at least other BSDs)?

Two reasons. First, because I'm part of the FreeBSD Security team -- I'm required to notify them about potential issues.

Second, because if I contacted lots of security teams with what I had on December 31st, they wouldn't have listened: "Umm, hey guys, there's a problem with hyperthreading. I've convinced myself that it is real, but I don't really have any evidence to give you, so you'll just have to believe me..."

Re:SCO....

[cperciva]

He alerted SCO to a flaw in their OS?

Actually, I posted to vendor-sec. I was rather surprised when I got an email back from SCO -- I didn't think that they'd be on vendor-sec.

Details now given.

[BrakesForElves]

Well, I just read the paper, and I applaud Colin on several levels. First off, the theory of the attack is rock-solid and well-written. Secondly, he describes very implementable OS work-arounds, crypto library fixes, and finally chip design corrections which will totally eliminate the security hole.

This is one of the best thought out, best written papers of its kind that I have read in my over thirty years of work in the engineering field.