Okay - I didn't realize that the CPU and the DMA engine can't access memory simultaneously. HOWEVER... that still doesn't make I/O ruin system performance.
First... any random disk read or write access is going to have millisecond-range latency - period. That's as fast as disks operate. (And unless you're exceptionally lucky or exceptionally clever, caches won't hide much of that latency). So when your P1 issues those commands to read some sectors from the disk, it will block for, in the best possible cases, 5ms. That's enough time for five million instructions to execute in process two. Memory busses run at ten-nanosecond latencies - there's enough time for several hundred thousand memory bus transactions. Or, tens of thousands of DMA transfers (transferring kilobytes instead of cache lines takes slightly longer). The DMA's control of the bus is rather long, yes - 4000 CPU cycles is a pretty reasonable number (which translates to no more than a microsecond or so). But compared to the time the disk access takes, the I/O-induced memory transfers' impact on the bus is completely negligible.
Disk I/Os are actually a very bad example - let me give you a better one. Gigabit ethernet - yes, a modern x86 processor can saturate one of those. (Actually, it can saturate about two or three of those running a stock OS that just does routing). Good ethernet cards - which gigabit cards are - use DMA for transfers. The memory bandwidth of the same system is only 3.2 GB/s (for DDR400), call that 20-some gigabit, of which fully 2 gigabit must be DMA-controlled packet transfers. Plus, router software is looking at a lot of data - and generating a lot of cache misses. But the DMA access doesn't seem to hurt the system very much, now does it?
My point? A single process can stall on disk I/O. In the presence of a bad disk scheduler, even multiple processes can stall on disk I/O - that's thrashing. But the problems are OS-dependent, and are not related to the architecture. Dedicated I/O processors only help I/O performance in extreme cases (like protocol offload engines) because I/O performance is primarily dominated by the hardware - the disk latencies, or network latencies - and not by the system architecture.
I/O coprocessors only make sense in two cases. First, in very high-performance workloads which are optimized so deeply that any CPU time spent working on I/O is a direct hinderance to performance. That doesn't apply to 99% of the market. Second, in cases where the work done represents such a significant fraction of the workload that having a hardware assist is valuable - for example, hardware MPEG decoders or hardware SSL accelerators or onboard hardware IP/TCP engines. Again, doesn't apply to 99% of the market.
Your process scheduling models have two deep flaws. In the first model: P1 is not restarted as soon as data is available. The OS will only preempt P2 if P2 is a distinctly lower priority than P1, in which case it's perfectly reasonable that P2 suffers - it has the lower priority. In the second model: the accounting is too fine-grained. Time shares are not allocated at the granularity of individual cycles, or even thousands of cycles. Instead, they are allocated at millions of cycles (1ms per share is extremely fine-grained on a modern OS - at least, outside the real-time world). The 4000 cycles P2 lost during the I/O on P1's behalf are negligible beside the 4 million cycles P2 is allocated - of which probably 5% (200,000 cycles) are probably lost to cache misses or TLB misses or other stalls anyways.
Because SMP requires a LOT of synchronization logic. Cache coherency most importantly, but also system support (usually through the memory controller) for the synchronization primitives used to implement locking. Locking on a uniprocessor is trivial (disable interrupts); locking on SMP is difficult (test-and-set, load-locked / store-conditional, or something more modern). Add in the logic to verify that the synchronization parts are correct...
Building a fast CPU core is easy. Building a fast CPU core that can talk to other CPU cores with a tolerable latency (hint: CPU-to-CPU communication needs to be faster than CPU-to-memory) is hard.
These chips must share a relatively slow memory bus with other devices.
No... on AMD chips the memory bus is dedicated. Intel chips have a very different system architecture (which does saturate at ~2 CPUs), but AMD gives each chip its own memory controller and memory - scales perfectly. (By the way, this isn't new... big iron (e.g. Sparc) has been doing this for years).
Currently, the fastest FSB to date is 1033MHz - almost 1/3 of the max clock speed of the processor. Given that Intel's integer units operate at twice the clock speed, the fastest parts of the chip operate at 6 times faster than memory.
That's why modern processors use pipelining (in x86, since 486's) and caches (since, uh, 8086s ?). FSB only comes into play in 1-2% of the memory accesses. But those memory accesses are pipelined, interleaved, with multiple outstanding requests issued by the out-of-order pipeline... processor designers have been working around a slow bus for years, and the FSB is only the bottleneck in extreme, pathological cases.
The monolithic, synchrous, central-processing-unit design of the architecture prohibits optimizations such as using memory controllers for block moves and having dedicated IO processors
Ever heard of DMA? A DMA controller does that memory transfer... there are 2 DMA controllers with 8 channels on your current x86 PC. Heck, high-end PCI cards even have their own onboard DMA engines (it's called bus-mastering). I/O offload? You've obviously never written a device driver... modern drivers issue a few "start" instructions, then sleep; eventually the device completes the I/O and issues an interrupt to inform the CPU it's done. The last computer I had that stalled on disk I/O was running MS-DOS - nine years ago.
In all fairness, I thought exactly the same things four years ago. Then I learned about modern computer architecture. And in today's world (and, in fact, all PCs for the past ten years), your points are completely - and utterly - irrelevant.
Me? I consider myself American. Funny how there isn't an option for that on the census forms.
I know quite a bit about my family tree. Parts of it go back to colonial times; the most recent immigrant was before World War I. I'm a little bit English, a little bit German, a little bit Irish, a little bit a few other things. But I am not culturally any one of those - I don't identify with any European country, whereas I do identify with my home town.
What still bothers me is something one of my high school teachers said to me once. "Your only liabilities on that college application is that you are white and male." The part that bothers me? It's true.
Bochs is a full-out emulator; these products are virtual machines (e.g. they run almost all code natively). Bochs will NEVER match them in terms of performance - the rule of thumb is that emulation = 10x speed drop at least, and Bochs is running ~15-20x now. Virtual machines run at almost native speed. And anyways, I've dealt with the Bochs codebase; it's already incredibly tuned, rather unextensible, and doesn't have support for anything beyond a 486-class machine. (Yes it emulates higher processors, but the rest of the system hardware is archaic by today's standards).
The only open-source software in the same performance ballpark is QEMU, which does JIT x86 binary translation. It's about as mature as... VMware was 5 years ago.
Better standard maybe... but Linux + GCC + glibc is an incredibly difficult thing to standardize upon! Standardization at the industry level takes months if not years... have you seen any one of those three stay at the same version (exact same API and ABI, no patches, etc) for any fraction of that time? Linux (or GNU/Linux if you prefer) is too much of a moving target for anyone to hit.
Go out and look at industry certifications - they certify against RHES version X, or whatever else the flavor of the month is. UNIX is a heavy-duty standard, and Linux is still evolving too fast to qualify.
Actually, I have a more interesting thought: for Linux to qualify as a Unix, development would have to slow down - a lot. Arguably, it is doing so now... Linus claims there are no really large projects on the horizon that require a 2.7 branch, there haven't been any huge changes to the kernel in quite a while (not since the new VM in late 2.4 / early 2.5, or the O(1) scheduler in 2.5). If Linux sits back at this level for a good long while, I suspect it really will become a UNIX within a year or so. Alas, such a slowdown will leave Linux a long ways behind modern Unices, which is a shame. So... should Linux slow down and become Unix-certified, or should it speed up and catch up to modern Unix?
Or you can sit down and throw some linear algebra at the problem, and come up with a formula for successive fibanacci numbers that ISN'T dependent on the previous numbers. Hint: it's a geometric series, with the slight catch that you have to use complex numbers (embedded in 2x2 matrices) instead of scalars, and thus is slightly more difficult to solve (it takes about a chalkboard of math because it can only be solved in the eigenbasis). A Fibonacci problem can be solved with a better algorithm:-).
The difficult large-scale problems have "chunky" parallel solutions - each chunk is parallel, but some chunks take longer than others (it's difficult to know how long beforehand) and the overhead of scheduling and balancing all those chunks begins to dominate the actual computation. Combining large, arbitrary, sparse matricies would do it - some multiplications will result in very little work, while some will have lots of collisions and require a lot of work.
Good riddance! Get rid of everyone who needs handholding through the most basic of data structures. Get rid of that enormous frustration of someone who can't program or debug without a full-screen, GUI debugger integrated into their IDE, with auto-complete. Get rid of code bloat, get rid of inane, duplicate features, get rid of monster runtime environments, get rid of garbage collection, get rid of "patterns", and get rid of all the crud the high-level languages mandate.
You go play with your fancy toys, and let us systems programmers go about the business of making the world work.
Go ahead. Type "java" or "python" and take the marvels of modern programming for a spin. We'll patiently nod when you come back because you need to do real work.
[For those without a sense of humor, yes, that was (mostly) tongue-in-cheek;-). (Insert right-tool-for-the-job rant here) ]
*sigh*. I usually don't mind being flamed, but for someone trying to correct me, you completely missed the mark...
I cited the original definitions (as specified in 1791). One meter is one ten-millionth of the distance from the north pole to the equator through Paris, France. Alas, the Earth's physical dimensions vary over time, and the original measurement wasn't all that accurate anyways... so the meter has been redefined as a measurement against a specific bar (late 19th century), a multiple of a light wavelength (1960s), and is now a function of the speed of light in a perfect vacuum and time (1983). The meter itself is as much an arbitrary unit as any measure before it; we just got smarter with the base-10 prefixes.
The second? Yes, you managed to find the modern definition of the second, but once again failed to consider where that definition came from. I can't recall the origin of 24 (I believe it is Greek, possibly due to 12 (hours of daylight) being a mythically complete number, being a multiple of 2, 3, 4, and 6), but the 60 seconds in a minute and 60 minutes in an hour are from the base-60 Babylonian numbering system. Completely arbitrary! However, the earth's rotational period does vary slightly (and has in fact slowed down by a handful of seconds in the past several thousand years, if I recall correctly), so scientists fixed the second at something somewhat more constant (a cesium clock). Very precise, but still a modern embedding of an absurdly arbitrary (and ancient!) constant.
And finally, SI units... you are correct, the SI unit is the kilogram. But SI is not, in fact, the metric system - SI is very modern, while the metric system is over two hundred years old. When defined, both the kilogram (a cubic decimeter) and gram (a cubic centimeter) were set simultaneously - at the freezing point of water (0 celcius). The temperature was later moved to 4 celcius for exactly the reason you describe, yet that is still not entirely accurate - the full definition is 4 celcius at one atmosphere due to pressure effects on density (and defining one atmosphere is another matter entirely...). [As a curiosity, the kilogram is the most unstable of the fundamental measures - it is the only one still defined with respect to a specific object.]
Getting back on topic at long last... the rest of the SI units are so phenomenally screwy that having 1024 bytes to a kilobyte, and so on, is downright sane. (What's not sane, however, is that byte RATES, e.g. 1 Mb/sec, are powers of ten! See any networking text for confirmation...)
OLTP is the classic anti-cluster workload. Essentially random data access patterns, very large resident data sets with a huge amount of simultaneous (and synchronous) accesses. OLTP means low-latency, and OLTP will die on a cluster. By definition.
Now sure, some careful planning can take an OLTP system and make it more cleanly distributed, but at that point it isn't OLTP, because all the nasty bits that made it a hard workload are washed out. Running a constantly-changing database (e.g. financial market?) on a cluster is hard; running a mostly static database (e.g. shopping cart?) is easy.
However, I agree with your point. Very few people need the 32-cpu monster (although there are a few!). Handling transaction volume can be done two ways: buy a big general-purpose machine that can handle the volume, or buy a cheaper cluster that more closely matches the workload. And today, the cluster is the right answer.
I think the difference between then and now is that before, we didn't know what the workload was supposed to be. In that case, a big general-purpose monster server is the most flexible solution. But now, we know what workload we want, and it's cheaper to design a cluster for that workload.
Caveat: I think MPI itself is very recent (standardized only w/in the past few years), before that everyone used custom message-passing libraries.
It's a tradeoff. MPI is "preferred" because a properly written MPI program will run on both clusters and shared-memory equally fast, because all communication is explicit. It's also much harder to program, because all communication must be made explicit.
Shared-memory (e.g. pthreads) is easier to program in the first place (since you don't have to think about as many sharing issues) and more portable. However, it is very error-prone - get a little bit off on the cache alignment or contend too much for a lock, and you've lost much of the performance gain. And it can't run it on a cluster without horrible performance loss.
If it's the difference between spending two months writing the shared-memory sim and four months writing the message-passing sim that runs two times faster on cheaper hardware, well, which would you choose? Is the savings in CPU time worth the investment in programmer time?
Alas, the latencies on a 1024-way machine are pretty bad anyway. If they use the same interconnect as the SGI Origin, it's 300-3000 cycles for each interconnect transaction (depending on distance and number of hops in the transaction). Technically that's low-latency... but drop below 32 processors or so, and the interconnect is a bus with 100 cycle latencies, so those extra processors cause a lot of lost cycles.
Sun very well could support that many CPUs. Sun just doesn't sell hardware that has that many (and therefor won't claim to support that many) - mainly because that kind of hardware is so expensive as to make SPARC look cheap!
My opinion is that Linux on a 1024-way is a spectacularly stupid idea, introduced more for the sexiness of having a 1024-way machine than for any practical benefits. Linux is simply not designed for scaling that large. And there is a huge difference between an OS designed to scale that large, and an OS hacked up to support something that large, without actually making the appropriate design choices. SGI may know about those choices (and probably better than Sun), but I highly doubt they'd throw them into a GPLed Linux kernel - they still want to sell their own version of Unix!
I expect (yes, a wild pie-in-the-sky guess) that the advantage of a 1024-way machine over a 512-way machine, both running Linux, is going to be maybe 20-30% performance, far from the 100% the numbers might claim or the 70-80% that might be tolarable. For a supercomputer where that 20-30% is irrelevant because no other machine can crunch the data, cool; for everyone else, two 512-ways running unconnected will be better, cheaper, and faster. [At least, until Linux can scale that large... maybe in 5 years or so?]
That means that if I were to spend the same, lifecycle-wise, on a Linux cluster as I would on a big Sun box like the 10k or 15k, I'd end up with 12x to 16x the performance of the Sun solution.
And if you are trying to use a 10k or 15k for a job where a simple cluster will suffice, you DESERVE to be fleeced! If the application is simple enough to run on a cluster, where parallelism means high-latency message passing, then you DON'T need big iron to run it. You know this, I know this... yet people out there keep buying 15k's and paying three or four times as much as an equivalently powerful cluster would cost.
Big iron really is a niche market - they only belong in specific scenarios where the latencies of a cluster are too large and the applications themselves really need shared-memory. A really heavy-duty database qualifies, simulations with large (multi-gigabyte) active datasets qualify, rendering qualifies, OLTP-type apps qualify. Your web server cluster doesn't, your "one giant machine to run everything" really doesn't, and a sufficiently distributed database (like Google or some other well-designed database) doesn't.
Honestly, to hear someone complaining about how expensive a 15k is, and how nice the replacement cluster is, tells me that some purchaser in the past was stupid. It's like buying an SUV to commute in, and then complaining about the excessive gas consumption. Well, duh!
Okay, since you've felt the need to hype DProbes at least a couple of times, I'll pull from the DTrace USENIX paper (link was somewhere up above).
The major difference: a huge amount of polish. DProbes can crash your system if used incorrectly; DTrace is guaranteed safe. DProbes reports just about everything; DTrace supports speculative tracing and predicates so you can narrow down the fire-hose of profiling data into the useful information. DProbes doesn't scale well across multiple CPUs; DTrace does. (This last one is just like Linux versus Solaris: one scales, the other doesn't. Sorry!)
So, yes. If IBM had spent the past four years tuning DProbes and turning it into a useful tool, they might have something. As it is, Sun spent quite a bit of time and money, and ACTUALLY CREATED a very powerful tool.
[rant]What is increasingly bugging me about open source advocates on Slashdot is how they'll look at any (ANY!) closed-source tool and say "oh, open-source tool XYZ does this / did this a few years ago / will make you walk on water". These advocates will cheerfully ignore that the closed-source tool has months or years of additional polish that have brought it from an internal company toy project into a commercially viable tool - that the open source tool of now is no more advanced than the closed source tool of years ago, except that most open source tools are stagnant and will never leave development and never obtain the degree of polish wide-scale use demands.
A message to open-source advocates: if you don't spend the time turning your hobbyist code into solid, professional code, the closed-source companies that DO spend the time will leave you in the dust - freedom of code (both free as in beer and free as in GNU) notwithstanding.[/rant]
I'll be blunt: Shark and DTrace are two entirely different animals.
Looking at that Apple blurb (and reports I heard from WWDC attendees), Shark is a performance tweaking tool - a profiler. If you had ever bought the heavy-duty Sun compilers, you'd have a functionally similar set of tools (collect, analyzer, and cc with various flags that generates annotated source). Admittedly not nearly as polished as Apple's tools (and not as cheap), but Shark is also several years newer - and having done exactly the optimization that article describes on my own code with the Sun tools, I assure you they have been around for quite a while.
However, DTrace is NOT an application profiler - though it can be used for that. And claiming that it is a profiler would be like claiming C is just an extension to assembly language. It could be used as such, but you'd be ignoring all the power of the tool.
DTrace is far more powerful: it is a whole system profiling tool. It can give debugging data, performance data, or whatever, for any process, thread, driver, kernel module, or other system component. It includes a moderately complex programming language - YOU get to write code that runs in REAL TIME within any code path anywhere in the system. And, you get zero performance loss in any part of the system that isn't actively being traced.
The DTrace manual is extremely informative. Of course, actually trying it on Solaris 10 Beta is even more so:-)
To all the clowns posting here about problem X, Y, and Z in the article, and about how the reporter just discovered the page file...
This is the media version of an academic paper for USENIX Security '04. It glosses over a lot of details.
Examples:
- mlock(). Available to root only under Linux, so useless outside of setuid programs - and we all have so many of those we trust, right?
- VirtualLock()/VirtualUnlock(). Win32 versions of mlock(). Not implemented in the 9x series, advisory in a few other Windowses (I can't find the docs on where, but it's in the original paper).
- zeroing memory. Oops, your optimizing compiler just optimized away that memset() call as dead code. This was a known flaw in some crypto libraries a few years ago.
The system described is a whole-system simulator, it traces bytes of input from the moment they pass the keyboard through the kernel, into the user-mode applications that use the bytes (e.g. kernel to X server to Mozilla), and how long those bytes hang around in the physical RAM of the machine.
This does not necessarily describe a highly practical attack, but more a quantification of how vunerable systems are to such an attack. In fact, the original paper is about data lifetime information.
- Did you know the most recent 4K keystrokes (passwords included) are stored in the kernel's tty buffer?
- Did you know several dozen of your keystrokes are stored in the Linux kernel's entropy buffer (for random number generation)? They aren't actually consumed for as long as several hours.
Linux requires you to be root to lock pages in RAM (it's called mlock()), Windows has a similar method (VirtualLock() / VirtualUnlock()) that are unimplemented in old versions of Windows and advisory in some others.
Yes the OS has this feature - no, the feature doesn't work as expected. (And you are exactly right - a reason to not quite fully implement this feature is that DoS attack).
Actually... mlock() is a POSIX call that isn't native to Windows. The native methods, VirtualLock() and VirtualUnlock(), "are simply advisory" according to the Win32 docs - which suggests to me that even mlock() is advisory under Win32.
memset() has another accidental bug - under high optimization, the oh-so-clever compiler optimizes away your memset call because it's "dead code". This flaw affected a lot of crypto libraries some time ago. Are you really sure that those bytes were overwritten?
Of course, the reporter glossed over these details:-)
Much more involved than that, I assure you.
The question is, how long does that information sit around in RAM? There are a few tricks to prevent memory from being swapped out (note: they aren't guaranteed to work on some OSes, like Windows!), which is the page file you describe, but our tracking was at the level of bytes floating around memory.
Potential sources of leaks:
1) Laptop suspend-to-disk writes out your entire RAM state to the hard drive. Care to bet on the safety of the encryption? I wouldn't...
2) you are running some sort of virtual machine, which you log out of and then suspend. Someone obtains the suspended memory image.
3) Public kiosk. You walk up, log in, fool around, log off, and walk away. The person behind you comes up, logs in with root privilages, then starts shifting around in the kernel's memory looking for anything you typed. The person doesn't actually have to have root - maybe he has a root exploit.
Yes, the system is much more involved (I'm quite sure - I wrote parts of it!).;-)
At the same time, I agree it's a pretty hard attack to mount. The catch is, it's finally possible to mount the attack in the first place - and no one is studying the risks involved. I too think the risks are small - we were trying to quantify them.
Close, but not quite. Missing one thing: if you only have read-only access to the hard drive (for some reason or another).
1) Maybe you found some sort of image file sitting on a server.
2) Maybe you stole someone's computer. Keylogger is useless because that person's not using his/her computer again. It's pretty easy to encrypt the filesystem - what about other channels (like suspend-to-disk?)?
Worse, the systems ended up with the whole password in cleartext. Try it - run "strings | wc" on/proc/kmem as root, see how much comes up. It's only a couple thousand strings, and passwords were sitting around in them plain as day.
Much as this is good for Paul Parenthood, what about Tom Temp? He gets hired on knowing his job will disappear in two years, gets minimal training (he's gone in two years) and has no career prospects. As much as I sympathize with Paul Parenthood, your proposed fix is worse than the original problem! There is a very good reason the United States wants to stay as far from socialism as possible. - in attempting to be nice to Paul, you're exploiting Tom even worse.
Slashdot Mirror
First ... any random disk read or write access is going to have millisecond-range latency - period. That's as fast as disks operate. (And unless you're exceptionally lucky or exceptionally clever, caches won't hide much of that latency). So when your P1 issues those commands to read some sectors from the disk, it will block for, in the best possible cases, 5ms. That's enough time for five million instructions to execute in process two. Memory busses run at ten-nanosecond latencies - there's enough time for several hundred thousand memory bus transactions. Or, tens of thousands of DMA transfers (transferring kilobytes instead of cache lines takes slightly longer). The DMA's control of the bus is rather long, yes - 4000 CPU cycles is a pretty reasonable number (which translates to no more than a microsecond or so). But compared to the time the disk access takes, the I/O-induced memory transfers' impact on the bus is completely negligible.
Disk I/Os are actually a very bad example - let me give you a better one. Gigabit ethernet - yes, a modern x86 processor can saturate one of those. (Actually, it can saturate about two or three of those running a stock OS that just does routing). Good ethernet cards - which gigabit cards are - use DMA for transfers. The memory bandwidth of the same system is only 3.2 GB/s (for DDR400), call that 20-some gigabit, of which fully 2 gigabit must be DMA-controlled packet transfers. Plus, router software is looking at a lot of data - and generating a lot of cache misses. But the DMA access doesn't seem to hurt the system very much, now does it?
My point? A single process can stall on disk I/O. In the presence of a bad disk scheduler, even multiple processes can stall on disk I/O - that's thrashing. But the problems are OS-dependent, and are not related to the architecture. Dedicated I/O processors only help I/O performance in extreme cases (like protocol offload engines) because I/O performance is primarily dominated by the hardware - the disk latencies, or network latencies - and not by the system architecture.
I/O coprocessors only make sense in two cases. First, in very high-performance workloads which are optimized so deeply that any CPU time spent working on I/O is a direct hinderance to performance. That doesn't apply to 99% of the market. Second, in cases where the work done represents such a significant fraction of the workload that having a hardware assist is valuable - for example, hardware MPEG decoders or hardware SSL accelerators or onboard hardware IP/TCP engines. Again, doesn't apply to 99% of the market.
Your process scheduling models have two deep flaws. In the first model: P1 is not restarted as soon as data is available. The OS will only preempt P2 if P2 is a distinctly lower priority than P1, in which case it's perfectly reasonable that P2 suffers - it has the lower priority. In the second model: the accounting is too fine-grained. Time shares are not allocated at the granularity of individual cycles, or even thousands of cycles. Instead, they are allocated at millions of cycles (1ms per share is extremely fine-grained on a modern OS - at least, outside the real-time world). The 4000 cycles P2 lost during the I/O on P1's behalf are negligible beside the 4 million cycles P2 is allocated - of which probably 5% (200,000 cycles) are probably lost to cache misses or TLB misses or other stalls anyways.
Building a fast CPU core is easy. Building a fast CPU core that can talk to other CPU cores with a tolerable latency (hint: CPU-to-CPU communication needs to be faster than CPU-to-memory) is hard.
No... on AMD chips the memory bus is dedicated. Intel chips have a very different system architecture (which does saturate at ~2 CPUs), but AMD gives each chip its own memory controller and memory - scales perfectly. (By the way, this isn't new ... big iron (e.g. Sparc) has been doing this for years).
Currently, the fastest FSB to date is 1033MHz - almost 1/3 of the max clock speed of the processor. Given that Intel's integer units operate at twice the clock speed, the fastest parts of the chip operate at 6 times faster than memory.
That's why modern processors use pipelining (in x86, since 486's) and caches (since, uh, 8086s ?). FSB only comes into play in 1-2% of the memory accesses. But those memory accesses are pipelined, interleaved, with multiple outstanding requests issued by the out-of-order pipeline ... processor designers have been working around a slow bus for years, and the FSB is only the bottleneck in extreme, pathological cases.
The monolithic, synchrous, central-processing-unit design of the architecture prohibits optimizations such as using memory controllers for block moves and having dedicated IO processors
Ever heard of DMA? A DMA controller does that memory transfer ... there are 2 DMA controllers with 8 channels on your current x86 PC. Heck, high-end PCI cards even have their own onboard DMA engines (it's called bus-mastering). I/O offload? You've obviously never written a device driver... modern drivers issue a few "start" instructions, then sleep; eventually the device completes the I/O and issues an interrupt to inform the CPU it's done. The last computer I had that stalled on disk I/O was running MS-DOS - nine years ago.
In all fairness, I thought exactly the same things four years ago. Then I learned about modern computer architecture. And in today's world (and, in fact, all PCs for the past ten years), your points are completely - and utterly - irrelevant.
323475 A "Stop 0xc0000244" Error Occurs When You Audit Policy Changes If CrashOnAuditFail Is Turned On
So they fixed it... but why exactly is there a setting to deliberately crash? What ever happend to returning errors?
(Note: no, I haven't looked up this setting and there may be a perfectly logical explanation. But a setting to crash the computer ... hahaha!)
I know quite a bit about my family tree. Parts of it go back to colonial times; the most recent immigrant was before World War I. I'm a little bit English, a little bit German, a little bit Irish, a little bit a few other things. But I am not culturally any one of those - I don't identify with any European country, whereas I do identify with my home town.
What still bothers me is something one of my high school teachers said to me once. "Your only liabilities on that college application is that you are white and male." The part that bothers me? It's true.
The only open-source software in the same performance ballpark is QEMU, which does JIT x86 binary translation. It's about as mature as ... VMware was 5 years ago.
Go out and look at industry certifications - they certify against RHES version X, or whatever else the flavor of the month is. UNIX is a heavy-duty standard, and Linux is still evolving too fast to qualify.
Actually, I have a more interesting thought: for Linux to qualify as a Unix, development would have to slow down - a lot. Arguably, it is doing so now ... Linus claims there are no really large projects on the horizon that require a 2.7 branch, there haven't been any huge changes to the kernel in quite a while (not since the new VM in late 2.4 / early 2.5, or the O(1) scheduler in 2.5). If Linux sits back at this level for a good long while, I suspect it really will become a UNIX within a year or so. Alas, such a slowdown will leave Linux a long ways behind modern Unices, which is a shame. So... should Linux slow down and become Unix-certified, or should it speed up and catch up to modern Unix?
The difficult large-scale problems have "chunky" parallel solutions - each chunk is parallel, but some chunks take longer than others (it's difficult to know how long beforehand) and the overhead of scheduling and balancing all those chunks begins to dominate the actual computation. Combining large, arbitrary, sparse matricies would do it - some multiplications will result in very little work, while some will have lots of collisions and require a lot of work.
You go play with your fancy toys, and let us systems programmers go about the business of making the world work.
Go ahead. Type "java" or "python" and take the marvels of modern programming for a spin. We'll patiently nod when you come back because you need to do real work.
[For those without a sense of humor, yes, that was (mostly) tongue-in-cheek ;-). (Insert right-tool-for-the-job rant here) ]
I cited the original definitions (as specified in 1791). One meter is one ten-millionth of the distance from the north pole to the equator through Paris, France. Alas, the Earth's physical dimensions vary over time, and the original measurement wasn't all that accurate anyways... so the meter has been redefined as a measurement against a specific bar (late 19th century), a multiple of a light wavelength (1960s), and is now a function of the speed of light in a perfect vacuum and time (1983). The meter itself is as much an arbitrary unit as any measure before it; we just got smarter with the base-10 prefixes.
The second? Yes, you managed to find the modern definition of the second, but once again failed to consider where that definition came from. I can't recall the origin of 24 (I believe it is Greek, possibly due to 12 (hours of daylight) being a mythically complete number, being a multiple of 2, 3, 4, and 6), but the 60 seconds in a minute and 60 minutes in an hour are from the base-60 Babylonian numbering system. Completely arbitrary! However, the earth's rotational period does vary slightly (and has in fact slowed down by a handful of seconds in the past several thousand years, if I recall correctly), so scientists fixed the second at something somewhat more constant (a cesium clock). Very precise, but still a modern embedding of an absurdly arbitrary (and ancient!) constant.
And finally, SI units... you are correct, the SI unit is the kilogram. But SI is not, in fact, the metric system - SI is very modern, while the metric system is over two hundred years old. When defined, both the kilogram (a cubic decimeter) and gram (a cubic centimeter) were set simultaneously - at the freezing point of water (0 celcius). The temperature was later moved to 4 celcius for exactly the reason you describe, yet that is still not entirely accurate - the full definition is 4 celcius at one atmosphere due to pressure effects on density (and defining one atmosphere is another matter entirely...). [As a curiosity, the kilogram is the most unstable of the fundamental measures - it is the only one still defined with respect to a specific object.]
Getting back on topic at long last... the rest of the SI units are so phenomenally screwy that having 1024 bytes to a kilobyte, and so on, is downright sane. (What's not sane, however, is that byte RATES, e.g. 1 Mb/sec, are powers of ten! See any networking text for confirmation...)
1 meter = 1 ten-millionth of the distance from the north pole to the equator, through Paris, France.
1 gram = mass of one cubic centimeter of water... at 4 DEGREES CELCIUS.
1 second ... well, the second was in the Imperial system first, and it still follows the same units.
Now sure, some careful planning can take an OLTP system and make it more cleanly distributed, but at that point it isn't OLTP, because all the nasty bits that made it a hard workload are washed out. Running a constantly-changing database (e.g. financial market?) on a cluster is hard; running a mostly static database (e.g. shopping cart?) is easy.
However, I agree with your point. Very few people need the 32-cpu monster (although there are a few!). Handling transaction volume can be done two ways: buy a big general-purpose machine that can handle the volume, or buy a cheaper cluster that more closely matches the workload. And today, the cluster is the right answer.
I think the difference between then and now is that before, we didn't know what the workload was supposed to be. In that case, a big general-purpose monster server is the most flexible solution. But now, we know what workload we want, and it's cheaper to design a cluster for that workload.
It's a tradeoff. MPI is "preferred" because a properly written MPI program will run on both clusters and shared-memory equally fast, because all communication is explicit. It's also much harder to program, because all communication must be made explicit.
Shared-memory (e.g. pthreads) is easier to program in the first place (since you don't have to think about as many sharing issues) and more portable. However, it is very error-prone - get a little bit off on the cache alignment or contend too much for a lock, and you've lost much of the performance gain. And it can't run it on a cluster without horrible performance loss.
If it's the difference between spending two months writing the shared-memory sim and four months writing the message-passing sim that runs two times faster on cheaper hardware, well, which would you choose? Is the savings in CPU time worth the investment in programmer time?
Alas, the latencies on a 1024-way machine are pretty bad anyway. If they use the same interconnect as the SGI Origin, it's 300-3000 cycles for each interconnect transaction (depending on distance and number of hops in the transaction). Technically that's low-latency... but drop below 32 processors or so, and the interconnect is a bus with 100 cycle latencies, so those extra processors cause a lot of lost cycles.
My opinion is that Linux on a 1024-way is a spectacularly stupid idea, introduced more for the sexiness of having a 1024-way machine than for any practical benefits. Linux is simply not designed for scaling that large. And there is a huge difference between an OS designed to scale that large, and an OS hacked up to support something that large, without actually making the appropriate design choices. SGI may know about those choices (and probably better than Sun), but I highly doubt they'd throw them into a GPLed Linux kernel - they still want to sell their own version of Unix!
I expect (yes, a wild pie-in-the-sky guess) that the advantage of a 1024-way machine over a 512-way machine, both running Linux, is going to be maybe 20-30% performance, far from the 100% the numbers might claim or the 70-80% that might be tolarable. For a supercomputer where that 20-30% is irrelevant because no other machine can crunch the data, cool; for everyone else, two 512-ways running unconnected will be better, cheaper, and faster. [At least, until Linux can scale that large... maybe in 5 years or so?]
And if you are trying to use a 10k or 15k for a job where a simple cluster will suffice, you DESERVE to be fleeced! If the application is simple enough to run on a cluster, where parallelism means high-latency message passing, then you DON'T need big iron to run it. You know this, I know this... yet people out there keep buying 15k's and paying three or four times as much as an equivalently powerful cluster would cost.
Big iron really is a niche market - they only belong in specific scenarios where the latencies of a cluster are too large and the applications themselves really need shared-memory. A really heavy-duty database qualifies, simulations with large (multi-gigabyte) active datasets qualify, rendering qualifies, OLTP-type apps qualify. Your web server cluster doesn't, your "one giant machine to run everything" really doesn't, and a sufficiently distributed database (like Google or some other well-designed database) doesn't.
Honestly, to hear someone complaining about how expensive a 15k is, and how nice the replacement cluster is, tells me that some purchaser in the past was stupid. It's like buying an SUV to commute in, and then complaining about the excessive gas consumption. Well, duh!
The major difference: a huge amount of polish. DProbes can crash your system if used incorrectly; DTrace is guaranteed safe. DProbes reports just about everything; DTrace supports speculative tracing and predicates so you can narrow down the fire-hose of profiling data into the useful information. DProbes doesn't scale well across multiple CPUs; DTrace does. (This last one is just like Linux versus Solaris: one scales, the other doesn't. Sorry!)
So, yes. If IBM had spent the past four years tuning DProbes and turning it into a useful tool, they might have something. As it is, Sun spent quite a bit of time and money, and ACTUALLY CREATED a very powerful tool.
[rant]What is increasingly bugging me about open source advocates on Slashdot is how they'll look at any (ANY!) closed-source tool and say "oh, open-source tool XYZ does this / did this a few years ago / will make you walk on water". These advocates will cheerfully ignore that the closed-source tool has months or years of additional polish that have brought it from an internal company toy project into a commercially viable tool - that the open source tool of now is no more advanced than the closed source tool of years ago, except that most open source tools are stagnant and will never leave development and never obtain the degree of polish wide-scale use demands.
A message to open-source advocates: if you don't spend the time turning your hobbyist code into solid, professional code, the closed-source companies that DO spend the time will leave you in the dust - freedom of code (both free as in beer and free as in GNU) notwithstanding.[/rant]
Looking at that Apple blurb (and reports I heard from WWDC attendees), Shark is a performance tweaking tool - a profiler. If you had ever bought the heavy-duty Sun compilers, you'd have a functionally similar set of tools (collect, analyzer, and cc with various flags that generates annotated source). Admittedly not nearly as polished as Apple's tools (and not as cheap), but Shark is also several years newer - and having done exactly the optimization that article describes on my own code with the Sun tools, I assure you they have been around for quite a while.
However, DTrace is NOT an application profiler - though it can be used for that. And claiming that it is a profiler would be like claiming C is just an extension to assembly language. It could be used as such, but you'd be ignoring all the power of the tool.
DTrace is far more powerful: it is a whole system profiling tool. It can give debugging data, performance data, or whatever, for any process, thread, driver, kernel module, or other system component. It includes a moderately complex programming language - YOU get to write code that runs in REAL TIME within any code path anywhere in the system. And, you get zero performance loss in any part of the system that isn't actively being traced.
The DTrace manual is extremely informative. Of course, actually trying it on Solaris 10 Beta is even more so :-)
Origin: it's one of the dis-inspiration posters.
This is the media version of an academic paper for USENIX Security '04. It glosses over a lot of details.
Examples:
- mlock(). Available to root only under Linux, so useless outside of setuid programs - and we all have so many of those we trust, right?
- VirtualLock()/VirtualUnlock(). Win32 versions of mlock(). Not implemented in the 9x series, advisory in a few other Windowses (I can't find the docs on where, but it's in the original paper).
- zeroing memory. Oops, your optimizing compiler just optimized away that memset() call as dead code. This was a known flaw in some crypto libraries a few years ago.
The system described is a whole-system simulator, it traces bytes of input from the moment they pass the keyboard through the kernel, into the user-mode applications that use the bytes (e.g. kernel to X server to Mozilla), and how long those bytes hang around in the physical RAM of the machine.
This does not necessarily describe a highly practical attack, but more a quantification of how vunerable systems are to such an attack. In fact, the original paper is about data lifetime information.
- Did you know the most recent 4K keystrokes (passwords included) are stored in the kernel's tty buffer?
- Did you know several dozen of your keystrokes are stored in the Linux kernel's entropy buffer (for random number generation)? They aren't actually consumed for as long as several hours.
Yes the OS has this feature - no, the feature doesn't work as expected. (And you are exactly right - a reason to not quite fully implement this feature is that DoS attack).
Under some versions of Windows, VirtualLock is simply advisory. It wasn't even implemented in 95/98/ME.
memset() has another accidental bug - under high optimization, the oh-so-clever compiler optimizes away your memset call because it's "dead code". This flaw affected a lot of crypto libraries some time ago. Are you really sure that those bytes were overwritten?
Of course, the reporter glossed over these details :-)
Much more involved than that, I assure you. The question is, how long does that information sit around in RAM? There are a few tricks to prevent memory from being swapped out (note: they aren't guaranteed to work on some OSes, like Windows!), which is the page file you describe, but our tracking was at the level of bytes floating around memory. Potential sources of leaks: 1) Laptop suspend-to-disk writes out your entire RAM state to the hard drive. Care to bet on the safety of the encryption? I wouldn't... 2) you are running some sort of virtual machine, which you log out of and then suspend. Someone obtains the suspended memory image. 3) Public kiosk. You walk up, log in, fool around, log off, and walk away. The person behind you comes up, logs in with root privilages, then starts shifting around in the kernel's memory looking for anything you typed. The person doesn't actually have to have root - maybe he has a root exploit. Yes, the system is much more involved (I'm quite sure - I wrote parts of it!). ;-)
At the same time, I agree it's a pretty hard attack to mount. The catch is, it's finally possible to mount the attack in the first place - and no one is studying the risks involved. I too think the risks are small - we were trying to quantify them.
Close, but not quite. Missing one thing: if you only have read-only access to the hard drive (for some reason or another). 1) Maybe you found some sort of image file sitting on a server. 2) Maybe you stole someone's computer. Keylogger is useless because that person's not using his/her computer again. It's pretty easy to encrypt the filesystem - what about other channels (like suspend-to-disk?)? Worse, the systems ended up with the whole password in cleartext. Try it - run "strings | wc" on /proc/kmem as root, see how much comes up. It's only a couple thousand strings, and passwords were sitting around in them plain as day.
Much as this is good for Paul Parenthood, what about Tom Temp? He gets hired on knowing his job will disappear in two years, gets minimal training (he's gone in two years) and has no career prospects. As much as I sympathize with Paul Parenthood, your proposed fix is worse than the original problem! There is a very good reason the United States wants to stay as far from socialism as possible. - in attempting to be nice to Paul, you're exploiting Tom even worse.