Then don't compile it in while you're debugging. Most *users* don't debug their kernels.
If a user gets an oops and submits a bug report to linux-kernel while running preempt, the bug report is a lot harder to decipher.
True, but the SMP support introduced these to begin with. The preemptive patches just bring that danger to UP machines.
Not true. preempt introduces new hangs. Read the threads on linux-kernel, especially "Re: [2.4.17/18pre] VM and swap - it's really unusable".
Some things that are broken by preempt:
* Network drivers which disable IRQs to avoid spinlocking on uniprocs (major performance win)
* Drivers which use per-CPU data to avoid spinlocking at all
* Drivers which disable individual interrupts for long periods of time
* Drivers which depend on consecutive lines of code executing near each other in time, especially serial drivers
That thread has details on all of these.
And there's the priority inversion scenario:
SCHED_OTHER process 1 acquires a semaphore in kernel mode
SCHED_FIFO realtime process needs the semaphore, blocks on it
Now the rt process is stuck pending progress from the SCHED_OTHER process. Without preemption, the SCHED_OTHER process would have done whatever required the semaphore and released it. Now, SCHED_OTHER process 2, 3,... n may be scheduled, leaving the realtime process twisting in the wind until the system eventually gets around to rescheduling process 1. If process 1 is e.g. SETIatHome or another nice 19 CPU hog, and process 2 is mozilla, you can get huge latencies on the realtime process, way bigger than without preempt. And even without SCHED_OTHER you run into the same problem, it's just not quite as easy to illustrate.
[Note that I said semaphore, not spinlock, so the lock-break code won't help.]
Do you not remember all the problems with priority inversion and the SCHED_IDLE patches? This is exactly the same problem, it's not like it's something new and mysterious that people are making up as FUD to stop preempt from getting in to the kernel. Any introductory OS textbook discusses it, and priority inheritance is the only robust way to eliminate it--with all the problems that come along with priority inheritance.
4. It doesn't improve the worst-case latency.
>>>>>>>>>>>
It's not designed to. Love has already started work on a lock-breaking patch to get rid of long-held locks It won't help. The fundamental problem is that preempt in interrupts is impossible with this scheme (and AFAIK nobody has proposed a scheme that could even theoretically work), and the worst-case latencies are in interrupts. The secondary problem is that a lot of these issues are highly hardware sensitive; different hardware has different timing requirements, and the only way to find them all is to audit every driver and deal with it appropriately (as, e.g., the LL patches do).
As Alan Cox wrote,
The pre-emption patch doesn't change the average latencies. Go run some real benchmarks. Its lost in the noise after the low latency patch. A single inw from some I/O cards can cost as much as the latency target we hit.
Its not a case of the 90% of the result with 10% of the work, the pre-empt patch is firmly in the all pain no gain camp
Those that *need* LL for soft-realtime and those that *want* a LL-like environment so they have click a button more timely.
Well, I think that some sort of soft realtime is wanted pretty widely, if only to stop music and movies from skipping. LL does that fairly well, O(1) doesn't. But O(1) is definitely a win, modulo a few nits, and it's already been integrated into 2.5.x--with some work on those nits courtesy of Davide Libenzi.
So, I think O(1) is good but it's not a replacement for LL or another worst-case latency reducer, even among people who don't know what "soft realtime" means. They still don't want their mp3s to skip.
The preemption patch is available now, and works now, and I am using it now.
The problems with the preempt patch are several:
1. It makes kernel debugging considerably harder
2. It introduces potential livelocks which could hang the system (this is an inherent part of the design and not an implementation bug)
3. It doesn't work with SMP (yet), this is fixable
4. It doesn't improve the worst-case latency.
#4 means it isn't a viable alternative to low-latency style patches; Love has said as much on linux-kernel, though obviously he feels it's a nice complement. 1 is a large hill to climb on its own, and 2 basically dooms it unless a clean solution can be found (most proposed solutions involve some sort of priority inheritance, which is a rat's nest).
In other words, the patch "works" only in a very limited sense; it doesn't decrease worst-case latency, can potentially hang the system, and doesn't interact well with SMP. It does decrease average-case latency.
The preemption patch is comparatively small and elegant.
Uh, that's in the eye of the beholder I guess. It is small, but it's not really elegant; it essentially causes conditional reschedules to happen all over the place, and by design cannot deal with latency in noninterruptable interrupt contexts--which happen to be exactly where the worst-case latencies are. So preempt as it's currently designed simply cannot address the worst-case latency. And it causes far more runtime conditional reschedules than the LL patches.
The low-latency patch decreases both worst and average-case latencies. It also doesn't have livelocks as an inherent part of its design. And average-case latencies with LL are as good as those with LL+preempt (in other words, preempt gains you nothing over LL). LL, unfortunately, touches a lot of code and has conditional schedules sprinkled all over the place--it's not elegant either, and certainly not small. But unlike preempt, it actually decreases the worst-case latency, which is what's important to the soft realtime applications that gave birth to both patches. There's a mini-LL patch as well which has far fewer conditional rescheds, that's being discussed for potential inclusion in 2.4.x and/or 2.5.x
My gut feeling is that preempt is the wrong approach to the latency problem, but the jury is still out on it. I don't think either the current preempt or the current LL are ready to go in the kernel. If you're doing e.g sound mixing or other latency-sensitive work right now, the full LL patches seem like the best alternative available. They get latencies down extremely low, well below those of e.g. BeOS.
Sure. But that goes both ways. _If_ you know asm and C well, going from kernel code to GUI code has a lot of pitfalls (as does the opposite). Or going to numerical recipe code, or whatever.
But fundamentally, code is code and rather than be intimidated you should start reading and playing with it. Even if you never become a hardcore kernel hacker, it's another world that's definitely worth jumping into and not nearly as scary as you think.
Alan's Gearheads Only articles in Linux Magazine are a great way to start out.
(FWIW, I posted mainly as a gedanken piece and not because I think you're way off base; sometimes you see a quote that just seems applicable to another conversation...)
From Alan Cox, in the kerneltrap interview running on the front page of slashdot right now:
Ignore everyone who tells you kernel hacking is hard, special or different. It's a large program, and bug fixing or driver tweaking can be a best starting point. It is however not magic, nor written in a secret language that only deep initiates with beards can read.
Play with it, try things, break it horribly and enjoy yourself. I started on the networking code because it didn't work very well. Everything I knew about TCP/IP I had downloaded the same day I started hacking the net code. My first attempts were not pretty but it was *fun*.
I still don't see any reason that processes are inherently more efficient than threads.
Right, that's why I said that for some things threads are better. But a lot of people pick threads over processes for reasons other than "I need to share all my memory". Which is bad. Choosing to use threads should be a conscious decision, made only when you've determined that e.g. processes with a small shared memory segment, message passing, or other approaches are worse.
Well, with paper junkmai the sender has to pay for it (rather than having it subsidized by the recipient). It keeps your postal rates low, and it encourages the sender to target the mailings to people who might be interested (e.g. open-source mailings to people having licensing trouble...) because they have to pay for every piece of mail sent.
I've not had to confront an employer about this explicitly, but having flexible hours is on my list of things I require when doing a job search. For the right incentives it might be negotiable, but in my area traffic is so bad that I need to leave the house at 8:00 to get to work by 9:00 but I can leave at 9:00 and be in by 9:20 easily. 40 minutes a day in traffic adds up, and for people who live further away it's even more important.
The two other big items on my "must have" list before taking a job are 1) no cell phone/pager--when I leave work, I'm on _my_ time until I next come int--and 2) reasonable IP terms in the employee agreement so that software I develop on my own time without using company resources is mine and mine alone.
"The advantage of the virtual server via NAT is that real servers can run any operating system that supports TCP/IP protocol, real servers can use private Internet addresses, and only an IP address is needed for the load balancer"
So you can keep your application on Solaris with LVS.
First off, the Kegel's c10k page referenced earlier is definitely worth a read. And if you're under the impression that having 2^16 TCP port numbers limits you to 2^16 connections, that's not accurate. You can have hundreds of thousands of connections to one machine, presuming you manage them properly (as the c10k page points out).
More importantly, you should check out http://www.linuxvirtualserver.org/ ; it's aim is exactly what you want:
"The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the cluster is transparent to end users. End users only see a single virtual server."
This is a terrible idea. The fact is, for 99% of applications the scheduler isn't a bottleneck. Even on Alan Cox's worst-case 8-way SMP machines the scheduler doesn't eat tremendous amounts of CPU; _way_ more than it should, but an approach like Ingo Molnar's or Davide Libenzi's will get rid of most of that overhead.
But the real reason it's a terrible idea is because of the cache-line impact. At this level of the system, keeping the CPU cache intact and full of process data is critical to good performance. Otherwise when you switch tasks you wind up having to go out to main memory, which is sloooow. So compact code is more important than fast code in a lot of these situations, and a neural net like you describe is going to be enormous relative to the approaches of Davide and Mingo.
And Linus rightly pointed out that as simple as the current scheduler is, it's basically the only part of the kernel that's pretty much the same code as it was 8 years ago, mostly because it does work very well for the vast majority of cases. Not to say that Mingo, Davide, Rob Love, et al aren't doing a huge boon, and on 4-way and bigger systems it's more of a concern, but e.g. the VM and I/O subsystems are areas that could reap far greater performance rewards for the most common cases (and the VM is getting a ton of eyes, and the I/O subsystem has been the focus of most of the early 2.5 work thanks to Jens Axboe and others)
Davide Libenzi's been shooting some holes in the architecture of this patch--he's also written a highly scalable multiqueue scheduler. He and Mingo are currently hammering out some ideas, almost certainly something will go into 2.5 but this may not be it.
The real reason threads can be faster than processes if all other things are equal is context switch time. Switching between threads that share all their page tables is just a matter of restoring all the registers. When switching between processes, you must flush the TLB also.
But when threads modify memory on SMP, they have to ensure cache coherency between CPUs. Processes can don't have this restriction, which is why processes can often be faster than threads. It all depends on your usage patterns and OS. On e.g. Solaris or Win32, threads are far faster at context switching than processes (mainly because processes are so heavyweight on those systems; Win32 also doesn't offer an API to efficiently implement multiprocess programs). On identical hardware, Linux's processes switches are faster than Solaris or Win32 thread switches and Linux thread switches aren't that much faster than process switches (beacause process switching is implemented in an efficient manner).
So on Linux, the fact that memory isn't shared between processes (hence CPU coherency isn't the same issue) often makes processes a substantial performance _win_.
This is aside from the fact that threads are extremely difficult to program properly--there's a reason that OSes spent all that time implementing protected memory (remember all the hoopla about that?) and threads throw that concept out the window, plus tend to lure the programmer into architectures that need nasty locking and synchronization.
That said, for some applications threads are the right answer. But using multiple processes, possibly with shared memory for a limited set of data, is generally a more maintainable solution that's easier to architect and implement. It certainly makes it explicit which data structures are meant to be shared, which a multithreaded solution does not.
I was under the impression that MITSHM supported shared memory images, but that shared memory pixmaps were something different
Yeah, I mentioned Xshm as well. Forgot to mention Xvideo, which is also worth looking at for video applications.
Finally, I must disagree about the UNIX socket. 5*10^-4 seconds seems short, but if that is the time it takes to draw several pixels (assuming the draw requests are buffered in Xlib), then you are in trouble. In that amount of time, a 500 MHz CPU goes through 250,000 cycles. Imagine how many pixels you could draw into a framebuffer in that amount of time
Not nearly that many. Going over the PCI or AGP bus is not going to run anywhere near that 500 MHz CPU, not to mention that drawing a pixel is more than one cycle _and_ you're going to be limited by memory bandwidth to get at your render data, _and_ most importantly if you update faster than the refresh rate of your display then you're just doing work that'll never be displayed. 85-90 Hz is about as fast as most displays are set.
Moreover, you're generally doing actual work to determine what pixel to display. If you're just blitting images as fast as possible to the screen, then either Xshm or or Xvideo or GLX extensions are probably what you want (XFree has them all).
If the compiler can build itself, then people can work on improving the compiler without needing to have a different compiler to build it with.
In this case, until now you needed to have Windows and a C# compiler for Windows in order to work on the C# compiler for Linux--that would shut someone like me (who has no Windows) out from being able to do compiler work. Not that I want to work on C# anyway, but you get the point.
Imagine if all the gcc developers had to buy a C compiler to work on developing the free one!
Double-buffering is tricky to implement, and can be slow. Unless your server supports shared-memory pixmaps, in order to do double buffering you must
Moot point, every serious X server (including XFree86, which most Slashdot users have) _does_ implement shared-memory pixmaps. Indeed, this has been true for better than 7 years now, when I started running XFree in 1994 both MIT-SHM and XShm were certainly supported.
Basically, X is great for usual GUI applications, but if you want to do something like, for example, an xmms plugin, you are incurring much overhead as opposed to direct access to some type of framebuffer
Not at all, there are plenty of ways to bang on the framebuffer without the round-trip to the server. One of the most prevalent is GLX, the OpenGL X extension which is supported on many card for XFree these days. It's been around on other platforms for over a decade.
Remember, SGI ruled the graphics performance world for a long time _using_ X11. X was designed to be very flexible through the use of extensions. My bet is that in 1-2 years GLX will be standard on pretty much all new Linux desktop installations, combined with DRM that can give darned good performance. Of course, good drivers are a prerequisite to good performance.
And yes, GLX works great for high-performance 2D graphics as well as 3d.
Couple of other points:
The protocol is unavoidable overhead: even if client and server are running on the same host, a call to, for example, XPutPixel() still has to pack a protocol request
Not true in the general case. Xlib is quite good at buffering many Xlib calls into far fewer X protocol requests.
and send it over a UNIX socket
Even on my old PPro 200 we're talking order 5*10^-4 seconds for a server round-trip over a Unix socket. That's pretty quick. And there's work being done on X servers that don't use UNIX sockets but rather SHM or other transports for protocol submissions. Jim Gettys had a very nice post on the subject that I'll try to dig up and link here.
Python fits the efficiency bill very will, and the psyco project is taking steps to address runtime for the speed freaks out there. Pretty interesting, although it's not ready for serious use quite yet.
http://sourceforge.net/projects/psyco/
Personally I prefer Python from a programmer efficiency standpoint, but ruby's good too.
I have to consider C++ a high level language because it's possible (and not too hard) to keep everything pretty well abstracted. I would say that of the high level languages C++ is the lowest, but that doesn't make much sense:)
The point was that in this context referring to C++ as a high-level language is pretty iffy. To an EE designing a chip, i386 assembly is a high-level language. To an OS implementor, asm is a low-level language and C is a high-level language. To your typical application developer, C is a low level language and Python and VB are high-level languages. In the context of Smalltalk, Python, Java, C#, and C++--which is what we were talking about--C++ is unquestionably a low-level language.
It's all relative to your point of reference.
parameterized types are basically a must-have.
I've seen some huge Java projects with very clean designs, so "must-have" seems awfully strong.
Way to snip the context. I said something along the lines of "in a strongly statically typed language, parameterized types are basically a must-have". Java purports to be such a language, but any sizeable program using collections foists off a fair amount of manual dynamic type-checking on the programmer. All this without the RAD and code-density advantages of a true dynamically typed language.
Seriously, this and the borked up I/O and threading capabilities are some of the worst parts of Java. That's not to say you can't write large projects in Java--_any_ language can be used to write large, successful projects with enough skill and effort. Java simply doesn't, in my opinion, approach either the dynamic (Smalltalk, Python) languages or the more mature static (OCaml, SML) languages as far as ease of implementation for large projects. It's libraries are it's strength and weakness; many of them are horribly designed, but they're bog-standard and they have an API (however poor) for basically everything. But Java still doesn't come anywhere near approaching especially Smalltalk in terms of installed base by application size and transaction counts, nor by total $$$ handled. And it doesn't approach even a goofy language like VB for worldwide LOCs or apps written. This despite a lot more marketing behind Java than Smalltalk ever had.
It's hardly the worst language ever written, in fact the language (as opposed to the libraries and the JVM platform) is a decent one if they'd just add parameterized types. But there are much better ones out there that have open licensing and source.
Personally, I'd recommend the following 4 language groups to any computer scientist wanting to learn several kinds of programming styles:
1. C, as a basic procedural language and because most other languages are implemented in it.
2. A strong statically typed functional language like OCaml, ML, or Haskell; OCaml wins based mostly on quality of implementation and availability, as well as offering an exposure to OO in a statically typed functional context.
3. A highly dynamic language like Lisp, Scheme, Smalltalk, or Python; Python and Smalltalk are my tossup choices here because they also offer an exposure to true OO programming (not simple class-oriented programming a la C++ and Java)
4. A weakly or non-typed language like typical Perl 4 or Tcl just to appreciate the alternatives. (Perl 4 wins because Perl knowledge is often handy)
Those I'd recommend well before C++ or Java (which offer comprimises on categories 1-3), although those 2 would probably be next (simply because of increasing prevalence) along with a programming by contract language like Eiffel and a logic-based language like Prolog.
Even if you wind up using VB on a daily basis, a good exposure to functional programming, static typing, etc will make you a better software engineer.
For real-world problems I almost always find myself falling on a mix of C, Python, Smalltalk, and high-level script (awk, Bourne, sed, etc). But I wouldn't give up my fp experience for the world.
Java and C# don't have real templates. That was intentional. It sounds like C++ is the only high level language you know.
If you're talking asm vs. C vs. C++, you can call C++ a high level language. If you're talking C++ vs. Java, C#, Smalltalk, and Python (as was the context here) then it's disingenuous to the point of laughability.
And the lack of parameterized types in Java is just ludicrous; if you have static type-checking, parameterized types are basically a must-have. Generic Java generates a lot of interest precisely because of this. Not to defend templates per se, but the concept is inseperable from a strong static typing system (see: Dylan, Haskell, ML)
If you can't tell the difference between Java and C#, you don't know Java or C#.
I think the implication wasn't that they aren't different in details, but rather that they aren't different in any interesting high-level PL sense. Not sure I agree with that, but I do agree that smalltalk or python or another dynamically typed language (not an untyped language) is often a far superior way to attack the problem spaces that.NET and Java seem to pursue. And if you really want static typing, the ML/OCaml approach is way ahead of the Java approach (and OCaml is blazingly fast and an open spec, unlike Java).
The CLI is way better designed than the JVM's bytecodes, you have to at least give it that.
(Insert obligatory comment about factoring primes being trivial here)
Symmetric crypto is already considered to be stronger than the available public-key methods. Well, stronger may be the wrong word: better studied and less likely to fall to a sudden advance in mathematics.
Most systems that use public key crypto (including e.g. PGP, ssh, ssl, all of which use RSA most commonly) use the public key method only to exchange a key and then use that key with a symmetric algorithm (most widely 3DES (SSH), IDEA (PGP), and RC4 (SSL), but plenty of others are common)
This approach limits the amount of crypto text (and known plaintext) available to attack the PK system with.
The problem is that most of the canonical methods of generating One Time Pads are based on discrete log (in the form of the Diffie-Helman (sp?) key exchange), which can be solved by factoring.
You're mixing things up. One-Time Pads are simply long strings of random numbers, you don't use Diffie-Hellman to generate them. And if you used DH to exchange them you defeat the point (which is that they're provably secure). They're generally delivered by trusted courier. OTPs are used (for instance) for some sensitive communications with US embassies and presumably by the CIA and NSA as well.
2) No one over the BAL in an emergency situation will ever need to drive
Even today they'd be ticketed. Aggrivated DWI is still DWI. And with the GPS "help me God" switch, you'd be able to call for help ASAP, too.
I think the concern is more like:
John and Tom go out camping. They take a few sixes, get drunk. Not planning on driving.
Tom falls and suffers a head wound/is mauled by a bear/otherwise needs immediate medical attention. They're out camping, it's 1/2 hour drive to the nearest town.
If John can drive his car, it may (depending on the nature of Tom's wounds) be less likely that he'll hurt someone on the way to the hospital than that Tom will suffer permanent injury or death because of the extra half hour wait when John activates the "help me God" feature and then sits waiting for the help to come take them to the hospital. In many rural areas, you don't even need to be camping to be 1/2 hour or more from the nearest hospital and hours from the nearest police department.
Right now, I'd expect that if you had someone bleeding in your car and were drunk while driving them to the hospital, the court would consider the mitigating circumstances. Certainly unlicensed drivers have been let off under these conditions. Even if it's still technically illegal, courts are usually highly amenable toward bending the law if a life or permanent injury is at stake.
If such a BAL meter were installed, hopefully there would be some sort of override switch that would flag your location to the police but allow you to start the car and drive it, then you could e.g. drive 15 minutes toward town, meet the police escort, and have them drive the rest of the way.
Slashdot Mirror
If a user gets an oops and submits a bug report to linux-kernel while running preempt, the bug report is a lot harder to decipher.
True, but the SMP support introduced these to begin with. The preemptive patches just bring that danger to UP machines.
Not true. preempt introduces new hangs. Read the threads on linux-kernel, especially "Re: [2.4.17/18pre] VM and swap - it's really unusable".
Some things that are broken by preempt:
* Network drivers which disable IRQs to avoid spinlocking on uniprocs (major performance win)
* Drivers which use per-CPU data to avoid spinlocking at all
* Drivers which disable individual interrupts for long periods of time
* Drivers which depend on consecutive lines of code executing near each other in time, especially serial drivers
That thread has details on all of these.
And there's the priority inversion scenario:
SCHED_OTHER process 1 acquires a semaphore in kernel mode
SCHED_FIFO realtime process needs the semaphore, blocks on it
Now the rt process is stuck pending progress from the SCHED_OTHER process. Without preemption, the SCHED_OTHER process would have done whatever required the semaphore and released it. Now, SCHED_OTHER process 2, 3,
[Note that I said semaphore, not spinlock, so the lock-break code won't help.]
Do you not remember all the problems with priority inversion and the SCHED_IDLE patches? This is exactly the same problem, it's not like it's something new and mysterious that people are making up as FUD to stop preempt from getting in to the kernel. Any introductory OS textbook discusses it, and priority inheritance is the only robust way to eliminate it--with all the problems that come along with priority inheritance.
4. It doesn't improve the worst-case latency.
>>>>>>>>>>>
It's not designed to. Love has already started work on a lock-breaking patch to get rid of long-held locks
It won't help. The fundamental problem is that preempt in interrupts is impossible with this scheme (and AFAIK nobody has proposed a scheme that could even theoretically work), and the worst-case latencies are in interrupts. The secondary problem is that a lot of these issues are highly hardware sensitive; different hardware has different timing requirements, and the only way to find them all is to audit every driver and deal with it appropriately (as, e.g., the LL patches do).
As Alan Cox wrote,
Sumner
Those that *need* LL for soft-realtime and those that *want* a LL-like environment so they have click a button more timely.
Well, I think that some sort of soft realtime is wanted pretty widely, if only to stop music and movies from skipping. LL does that fairly well, O(1) doesn't. But O(1) is definitely a win, modulo a few nits, and it's already been integrated into 2.5.x--with some work on those nits courtesy of Davide Libenzi.
So, I think O(1) is good but it's not a replacement for LL or another worst-case latency reducer, even among people who don't know what "soft realtime" means. They still don't want their mp3s to skip.
Sumner
The preemption patch is available now, and works now, and I am using it now.
The problems with the preempt patch are several:
1. It makes kernel debugging considerably harder
2. It introduces potential livelocks which could hang the system (this is an inherent part of the design and not an implementation bug)
3. It doesn't work with SMP (yet), this is fixable
4. It doesn't improve the worst-case latency.
#4 means it isn't a viable alternative to low-latency style patches; Love has said as much on linux-kernel, though obviously he feels it's a nice complement. 1 is a large hill to climb on its own, and 2 basically dooms it unless a clean solution can be found (most proposed solutions involve some sort of priority inheritance, which is a rat's nest).
In other words, the patch "works" only in a very limited sense; it doesn't decrease worst-case latency, can potentially hang the system, and doesn't interact well with SMP. It does decrease average-case latency.
The preemption patch is comparatively small and elegant.
Uh, that's in the eye of the beholder I guess. It is small, but it's not really elegant; it essentially causes conditional reschedules to happen all over the place, and by design cannot deal with latency in noninterruptable interrupt contexts--which happen to be exactly where the worst-case latencies are. So preempt as it's currently designed simply cannot address the worst-case latency. And it causes far more runtime conditional reschedules than the LL patches.
The low-latency patch decreases both worst and average-case latencies. It also doesn't have livelocks as an inherent part of its design. And average-case latencies with LL are as good as those with LL+preempt (in other words, preempt gains you nothing over LL). LL, unfortunately, touches a lot of code and has conditional schedules sprinkled all over the place--it's not elegant either, and certainly not small. But unlike preempt, it actually decreases the worst-case latency, which is what's important to the soft realtime applications that gave birth to both patches. There's a mini-LL patch as well which has far fewer conditional rescheds, that's being discussed for potential inclusion in 2.4.x and/or 2.5.x
My gut feeling is that preempt is the wrong approach to the latency problem, but the jury is still out on it. I don't think either the current preempt or the current LL are ready to go in the kernel. If you're doing e.g sound mixing or other latency-sensitive work right now, the full LL patches seem like the best alternative available. They get latencies down extremely low, well below those of e.g. BeOS.
Sumner
Linux Magazine's Gearheads Only is a great column to read for this, especially the mouse driver and Alan Cox's articles.
Their web site should have archives.
Sumner
Sure. But that goes both ways. _If_ you know asm and C well, going from kernel code to GUI code has a lot of pitfalls (as does the opposite). Or going to numerical recipe code, or whatever.
But fundamentally, code is code and rather than be intimidated you should start reading and playing with it. Even if you never become a hardcore kernel hacker, it's another world that's definitely worth jumping into and not nearly as scary as you think.
Alan's Gearheads Only articles in Linux Magazine are a great way to start out.
(FWIW, I posted mainly as a gedanken piece and not because I think you're way off base; sometimes you see a quote that just seems applicable to another conversation...)
Sumner
From Alan Cox, in the kerneltrap interview running on the front page of slashdot right now:
Ignore everyone who tells you kernel hacking is hard, special or different. It's a large program, and bug fixing or driver tweaking can be a best starting point. It is however not magic, nor written in a secret language that only deep initiates with beards can read.
Play with it, try things, break it horribly and enjoy yourself. I started on the networking code because it didn't work very well. Everything I knew about TCP/IP I had downloaded the same day I started hacking the net code. My first attempts were not pretty but it was *fun*.
Sumner
I still don't see any reason that processes are inherently more efficient than threads.
Right, that's why I said that for some things threads are better. But a lot of people pick threads over processes for reasons other than "I need to share all my memory". Which is bad. Choosing to use threads should be a conscious decision, made only when you've determined that e.g. processes with a small shared memory segment, message passing, or other approaches are worse.
Sumner
Well, with paper junkmai the sender has to pay for it (rather than having it subsidized by the recipient). It keeps your postal rates low, and it encourages the sender to target the mailings to people who might be interested (e.g. open-source mailings to people having licensing trouble...) because they have to pay for every piece of mail sent.
I've not had to confront an employer about this explicitly, but having flexible hours is on my list of things I require when doing a job search. For the right incentives it might be negotiable, but in my area traffic is so bad that I need to leave the house at 8:00 to get to work by 9:00 but I can leave at 9:00 and be in by 9:20 easily. 40 minutes a day in traffic adds up, and for people who live further away it's even more important.
The two other big items on my "must have" list before taking a job are 1) no cell phone/pager--when I leave work, I'm on _my_ time until I next come int--and 2) reasonable IP terms in the employee agreement so that software I develop on my own time without using company resources is mine and mine alone.
Sumner
Nope. It's $2.56 a check.
He stopped doubling it years ago.
Just to make it clear:
"The advantage of the virtual server via NAT is that real servers can run any operating system that supports TCP/IP protocol, real servers can use private Internet addresses, and only an IP address is needed for the load balancer"
So you can keep your application on Solaris with LVS.
Sumner
First off, the Kegel's c10k page referenced earlier is definitely worth a read. And if you're under the impression that having 2^16 TCP port numbers limits you to 2^16 connections, that's not accurate. You can have hundreds of thousands of connections to one machine, presuming you manage them properly (as the c10k page points out).
More importantly, you should check out http://www.linuxvirtualserver.org/ ; it's aim is exactly what you want:
"The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the cluster is transparent to end users. End users only see a single virtual server."
Sounds like a perfect match.
Sumner
This is a terrible idea. The fact is, for 99% of applications the scheduler isn't a bottleneck. Even on Alan Cox's worst-case 8-way SMP machines the scheduler doesn't eat tremendous amounts of CPU; _way_ more than it should, but an approach like Ingo Molnar's or Davide Libenzi's will get rid of most of that overhead.
But the real reason it's a terrible idea is because of the cache-line impact. At this level of the system, keeping the CPU cache intact and full of process data is critical to good performance. Otherwise when you switch tasks you wind up having to go out to main memory, which is sloooow. So compact code is more important than fast code in a lot of these situations, and a neural net like you describe is going to be enormous relative to the approaches of Davide and Mingo.
And Linus rightly pointed out that as simple as the current scheduler is, it's basically the only part of the kernel that's pretty much the same code as it was 8 years ago, mostly because it does work very well for the vast majority of cases. Not to say that Mingo, Davide, Rob Love, et al aren't doing a huge boon, and on 4-way and bigger systems it's more of a concern, but e.g. the VM and I/O subsystems are areas that could reap far greater performance rewards for the most common cases (and the VM is getting a ton of eyes, and the I/O subsystem has been the focus of most of the early 2.5 work thanks to Jens Axboe and others)
Sumner Hayes
Davide Libenzi's been shooting some holes in the architecture of this patch--he's also written a highly scalable multiqueue scheduler. He and Mingo are currently hammering out some ideas, almost certainly something will go into 2.5 but this may not be it.
The real reason threads can be faster than processes if all other things are equal is context switch time. Switching between threads that share all their page tables is just a matter of restoring all the registers. When switching between processes, you must flush the TLB also.
But when threads modify memory on SMP, they have to ensure cache coherency between CPUs. Processes can don't have this restriction, which is why processes can often be faster than threads. It all depends on your usage patterns and OS. On e.g. Solaris or Win32, threads are far faster at context switching than processes (mainly because processes are so heavyweight on those systems; Win32 also doesn't offer an API to efficiently implement multiprocess programs). On identical hardware, Linux's processes switches are faster than Solaris or Win32 thread switches and Linux thread switches aren't that much faster than process switches (beacause process switching is implemented in an efficient manner).
So on Linux, the fact that memory isn't shared between processes (hence CPU coherency isn't the same issue) often makes processes a substantial performance _win_.
This is aside from the fact that threads are extremely difficult to program properly--there's a reason that OSes spent all that time implementing protected memory (remember all the hoopla about that?) and threads throw that concept out the window, plus tend to lure the programmer into architectures that need nasty locking and synchronization.
That said, for some applications threads are the right answer. But using multiple processes, possibly with shared memory for a limited set of data, is generally a more maintainable solution that's easier to architect and implement. It certainly makes it explicit which data structures are meant to be shared, which a multithreaded solution does not.
Sumner
I was under the impression that MITSHM supported shared memory images, but that shared memory pixmaps were something different
Yeah, I mentioned Xshm as well. Forgot to mention Xvideo, which is also worth looking at for video applications.
Finally, I must disagree about the UNIX socket. 5*10^-4 seconds seems short, but if that is the time it takes to draw several pixels (assuming the draw requests are buffered in Xlib), then you are in trouble. In that amount of time, a 500 MHz CPU goes through 250,000 cycles. Imagine how many pixels you could draw into a framebuffer in that amount of time
Not nearly that many. Going over the PCI or AGP bus is not going to run anywhere near that 500 MHz CPU, not to mention that drawing a pixel is more than one cycle _and_ you're going to be limited by memory bandwidth to get at your render data, _and_ most importantly if you update faster than the refresh rate of your display then you're just doing work that'll never be displayed. 85-90 Hz is about as fast as most displays are set.
Moreover, you're generally doing actual work to determine what pixel to display. If you're just blitting images as fast as possible to the screen, then either Xshm or or Xvideo or GLX extensions are probably what you want (XFree has them all).
Sumner
If the compiler can build itself, then people can work on improving the compiler without needing to have a different compiler to build it with.
In this case, until now you needed to have Windows and a C# compiler for Windows in order to work on the C# compiler for Linux--that would shut someone like me (who has no Windows) out from being able to do compiler work. Not that I want to work on C# anyway, but you get the point.
Imagine if all the gcc developers had to buy a C compiler to work on developing the free one!
Sumner
Double-buffering is tricky to implement, and can be slow. Unless your server supports shared-memory pixmaps, in order to do double buffering you must
Moot point, every serious X server (including XFree86, which most Slashdot users have) _does_ implement shared-memory pixmaps. Indeed, this has been true for better than 7 years now, when I started running XFree in 1994 both MIT-SHM and XShm were certainly supported.
Basically, X is great for usual GUI applications, but if you want to do something like, for example, an xmms plugin, you are incurring much overhead as opposed to direct access to some type of framebuffer
Not at all, there are plenty of ways to bang on the framebuffer without the round-trip to the server. One of the most prevalent is GLX, the OpenGL X extension which is supported on many card for XFree these days. It's been around on other platforms for over a decade.
Remember, SGI ruled the graphics performance world for a long time _using_ X11. X was designed to be very flexible through the use of extensions. My bet is that in 1-2 years GLX will be standard on pretty much all new Linux desktop installations, combined with DRM that can give darned good performance. Of course, good drivers are a prerequisite to good performance.
And yes, GLX works great for high-performance 2D graphics as well as 3d.
Couple of other points:
The protocol is unavoidable overhead: even if client and server are running on the same host, a call to, for example, XPutPixel() still has to pack a protocol request
Not true in the general case. Xlib is quite good at buffering many Xlib calls into far fewer X protocol requests.
and send it over a UNIX socket
Even on my old PPro 200 we're talking order 5*10^-4 seconds for a server round-trip over a Unix socket. That's pretty quick. And there's work being done on X servers that don't use UNIX sockets but rather SHM or other transports for protocol submissions. Jim Gettys had a very nice post on the subject that I'll try to dig up and link here.
Sumner
Python fits the efficiency bill very will, and the psyco project is taking steps to address runtime for the speed freaks out there. Pretty interesting, although it's not ready for serious use quite yet.
http://sourceforge.net/projects/psyco/
Personally I prefer Python from a programmer efficiency standpoint, but ruby's good too.
Sumner
I have to consider C++ a high level language because it's possible (and not too hard) to keep everything pretty well abstracted. I would say that of the high level languages C++ is the lowest, but that doesn't make much sense :)
The point was that in this context referring to C++ as a high-level language is pretty iffy. To an EE designing a chip, i386 assembly is a high-level language. To an OS implementor, asm is a low-level language and C is a high-level language. To your typical application developer, C is a low level language and Python and VB are high-level languages. In the context of Smalltalk, Python, Java, C#, and C++--which is what we were talking about--C++ is unquestionably a low-level language.
It's all relative to your point of reference.
parameterized types are basically a must-have.
I've seen some huge Java projects with very clean designs, so "must-have" seems awfully strong.
Way to snip the context. I said something along the lines of "in a strongly statically typed language, parameterized types are basically a must-have". Java purports to be such a language, but any sizeable program using collections foists off a fair amount of manual dynamic type-checking on the programmer. All this without the RAD and code-density advantages of a true dynamically typed language.
Seriously, this and the borked up I/O and threading capabilities are some of the worst parts of Java. That's not to say you can't write large projects in Java--_any_ language can be used to write large, successful projects with enough skill and effort. Java simply doesn't, in my opinion, approach either the dynamic (Smalltalk, Python) languages or the more mature static (OCaml, SML) languages as far as ease of implementation for large projects. It's libraries are it's strength and weakness; many of them are horribly designed, but they're bog-standard and they have an API (however poor) for basically everything. But Java still doesn't come anywhere near approaching especially Smalltalk in terms of installed base by application size and transaction counts, nor by total $$$ handled. And it doesn't approach even a goofy language like VB for worldwide LOCs or apps written. This despite a lot more marketing behind Java than Smalltalk ever had.
It's hardly the worst language ever written, in fact the language (as opposed to the libraries and the JVM platform) is a decent one if they'd just add parameterized types. But there are much better ones out there that have open licensing and source.
Personally, I'd recommend the following 4 language groups to any computer scientist wanting to learn several kinds of programming styles:
1. C, as a basic procedural language and because most other languages are implemented in it.
2. A strong statically typed functional language like OCaml, ML, or Haskell; OCaml wins based mostly on quality of implementation and availability, as well as offering an exposure to OO in a statically typed functional context.
3. A highly dynamic language like Lisp, Scheme, Smalltalk, or Python; Python and Smalltalk are my tossup choices here because they also offer an exposure to true OO programming (not simple class-oriented programming a la C++ and Java)
4. A weakly or non-typed language like typical Perl 4 or Tcl just to appreciate the alternatives. (Perl 4 wins because Perl knowledge is often handy)
Those I'd recommend well before C++ or Java (which offer comprimises on categories 1-3), although those 2 would probably be next (simply because of increasing prevalence) along with a programming by contract language like Eiffel and a logic-based language like Prolog.
Even if you wind up using VB on a daily basis, a good exposure to functional programming, static typing, etc will make you a better software engineer.
For real-world problems I almost always find myself falling on a mix of C, Python, Smalltalk, and high-level script (awk, Bourne, sed, etc). But I wouldn't give up my fp experience for the world.
Sumner
Java and C# don't have real templates. That was intentional. It sounds like C++ is the only high level language you know.
.NET and Java seem to pursue. And if you really want static typing, the ML/OCaml approach is way ahead of the Java approach (and OCaml is blazingly fast and an open spec, unlike Java).
If you're talking asm vs. C vs. C++, you can call C++ a high level language. If you're talking C++ vs. Java, C#, Smalltalk, and Python (as was the context here) then it's disingenuous to the point of laughability.
And the lack of parameterized types in Java is just ludicrous; if you have static type-checking, parameterized types are basically a must-have. Generic Java generates a lot of interest precisely because of this. Not to defend templates per se, but the concept is inseperable from a strong static typing system (see: Dylan, Haskell, ML)
If you can't tell the difference between Java and C#, you don't know Java or C#.
I think the implication wasn't that they aren't different in details, but rather that they aren't different in any interesting high-level PL sense. Not sure I agree with that, but I do agree that smalltalk or python or another dynamically typed language (not an untyped language) is often a far superior way to attack the problem spaces that
The CLI is way better designed than the JVM's bytecodes, you have to at least give it that.
Sumner
(Insert obligatory comment about factoring primes being trivial here)
Symmetric crypto is already considered to be stronger than the available public-key methods. Well, stronger may be the wrong word: better studied and less likely to fall to a sudden advance in mathematics.
Most systems that use public key crypto (including e.g. PGP, ssh, ssl, all of which use RSA most commonly) use the public key method only to exchange a key and then use that key with a symmetric algorithm (most widely 3DES (SSH), IDEA (PGP), and RC4 (SSL), but plenty of others are common)
This approach limits the amount of crypto text (and known plaintext) available to attack the PK system with.
Sumner
The problem is that most of the canonical methods of generating One Time Pads are based on discrete log (in the form of the Diffie-Helman (sp?) key exchange), which can be solved by factoring.
You're mixing things up. One-Time Pads are simply long strings of random numbers, you don't use Diffie-Hellman to generate them. And if you used DH to exchange them you defeat the point (which is that they're provably secure). They're generally delivered by trusted courier. OTPs are used (for instance) for some sensitive communications with US embassies and presumably by the CIA and NSA as well.
DH is completely unrelated to one time pads.
Sumner
He didn't say "the same file standard". They could add new features as long as they document the file format for them openly.
2) No one over the BAL in an emergency situation will ever need to drive
Even today they'd be ticketed. Aggrivated DWI is still DWI. And with the GPS "help me God" switch, you'd be able to call for help ASAP, too.
I think the concern is more like:
John and Tom go out camping. They take a few sixes, get drunk. Not planning on driving.
Tom falls and suffers a head wound/is mauled by a bear/otherwise needs immediate medical attention. They're out camping, it's 1/2 hour drive to the nearest town.
If John can drive his car, it may (depending on the nature of Tom's wounds) be less likely that he'll hurt someone on the way to the hospital than that Tom will suffer permanent injury or death because of the extra half hour wait when John activates the "help me God" feature and then sits waiting for the help to come take them to the hospital. In many rural areas, you don't even need to be camping to be 1/2 hour or more from the nearest hospital and hours from the nearest police department.
Right now, I'd expect that if you had someone bleeding in your car and were drunk while driving them to the hospital, the court would consider the mitigating circumstances. Certainly unlicensed drivers have been let off under these conditions. Even if it's still technically illegal, courts are usually highly amenable toward bending the law if a life or permanent injury is at stake.
If such a BAL meter were installed, hopefully there would be some sort of override switch that would flag your location to the police but allow you to start the car and drive it, then you could e.g. drive 15 minutes toward town, meet the police escort, and have them drive the rest of the way.