Is there a response? What incremental, random changes produced an eye such that each step conferred an evolutionary advantage?
It's well understood; the progression is roughly: light sensitive cell, opaque pigment in back, retreat into concavity, formation of pinhole camera, transparent covering, fixed lens, adaptable lens. Each of those has distinct and individual evolutionary advantages, sometimes related to improved predator evasion and sometimes merely related to improved protection of the existing structure. It seems to have happened several times in evolution, so it's not even anything unusual; if we ever encounter aliens, they probably have eyes, too.
The problem for teachers is that ID can't be tested using the scientific method, the system of making, testing and retesting hypotheses that is the bedrock of science.
That's false. ID can be tested (in the same way astronomy can be), and the answer is: there is not a shred of evidence to support ID. Every single test of evolution has come down on the side of evolution (mutation and selection) and against intelligent design (interference of an intelligent agent in the development of different life forms on earth). ID has the form of a scientific theory, but it happens to be an incorrect scientific theory according to overwhelming evidence.
The Chinese government does not want to go down in history as a government that sends people to their death in interplanetary travel; it's bad PR, it's bad for business, and it doesn't even fit in with the philosophy underlying Chinese governmental authority.
In any case, what would the motivation be? The only reason for manned travel to Mars in the near future is as a publicity stunt and to make people believe that the universe works like it does in Star Trek. That illusion is going to be destroyed if you plan on leaving people there to die.
Scientifically, rovers are a far better investment right now and for the coming decades. They can stay there for months with almost no resource requirements, and they really don't have to come back.
My point was that binary modules linked with the kernel or interpreted in the kernel VM would share the same heap as the kernel itself.
There is no such thing as "the heap" in a modern garbage collected system. Modern garbage collectors have variable numbers of regions and areas that are utilized for different objects. These regions are utilized adaptively by the garbage collector and optionally under supervisor or program control. You can limit their sizes, their growth, their allocation rates, their placement in memory, or whatever else you can imagine.
You have to start inventing ad-hoc quota mechanisms to start trying to limit the damage, but I've seen no accounting system that is flexible enough for all cases.
These techniques were invented in the 1970's and 1980's. It really sounds to me like EROS is reinventing a limited form of modern memory management techniques, and not doing a good job at it.
Regardless of how much better GC is, you simply cannot do better than a fully-pluggable memory management system.
Well, I think one can certainly do a lot better than a fully-pluggable memory management system based on a microkernel architecture, both in terms of functionality and in terms of performance.
No, [UNIX users on multiuser machines] are not malicious users, though they may be untrusted users. If they were actually malicious, that system would be taken down within minutes of it being brought up; certainly not the mark of a secure or reliable system architecture.
A UNIX machine with hundreds of CS undergraduates has malicious users on it: all the DoS and security hacks you can imagine have been tried on it. If it survives that (and it does), it's good enough for most applications. It's not secure because of any principles, it's because in the decade after BSD and SunOS got released, people spent huge amounts of time fixing bugs and problems.
suppose you're referring to uninitialized variables. Indeed, C imposes no restrictions on initialization, just like assembly. You are free to be as safe, or unsafe as you like within the limits provided by the language implementation.
No, I'm referring to the fact that the effect of dereferencing a pointer to an int as a char pointer is undefined, even if the int is initialized properly. All those things I mentioned are effects you might plausibly see even on an ANSI conformant C compiler for Pentium.
For the most part, that's very likely. But it's not here today, and that's all that matters when choosing a language here and now. Just cross your fingers that they get BitC right.
I don't have much hope for BitC; it may be technically alright, but if it doesn't look vaguely like something people know, it won't get used. I think the most likely next language for a widely used kernel will eventually be a Java-to-C translator, a C#-to-C translator, a safer variant of Objective C developed by Apple, or possibly ANSI C 2015 or something, where people finally clean up the mess.
This is a checklist item for Microsoft, to address problems like they are having in MA. It lets Microsoft zealots and people resistant to change in companies and the government buy MS Office even when there are regulations requiring support for open document formats. Microsoft has done the same with many other standards. For example, their POSIX support gets around POSIX requirements, but it is pretty much useless in practice. The fact that they are outsourcing this tells you how low priority it is for them--Microsoft almost never outsources anything that is of any importance to their business.
In my question I'm referring to malicious code which will not play by the rules of "good storage management".
Whatever constraints you impose on storage management without GC you can impose with GC. In fact, GCs are arguably inherently better at enforcing real-time or storage management constraint than manual allocators (Sun's tribulations with Java notwithstanding--a poor GC can indeed be a headache, but so can a poor malloc).
UNIX is inherently based on a collaborative system design; it was not designed to support secure collaboration however.
UNIX has been, and continues to be, widely used for large multiuser systems with sometimes thousands of untrusted and malicious users. It really works for that application.
Microkernels in safe verifiable languages like BitC+Coyotos will solve a great deal more of these problems (more than your design I'd wager). The solution is not to drag all that cruft into a single monolithic kernel. But that's just my opinion; well, mine and a great deal of others who share it.
Well, it's good that we do agree as much that writing a kernel in a new language is a good idea. The methodology and design of EROS/Coyotos is a separate discussion.
Because the "first byte of an int" depends on the endianness of the underlying machine.
Or you may get a completely unrelated value, an incorrect value every 4 billion accesses, or a crash.
This is a problem of ignorance, not the standard or the language itself
It's also a problem of software engineering because it's impossible looking at a piece of C code to determine if and where it uses unsafe or unportable constructs. That causes major problems when porting systems code (but I gather CapROS is Intel/gcc only at this point).
This sort of difficulty is completely unnecessary--it buys you nothing in terms of performance or expressiveness. It's a historical accident. It would be trivial to have a language that is in every respect like C but that cleanly separates its portable and defined constructs from implementation or undefined constructs.
Look up NeXTStep, Zmacs, Smalltalk-80, and Tcl/Tk, for examples of GUI designers, Intellisense, integrated development/debugging, and dynamic layout. All of them predate the introduction of even Visual C++ 1.0.
Of course, your logic also holds Bowing and Lockheed guilty of copying the Wright Brothers. After all, all of their designs are really just modifications of those planes that came before them.
The analogy doesn't hold. Boeing and Lockheed actually improved on their predecessors, Microsoft is still trying to catch up.
Show me a Lisp/Smalltalk/high-level language implementation right now, that permits interfacing with low-level hardware (either natively or through FFI), that permits no runtime environment (no GC, etc.), and that has a free, downloadable compiler that compiles to native code.
People have written operating systems in Modula-3 and Ada, so those would be obvious choices and they have free implementations. Of course, not many people know them, which is kind of a problem.
In practice, I'd probably go with one or another Java implementation. If you like, you can use Java (but not the standard Java libraries) without a GC, but, as I mentioned, kernels should start using GC.
I'd probably first look into using one of the FOSS VMs (IBM's is very good) and build around that. Failing that, a Java-to-C compiler or gcj would be other possible choices.
As a last resort, I might consider using a subset of C++ with an exact garbage collector (they can be written as libraries) and enforcing the use of relatively safe constructs through a (custom written) style checker.
You cannot disagree with my statement because it is a simple observable fact: C provides direct access to datatypes which map almost directly to base valuetypes on commodity hardware. Commodity hardware: 32-bit and 64-bit desktop and server byte-addressable CPU architectures. C datatypes: char, int, long, long long (and unsigned variants)
The C language makes few guarantees about how machine datatypes map onto C datatypes. The mapping may even change according to compiler versions and compiler flags, and the mappings are often not the "obvious" ones.
Yes, if you use gcc extensions like __inline__, it's no longer technically "C". That does not mean you cannot write a kernel in pure C even on gcc. All host-specific code is encapsulated in separate asm files and linked to the portable C.
Even something as simple as accessing the first byte of an int through a char pointer is undefined. And your remarks highlight another problem with C: its users don't even realize when they are using undefined constructs.
It can, if we impose a type system on top of it. See LLVM for example. You might be interested in LLVA in fact. ("LLVA: A Low-level Virtual Instruction Set Architecture")
OK, so to summarize, you agree now that safe, garbage collected, object-oriented languages can give performance competitive with C, then. (It doesn't take a VM or a compiler/runtime more complicated than C, but that's a separate issue.)
[using the MMU to decompose a large software system (kernel or otherwise) into modules is often bad.] This is pure speculation. You have no real evidence to support this conjecture.
There is extensive evidence. If you want to check it for yourself, just try using DCOM or RMI to put some objects of an application into a separate address space. Not only is it a significant amount of work, it's also slow.
I'm still not clear whether you actually have processes, or they're simulated on the kernel VM,
As I was saying: "Such a kernel would be a drop-in replacement for Linux with no user visible changes". It would run Linux binaries, do scheduling and accounting in the same way, etc. The architecture, drivers, etc. all would have nearly identical code to what they have now. The main difference would be that unsafe uses of pointers would be eliminated by a better type system and that low-level access would happen through explicitly marked constructs.
Let's say you find a way to handle the above memory accounting issue, what if the malicious object now accumulated garbage and freed it immediately, causing a great deal of work for the garbage collector?
Despite the name, modern garbage collectors don't work that way; they are more of "live object collectors and categorizers". Furthermore, the cost of collection is amortized over the allocation: if a caller allocates a lot of objects, he has already paid his share of CPU time to free them, just like with malloc/free. Finally, the use of garbage collection doesn't give you license to abandon all principles of good storage management. In an environment like a kernel, you would likely use arenas, allowing you to recycle all resources associated with a user or a process instantly.
Notice, the above accounting issues are all existing problems with just about every popular kernel in existence, particularly UNIX clones.
Well, as a UNIX user, admin, and developer for a quarter of a century now, I can say that they are not practical problems in any real-world usage I have encountered. The practical problems with UNIX are lack of robustness against errors in dynamically loadable modules, difficulties associated with developing new kernel modules, and problems with module versioning.
C provides direct access to datatypes which map almost directly to base valuetypes on commodity hardware. This is a statement of the current state of affairs.
No, it is not. C is a language and it has no facilities for low-level access. There are lots of C compilers that provide such access as implementation defined extensions (all in slightly different ways), but it's not part of C. EROS is a kernel written in one or two C compiler-defined dialects, not a kernel written in C. The distinction matters because you can do exactly the same thing in many other languages, and usually in better ways.
It's good to see that Windows programmers are thinking about this sort of thing. However, their sense of history is a little lacking.
IntelliSense is a technology that is inevitable. If Microsoft hadn't done it, somebody else would have.
In fact, someone else did it: Emacs, Zmacs, and Hemlock had this feature already about 20 years ago. Unlike IntelliSense, the information was displayed unobtrusively in the status line, and you got completion only if you asked for it with tabs. Unlike IntelliSense, the information was also based on the actually loaded code (at least when writing Lisp code), not some separate database that may be out of date and may contain irrelevant data. But maybe Microsoft will "invent" that in another 20 years, too.
And here is the problem with this design: you seem to be ignoring the fact that it's running on a real machine without these high-level features
That is an imaginary problem, since standard hardware can support runtime safety, garbage collection, and object-oriented dispatch efficiently.
In some sense a microkernel is a workaround for safety issues: by decomposing functionality, you are isolating faults.
Decomposition is good; using the MMU to decompose a large software system (kernel or otherwise) into modules is often bad.
So you're idea is something like J-Kernel from the paper I linked to earlier, but actually running in kernel space and executing the whole system written in Java?
No, it would most definitely not be like J-Kernel, which I consider impractical at this point. "My idea" is a system very much like the Linux or BSD kernel but written in a compiled language with garbage collection, runtime safety, well-defined primitives for accessing memory and hardware, and a simple object system. Such a kernel would be a drop-in replacement for Linux with no user visible changes (it could be system call compatible and user mode code could be written in any language whatsoever), and few changes in the way the kernel itself is developed. The main benefits would be easier testing, easier extensibility, fewer security problems, and better support for dynamically loadable modules.
In real life, I think one might have to go there incrementally, by adding support for another systems programming language to the Linux and/or BSD kernel. A JVM/JIT wouldn't be my first choice, but it seems like the most realistic, and would be useful for intelligent packet filtering, security, file systems, network protocols, and some drivers. There has been some work on that (TeaseMe), but it will only take off when a good JIT is integrated.
C provides direct access to datatypes which map almost directly to base valuetypes on commodity hardware. Good so far?
No, sorry, that is not true. There are languages that make such guarantees, but C isn't one of them. People using C for such purposes are relying on non-standard (if common) features; the problem is that C never alerts you to this fact, and that these features can behave differently even between different versions of the compiler or with different flags.
C itself makes no real runtime assumptions beyond the presence of a stack.
The C language makes no guarantees whatsoever about real time behavior or memory management behavior. There may or may not be a stack. There may or may not be dynamic memory allocation even if you don't ever call "malloc". There may or may not be a garbage collector.
That's just it, there effectively is no "C runtime" though
C has as much or as little of a runtime as the compiler implementor happend to choose. GNU C happens to use a fairly simple runtime, as did K&R C, but other C compilers don't. Some have garbage collectors, for example. Try running your code under tcc in "safe" mode.
Conversely, just like you can pick an implementation of C and use a stripped down runtime with it, you can do the same in Lisp, Modula-3, or Java, and people have. You can write code in those languages that never invokes the garbage collector.
I'm sorry, but you still don't get my point. C is a programming language defined by a language standard. You keep claiming that C permits efficient access to low-level features, but that claim is wrong: the C programming language provides no access to low level features at all. Any such access is provided via implementation-dependent features. And just like it's easy to add those features to C, it's easy to add them to other languages.
Similarly, the C runtime is unsuitable for use in a kernel; any kernel use of C needs its own runtime. Well, you do the same thing when writing a kernel in any other language.
I think I have stated this pretty clearly already: I'm looking for a kernel very much like the Linux or BSD kernel, but replacing C with a simple, safe object-oriented language that supports garbage collection. It would be nice if the runtime contained a JIT compiler, but that would not be necessary. Either way, modules would be dynamically loadable, communication between kernel components would happen through the language's object oriented mechanisms, and the language and runtime would guarantee fault isolation and resource reclamation (except in contexts where the programmer explicitly overrides it).
That's in contrast to microkernels that take the functionality currently contained in kernels like Linux or BSD and split it up among separate processes; the microkernel approach introduces unnecessary overhead and complications into the kernel design. I view microkernels as a workaround for the lack of runtime safety in languages like C.
It's also in contrast to the Lisp machine or some attempts at a Java OS, in which everything, including applications, runs inside a single big runtime and address space; resource management and other issues haven't been solved satisfactorily in that context and current applications aren't written for that kind of environment. However, once you have moved to a safe language and runtime, you can start exploring the issue of moving more functionality into the kernel's address space.
ANSI C strictly forbids field reordering. In-memory layout is precisely as defined in the struct to the nearest byte (bit fields are known to be non-portable and thus unusable).
ANSI C forbids reordering, but it doesn't define the size of a byte, nor does it precisely define alignment or padding.
Portable malloc/free API. Not sure what you're point is here.
The point is that the kernel doesn't use the language-supplied malloc/free; it uses its own unportable mechanisms.
"memory mapped I/O" Not part of the C standard.
My point exactly.
I merely said that C is the currently probably the best candidate, particularly for a microkernel implementation.
Well, and as we can see, you don't even know what features are part of the C language. C implementations can be garbage collected and abort on every one of those "low-level features" you so dearly love as a runtime error. The C language is not so much characterized by the presence of low level features, but by the absence of guarantees that high level features are present.
But even if you are trying to argue that using GNU C or a similar compiler on Intel is "the best candidate", that is also methodologically wrong: you claim that systems like EROS and CapROS are aimed at performance and security. But if you base those systems based on a language with no language mechanisms for fault isolation and a propensity for serious security problems, then EROS and CapROS can be argued to be merely workarounds for the limitations of their implementation language.
"The other big problem with C is that many of its low-level features unnecessarily rely on implementation-specific features that aren't part of the standard." Very few such features exist, they are well known, and they aren't used in operating system kernels. You are really grasping at straws here.
No, you simply do not understand C at all. Structure layout, memory access through cast pointers, memory allocation and management, and memory mapped I/O are all highly implementation dependent features.
Hence the reason for developing BitC and Coyotos.
Which only goes to show that all your objections to how one can't use anything other than C are groundless.
We could now start debating whether the methodology underlying the development of EROS and Coyotos ("principle based") is acceptable and whether the authors are actually properly testing their claims or just handwaving. We could also discuss whether even the principles that the authors of those systems pursue hold water or are misguided. But those are separate discussions.
1. low-level control over representation/memory layout (hardware interfacing demands this)
C does not have sufficient low-level control over representation/memory layout for hardware interfacing; any such "control" you get is highly implementation dependent, and the same kernel code compiled with different C compilers will often not work. Furthermore, there are no defined semantics for the code the compiler generates for accessing data structures, which can cause problems with memory mapped I/O.
2. control over runtime environment (you can't enforce a runtime environment on some kernel designs)
You have no more and no less control over the runtime environment in C than you do in any other language. A conforming C compiler and runtime is perfectly free to use garbage collection and to terminate your program the first time you use "float x; return *(char*)".
When C is used for implementing kernels, people generally take the C compiler and implement their own C runtime for the kernel. That's exactly what you do when you use any other language, including languages with garbage collection, for implementing kernels.
3. readily available compiler (preferably free as in beer, at the very least, since academic and free software work is, well, free)
Building a small compiler to support kernel development is negligible amounts of work compared to implementing the kernel, and there are plenty of compilers you could build on (Eiffel, Portable Object Compiler, tcc, Oberon, etc.).
The only candidates that fit that bill are, you guessed it: C, C++, Ada, Pascal
C and C++ do not fit your requirements at all; you only think they do because you confuse implementation-specific features that are not part of the language with the language itself. At best, you can argue that the specific language implemented by gcc with an Intel backend is good for your purposes, but the C programming language is, by itself, completely unsuitable for implementing kernels.
I'm not sure precisely how that translates into a conclusion that microkernels, and that CapROS in particular, aren't worthwhile.
Here, I have criticized EROS (and by extension CapROS) for its microkernel design--relying on memory management hardware rather than language and runtime mechanisms in order to achieve correctness. Apparently, the EROS authors themselves realized this, which is why Coyotos uses a new programming language.
I haven't said anything beyond that about EROS; that would be an entirely separate discussion. The fundamental problem I see with the EROS approach is that it is principle driven, rather than data driven. These people keep talking about "trust" and "security" without ever demonstrating measurable improvements in those areas under controlled conditions.
It really is a shame that people get so stuck in their mindsets and are so convinced they're right that they fail to acknowledge evidence to the contrary, isn't it?
I know that work. It's an article talking about retrofitting a process model into a JVM. I don't think that's the right approach right now for many reasons.
I think the right choice for an OS kernel at this point is something very similar to the Linux kernel, with separate user-mode processes. It's just that the OS kernel itself should be implemented in something a little higher level and a little safer than C, with the ability to drop down into unsafe operations when necessary and with some simple object oriented facilities. Oberon seems like roughly the right level, or maybe a Java implementation like gcj.
But tell you what: why don't you do some research about languages and OS implementations. Go look at languages like PL/1, Oberon, Algol, Modula-2, Modula-3, Ada, and Cedar/Mesa. Go look at operating systems like Alto, Multics, Oberon, Apple Lisa, and B5000.
Also, go look up the discussions on aliasing and pointers in C and all the problems that caused with optimization over the evolution of C since K&R. C is not, and has never been, a high performance language and often does not come even close to carefully written assembly language or well-optimized code for other languages. The other big problem with C is that many of its low-level features unnecessarily rely on implementation-specific features that aren't part of the standard. C was a charming language 20 years ago because it fit onto a PDP-11 and on a CP/M machine where nothing else would fit; but that appeal really doesn't hold anymore.
All early operating systems were written in low-level languages, essentially assembler.
Come on, "early" is an irrelevant qualifier. The fact remains that languages like Modula-2, Modula-3, PL/1, Ada, Oberon, Lisp, Smalltalk, Cedar/Mesa, and Object Pascal have all been used for implementing operating systems, many of the commercial and widely used. The use of C for implementing operating systems is a fairly recent and mostly academic phenomenon.
You mean Lisp machines? That requires special hardware. [...] C is barely a step up above assembler. You can't get higher performance than assembler. [...] Perhaps you should re-read my post. From application-level programming perspectives, these features are well-motivated. From a microkernel perspective these features are ill-motivated, if feasible at all.
That's just the usual kinds of unfounded prejudices and misconceptions C fanboys have against all other languages and systems. It's really pointless to try to argue against them--there is a certain degree of stubbornness one simply can't overcome.
Well, it's pretty clear we aren't going to convince each other. For my work, I need a kernel into which I can hack new security and networking facilities. Sadly, the only realistic choices are C-based at this point. But between the available choices, the monolithic kernels look more attractive than the microkernels in every respect.
In the end, people like you will have to convince people like me that kernels like Mach and CapROS are worth the hassle, and you consistently fail to do that, in papers as well as discussions. It's really not surprising that new operating system approaches have made so little headway over the last several decades.
I think its directory layout is a bad design even for non-professionals, it's just that non-professionals either don't know any better or don't bother doing anything about it.
iPhoto is fairly nice in other respects, in particular its integration of editing and browsing, but unfortunately its editing facilities are still a bit too limited even once you discover the "Adjust" button.
Slashdot Mirror
Is there a response? What incremental, random changes produced an eye such that each step conferred an evolutionary advantage?
It's well understood; the progression is roughly: light sensitive cell, opaque pigment in back, retreat into concavity, formation of pinhole camera, transparent covering, fixed lens, adaptable lens. Each of those has distinct and individual evolutionary advantages, sometimes related to improved predator evasion and sometimes merely related to improved protection of the existing structure. It seems to have happened several times in evolution, so it's not even anything unusual; if we ever encounter aliens, they probably have eyes, too.
The problem for teachers is that ID can't be tested using the scientific method, the system of making, testing and retesting hypotheses that is the bedrock of science.
That's false. ID can be tested (in the same way astronomy can be), and the answer is: there is not a shred of evidence to support ID. Every single test of evolution has come down on the side of evolution (mutation and selection) and against intelligent design (interference of an intelligent agent in the development of different life forms on earth). ID has the form of a scientific theory, but it happens to be an incorrect scientific theory according to overwhelming evidence.
The Chinese government does not want to go down in history as a government that sends people to their death in interplanetary travel; it's bad PR, it's bad for business, and it doesn't even fit in with the philosophy underlying Chinese governmental authority.
In any case, what would the motivation be? The only reason for manned travel to Mars in the near future is as a publicity stunt and to make people believe that the universe works like it does in Star Trek. That illusion is going to be destroyed if you plan on leaving people there to die.
Scientifically, rovers are a far better investment right now and for the coming decades. They can stay there for months with almost no resource requirements, and they really don't have to come back.
My point was that binary modules linked with the kernel or interpreted in the kernel VM would share the same heap as the kernel itself.
There is no such thing as "the heap" in a modern garbage collected system. Modern garbage collectors have variable numbers of regions and areas that are utilized for different objects. These regions are utilized adaptively by the garbage collector and optionally under supervisor or program control. You can limit their sizes, their growth, their allocation rates, their placement in memory, or whatever else you can imagine.
You have to start inventing ad-hoc quota mechanisms to start trying to limit the damage, but I've seen no accounting system that is flexible enough for all cases.
These techniques were invented in the 1970's and 1980's. It really sounds to me like EROS is reinventing a limited form of modern memory management techniques, and not doing a good job at it.
Regardless of how much better GC is, you simply cannot do better than a fully-pluggable memory management system.
Well, I think one can certainly do a lot better than a fully-pluggable memory management system based on a microkernel architecture, both in terms of functionality and in terms of performance.
No, [UNIX users on multiuser machines] are not malicious users, though they may be untrusted users. If they were actually malicious, that system would be taken down within minutes of it being brought up; certainly not the mark of a secure or reliable system architecture.
A UNIX machine with hundreds of CS undergraduates has malicious users on it: all the DoS and security hacks you can imagine have been tried on it. If it survives that (and it does), it's good enough for most applications. It's not secure because of any principles, it's because in the decade after BSD and SunOS got released, people spent huge amounts of time fixing bugs and problems.
In many environments, it's not necessary to patch the kernel when there are local vulnerabilities.
suppose you're referring to uninitialized variables. Indeed, C imposes no restrictions on initialization, just like assembly. You are free to be as safe, or unsafe as you like within the limits provided by the language implementation.
No, I'm referring to the fact that the effect of dereferencing a pointer to an int as a char pointer is undefined, even if the int is initialized properly. All those things I mentioned are effects you might plausibly see even on an ANSI conformant C compiler for Pentium.
For the most part, that's very likely. But it's not here today, and that's all that matters when choosing a language here and now. Just cross your fingers that they get BitC right.
I don't have much hope for BitC; it may be technically alright, but if it doesn't look vaguely like something people know, it won't get used. I think the most likely next language for a widely used kernel will eventually be a Java-to-C translator, a C#-to-C translator, a safer variant of Objective C developed by Apple, or possibly ANSI C 2015 or something, where people finally clean up the mess.
This is a checklist item for Microsoft, to address problems like they are having in MA. It lets Microsoft zealots and people resistant to change in companies and the government buy MS Office even when there are regulations requiring support for open document formats. Microsoft has done the same with many other standards. For example, their POSIX support gets around POSIX requirements, but it is pretty much useless in practice. The fact that they are outsourcing this tells you how low priority it is for them--Microsoft almost never outsources anything that is of any importance to their business.
In my question I'm referring to malicious code which will not play by the rules of "good storage management".
Whatever constraints you impose on storage management without GC you can impose with GC. In fact, GCs are arguably inherently better at enforcing real-time or storage management constraint than manual allocators (Sun's tribulations with Java notwithstanding--a poor GC can indeed be a headache, but so can a poor malloc).
UNIX is inherently based on a collaborative system design; it was not designed to support secure collaboration however.
UNIX has been, and continues to be, widely used for large multiuser systems with sometimes thousands of untrusted and malicious users. It really works for that application.
Microkernels in safe verifiable languages like BitC+Coyotos will solve a great deal more of these problems (more than your design I'd wager). The solution is not to drag all that cruft into a single monolithic kernel. But that's just my opinion; well, mine and a great deal of others who share it.
Well, it's good that we do agree as much that writing a kernel in a new language is a good idea. The methodology and design of EROS/Coyotos is a separate discussion.
Because the "first byte of an int" depends on the endianness of the underlying machine.
Or you may get a completely unrelated value, an incorrect value every 4 billion accesses, or a crash.
This is a problem of ignorance, not the standard or the language itself
It's also a problem of software engineering because it's impossible looking at a piece of C code to determine if and where it uses unsafe or unportable constructs. That causes major problems when porting systems code (but I gather CapROS is Intel/gcc only at this point).
This sort of difficulty is completely unnecessary--it buys you nothing in terms of performance or expressiveness. It's a historical accident. It would be trivial to have a language that is in every respect like C but that cleanly separates its portable and defined constructs from implementation or undefined constructs.
care to support your assertions?
Look up NeXTStep, Zmacs, Smalltalk-80, and Tcl/Tk, for examples of GUI designers, Intellisense, integrated development/debugging, and dynamic layout. All of them predate the introduction of even Visual C++ 1.0.
Of course, your logic also holds Bowing and Lockheed guilty of copying the Wright Brothers. After all, all of their designs are really just modifications of those planes that came before them.
The analogy doesn't hold. Boeing and Lockheed actually improved on their predecessors, Microsoft is still trying to catch up.
Show me a Lisp/Smalltalk/high-level language implementation right now, that permits interfacing with low-level hardware (either natively or through FFI), that permits no runtime environment (no GC, etc.), and that has a free, downloadable compiler that compiles to native code.
People have written operating systems in Modula-3 and Ada, so those would be obvious choices and they have free implementations. Of course, not many people know them, which is kind of a problem.
In practice, I'd probably go with one or another Java implementation. If you like, you can use Java (but not the standard Java libraries) without a GC, but, as I mentioned, kernels should start using GC.
I'd probably first look into using one of the FOSS VMs (IBM's is very good) and build around that. Failing that, a Java-to-C compiler or gcj would be other possible choices.
As a last resort, I might consider using a subset of C++ with an exact garbage collector (they can be written as libraries) and enforcing the use of relatively safe constructs through a (custom written) style checker.
You cannot disagree with my statement because it is a simple observable fact: C provides direct access to datatypes which map almost directly to base valuetypes on commodity hardware. Commodity hardware: 32-bit and 64-bit desktop and server byte-addressable CPU architectures. C datatypes: char, int, long, long long (and unsigned variants)
The C language makes few guarantees about how machine datatypes map onto C datatypes. The mapping may even change according to compiler versions and compiler flags, and the mappings are often not the "obvious" ones.
Yes, if you use gcc extensions like __inline__, it's no longer technically "C". That does not mean you cannot write a kernel in pure C even on gcc. All host-specific code is encapsulated in separate asm files and linked to the portable C.
Even something as simple as accessing the first byte of an int through a char pointer is undefined. And your remarks highlight another problem with C: its users don't even realize when they are using undefined constructs.
It can, if we impose a type system on top of it. See LLVM for example. You might be interested in LLVA in fact. ("LLVA: A Low-level Virtual Instruction Set Architecture")
OK, so to summarize, you agree now that safe, garbage collected, object-oriented languages can give performance competitive with C, then. (It doesn't take a VM or a compiler/runtime more complicated than C, but that's a separate issue.)
[using the MMU to decompose a large software system (kernel or otherwise) into modules is often bad.] This is pure speculation. You have no real evidence to support this conjecture.
There is extensive evidence. If you want to check it for yourself, just try using DCOM or RMI to put some objects of an application into a separate address space. Not only is it a significant amount of work, it's also slow.
I'm still not clear whether you actually have processes, or they're simulated on the kernel VM,
As I was saying: "Such a kernel would be a drop-in replacement for Linux with no user visible changes". It would run Linux binaries, do scheduling and accounting in the same way, etc. The architecture, drivers, etc. all would have nearly identical code to what they have now. The main difference would be that unsafe uses of pointers would be eliminated by a better type system and that low-level access would happen through explicitly marked constructs.
Let's say you find a way to handle the above memory accounting issue, what if the malicious object now accumulated garbage and freed it immediately, causing a great deal of work for the garbage collector?
Despite the name, modern garbage collectors don't work that way; they are more of "live object collectors and categorizers". Furthermore, the cost of collection is amortized over the allocation: if a caller allocates a lot of objects, he has already paid his share of CPU time to free them, just like with malloc/free. Finally, the use of garbage collection doesn't give you license to abandon all principles of good storage management. In an environment like a kernel, you would likely use arenas, allowing you to recycle all resources associated with a user or a process instantly.
Notice, the above accounting issues are all existing problems with just about every popular kernel in existence, particularly UNIX clones.
Well, as a UNIX user, admin, and developer for a quarter of a century now, I can say that they are not practical problems in any real-world usage I have encountered. The practical problems with UNIX are lack of robustness against errors in dynamically loadable modules, difficulties associated with developing new kernel modules, and problems with module versioning.
C provides direct access to datatypes which map almost directly to base valuetypes on commodity hardware. This is a statement of the current state of affairs.
No, it is not. C is a language and it has no facilities for low-level access. There are lots of C compilers that provide such access as implementation defined extensions (all in slightly different ways), but it's not part of C. EROS is a kernel written in one or two C compiler-defined dialects, not a kernel written in C. The distinction matters because you can do exactly the same thing in many other languages, and usually in better ways.
The places where Visual Studio excels is in some of the following:
Code/syntax highlighting
Structure/layout
Designing graphical aspects (forms, window layouts, etc.)
And others
It appears to "excel" in these areas if you don't know the IDEs that VisualStudio copied these features from.
In fact, someone else did it: Emacs, Zmacs, and Hemlock had this feature already about 20 years ago. Unlike IntelliSense, the information was displayed unobtrusively in the status line, and you got completion only if you asked for it with tabs. Unlike IntelliSense, the information was also based on the actually loaded code (at least when writing Lisp code), not some separate database that may be out of date and may contain irrelevant data. But maybe Microsoft will "invent" that in another 20 years, too.
And here is the problem with this design: you seem to be ignoring the fact that it's running on a real machine without these high-level features
That is an imaginary problem, since standard hardware can support runtime safety, garbage collection, and object-oriented dispatch efficiently.
In some sense a microkernel is a workaround for safety issues: by decomposing functionality, you are isolating faults.
Decomposition is good; using the MMU to decompose a large software system (kernel or otherwise) into modules is often bad.
So you're idea is something like J-Kernel from the paper I linked to earlier, but actually running in kernel space and executing the whole system written in Java?
No, it would most definitely not be like J-Kernel, which I consider impractical at this point. "My idea" is a system very much like the Linux or BSD kernel but written in a compiled language with garbage collection, runtime safety, well-defined primitives for accessing memory and hardware, and a simple object system. Such a kernel would be a drop-in replacement for Linux with no user visible changes (it could be system call compatible and user mode code could be written in any language whatsoever), and few changes in the way the kernel itself is developed. The main benefits would be easier testing, easier extensibility, fewer security problems, and better support for dynamically loadable modules.
In real life, I think one might have to go there incrementally, by adding support for another systems programming language to the Linux and/or BSD kernel. A JVM/JIT wouldn't be my first choice, but it seems like the most realistic, and would be useful for intelligent packet filtering, security, file systems, network protocols, and some drivers. There has been some work on that (TeaseMe), but it will only take off when a good JIT is integrated.
C provides direct access to datatypes which map almost directly to base valuetypes on commodity hardware. Good so far?
No, sorry, that is not true. There are languages that make such guarantees, but C isn't one of them. People using C for such purposes are relying on non-standard (if common) features; the problem is that C never alerts you to this fact, and that these features can behave differently even between different versions of the compiler or with different flags.
C itself makes no real runtime assumptions beyond the presence of a stack.
The C language makes no guarantees whatsoever about real time behavior or memory management behavior. There may or may not be a stack. There may or may not be dynamic memory allocation even if you don't ever call "malloc". There may or may not be a garbage collector.
That's just it, there effectively is no "C runtime" though
C has as much or as little of a runtime as the compiler implementor happend to choose. GNU C happens to use a fairly simple runtime, as did K&R C, but other C compilers don't. Some have garbage collectors, for example. Try running your code under tcc in "safe" mode.
Conversely, just like you can pick an implementation of C and use a stripped down runtime with it, you can do the same in Lisp, Modula-3, or Java, and people have. You can write code in those languages that never invokes the garbage collector.
I'm sorry, but you still don't get my point. C is a programming language defined by a language standard. You keep claiming that C permits efficient access to low-level features, but that claim is wrong: the C programming language provides no access to low level features at all. Any such access is provided via implementation-dependent features. And just like it's easy to add those features to C, it's easy to add them to other languages.
Similarly, the C runtime is unsuitable for use in a kernel; any kernel use of C needs its own runtime. Well, you do the same thing when writing a kernel in any other language.
I think I have stated this pretty clearly already: I'm looking for a kernel very much like the Linux or BSD kernel, but replacing C with a simple, safe object-oriented language that supports garbage collection. It would be nice if the runtime contained a JIT compiler, but that would not be necessary. Either way, modules would be dynamically loadable, communication between kernel components would happen through the language's object oriented mechanisms, and the language and runtime would guarantee fault isolation and resource reclamation (except in contexts where the programmer explicitly overrides it).
That's in contrast to microkernels that take the functionality currently contained in kernels like Linux or BSD and split it up among separate processes; the microkernel approach introduces unnecessary overhead and complications into the kernel design. I view microkernels as a workaround for the lack of runtime safety in languages like C.
It's also in contrast to the Lisp machine or some attempts at a Java OS, in which everything, including applications, runs inside a single big runtime and address space; resource management and other issues haven't been solved satisfactorily in that context and current applications aren't written for that kind of environment. However, once you have moved to a safe language and runtime, you can start exploring the issue of moving more functionality into the kernel's address space.
ANSI C strictly forbids field reordering. In-memory layout is precisely as defined in the struct to the nearest byte (bit fields are known to be non-portable and thus unusable).
ANSI C forbids reordering, but it doesn't define the size of a byte, nor does it precisely define alignment or padding.
Portable malloc/free API. Not sure what you're point is here.
The point is that the kernel doesn't use the language-supplied malloc/free; it uses its own unportable mechanisms.
"memory mapped I/O" Not part of the C standard.
My point exactly.
I merely said that C is the currently probably the best candidate, particularly for a microkernel implementation.
Well, and as we can see, you don't even know what features are part of the C language. C implementations can be garbage collected and abort on every one of those "low-level features" you so dearly love as a runtime error. The C language is not so much characterized by the presence of low level features, but by the absence of guarantees that high level features are present.
But even if you are trying to argue that using GNU C or a similar compiler on Intel is "the best candidate", that is also methodologically wrong: you claim that systems like EROS and CapROS are aimed at performance and security. But if you base those systems based on a language with no language mechanisms for fault isolation and a propensity for serious security problems, then EROS and CapROS can be argued to be merely workarounds for the limitations of their implementation language.
"The other big problem with C is that many of its low-level features unnecessarily rely on implementation-specific features that aren't part of the standard." Very few such features exist, they are well known, and they aren't used in operating system kernels. You are really grasping at straws here.
No, you simply do not understand C at all. Structure layout, memory access through cast pointers, memory allocation and management, and memory mapped I/O are all highly implementation dependent features.
Hence the reason for developing BitC and Coyotos.
Which only goes to show that all your objections to how one can't use anything other than C are groundless.
We could now start debating whether the methodology underlying the development of EROS and Coyotos ("principle based") is acceptable and whether the authors are actually properly testing their claims or just handwaving. We could also discuss whether even the principles that the authors of those systems pursue hold water or are misguided. But those are separate discussions.
and it should have been:
1. low-level control over representation/memory layout (hardware interfacing demands this)
C does not have sufficient low-level control over representation/memory layout for hardware interfacing; any such "control" you get is highly implementation dependent, and the same kernel code compiled with different C compilers will often not work. Furthermore, there are no defined semantics for the code the compiler generates for accessing data structures, which can cause problems with memory mapped I/O.
2. control over runtime environment (you can't enforce a runtime environment on some kernel designs)
You have no more and no less control over the runtime environment in C than you do in any other language. A conforming C compiler and runtime is perfectly free to use garbage collection and to terminate your program the first time you use "float x; return *(char*)".
When C is used for implementing kernels, people generally take the C compiler and implement their own C runtime for the kernel. That's exactly what you do when you use any other language, including languages with garbage collection, for implementing kernels.
3. readily available compiler (preferably free as in beer, at the very least, since academic and free software work is, well, free)
Building a small compiler to support kernel development is negligible amounts of work compared to implementing the kernel, and there are plenty of compilers you could build on (Eiffel, Portable Object Compiler, tcc, Oberon, etc.).
The only candidates that fit that bill are, you guessed it: C, C++, Ada, Pascal
C and C++ do not fit your requirements at all; you only think they do because you confuse implementation-specific features that are not part of the language with the language itself. At best, you can argue that the specific language implemented by gcc with an Intel backend is good for your purposes, but the C programming language is, by itself, completely unsuitable for implementing kernels.
I'm not sure precisely how that translates into a conclusion that microkernels, and that CapROS in particular, aren't worthwhile.
Here, I have criticized EROS (and by extension CapROS) for its microkernel design--relying on memory management hardware rather than language and runtime mechanisms in order to achieve correctness. Apparently, the EROS authors themselves realized this, which is why Coyotos uses a new programming language.
I haven't said anything beyond that about EROS; that would be an entirely separate discussion. The fundamental problem I see with the EROS approach is that it is principle driven, rather than data driven. These people keep talking about "trust" and "security" without ever demonstrating measurable improvements in those areas under controlled conditions.
It really is a shame that people get so stuck in their mindsets and are so convinced they're right that they fail to acknowledge evidence to the contrary, isn't it?
Well, you should know.
I know that work. It's an article talking about retrofitting a process model into a JVM. I don't think that's the right approach right now for many reasons.
I think the right choice for an OS kernel at this point is something very similar to the Linux kernel, with separate user-mode processes. It's just that the OS kernel itself should be implemented in something a little higher level and a little safer than C, with the ability to drop down into unsafe operations when necessary and with some simple object oriented facilities. Oberon seems like roughly the right level, or maybe a Java implementation like gcj.
But tell you what: why don't you do some research about languages and OS implementations. Go look at languages like PL/1, Oberon, Algol, Modula-2, Modula-3, Ada, and Cedar/Mesa. Go look at operating systems like Alto, Multics, Oberon, Apple Lisa, and B5000.
Also, go look up the discussions on aliasing and pointers in C and all the problems that caused with optimization over the evolution of C since K&R. C is not, and has never been, a high performance language and often does not come even close to carefully written assembly language or well-optimized code for other languages. The other big problem with C is that many of its low-level features unnecessarily rely on implementation-specific features that aren't part of the standard. C was a charming language 20 years ago because it fit onto a PDP-11 and on a CP/M machine where nothing else would fit; but that appeal really doesn't hold anymore.
All early operating systems were written in low-level languages, essentially assembler.
Come on, "early" is an irrelevant qualifier. The fact remains that languages like Modula-2, Modula-3, PL/1, Ada, Oberon, Lisp, Smalltalk, Cedar/Mesa, and Object Pascal have all been used for implementing operating systems, many of the commercial and widely used. The use of C for implementing operating systems is a fairly recent and mostly academic phenomenon.
You mean Lisp machines? That requires special hardware. [...] C is barely a step up above assembler. You can't get higher performance than assembler. [...] Perhaps you should re-read my post. From application-level programming perspectives, these features are well-motivated. From a microkernel perspective these features are ill-motivated, if feasible at all.
That's just the usual kinds of unfounded prejudices and misconceptions C fanboys have against all other languages and systems. It's really pointless to try to argue against them--there is a certain degree of stubbornness one simply can't overcome.
Well, it's pretty clear we aren't going to convince each other. For my work, I need a kernel into which I can hack new security and networking facilities. Sadly, the only realistic choices are C-based at this point. But between the available choices, the monolithic kernels look more attractive than the microkernels in every respect.
In the end, people like you will have to convince people like me that kernels like Mach and CapROS are worth the hassle, and you consistently fail to do that, in papers as well as discussions. It's really not surprising that new operating system approaches have made so little headway over the last several decades.
I think its directory layout is a bad design even for non-professionals, it's just that non-professionals either don't know any better or don't bother doing anything about it.
iPhoto is fairly nice in other respects, in particular its integration of editing and browsing, but unfortunately its editing facilities are still a bit too limited even once you discover the "Adjust" button.