Heap Protection Mechanism

An anonymous reader writes "There's an article by Jason Miller on innovation in Unix that talks about OpenBSD's new heap protection mechanism as a major boon for security. Sounds like OpenBSD is going to be the first to support this new security method."

Comments

Slowdown?

[(1 -sqrt(5))*(2**-1)]

Continues Theo,

A number of other similar changes which are too dangerous for normal software or cause too much of a slowdown are available as malloc options as described in the manual page.

Id est, they stopped before reaching a Java-like retardation.

Re:Slowdown?

[Anonymous Coward]

What are you talking about? Java is the answer to every problem.

Re:Slowdown?

[Anonymous Coward]

If I wanted an interpreted language, Id use QBasic!!

Re:Slowdown?

[Anonymous Coward]

the original post was the troll, really please stop kicking Java around

Re:Slowdown?

[(1 -sqrt(5))*(2**-1)]

Trying programming in Java sometime outside of a website applet, kid, and maybe you will learn something.

I'm quite willing to concede the argument that programmer time is dearer than processor time; but why has the the Real Time Specification for Java, for instance, only been able to achieve 100 microsecond interrupt response times, when 2.0 microsecond response times aren't unheard of in other domains?

Re:Slowdown?

[Anonymous Coward]

You Java apologists are worse than creationists.

Re:Slowdown?

[Anonymous Coward]

Ho hum.

http://www-128.ibm.com/developerworks/java/library /j-jtp09275.html

Malloc is slow. Per studies, 20-30% of CPU time wasted on memory management.

I haven't seen that level of retardation in JVM's since... oh... 1996?

But yeah, keep thinking you can do it better. Whatever. In the meanwhile, the rest of the world moves on.

Re:Slowdown?

[ergo98]

Malloc is slow. Per studies, 20-30% of CPU time wasted on memory management.

Interesting paper - thanks for the link.

However I find the conclusions given a bit dubious. For instance the claim that allocations are "free" is somewhat dubious - garbage collecting languages put all of the work on the back-end, giving the illusion of a free front-end (whereas non-GC languages put the hard work on the front-end). Yet for every object you create on the heap that is more work the heap walker has to do each GC to detect orphaned objects - a non-trivial task. It then has to free all of those objects, and because of the "free" allocation it has to move all of the objects and rebase every object pointer in the application. It doesn't take a genius to realize that is a signficiant task in an application of a real-world size.

I think the proof is in the empirical proof - how many high performance memory intensive Java applications are there?

Re:Slowdown?

[Retric]

Java's RTspec is not designed around those applications. In the Real time world a lot of people are still using ASM because in plenty of applications programmer time is much cheaper than CPU time. If your paying for 10,000,000 CPU's then and 6 programmers then paying for 50c CPU's vs. 1$ CPU's is worth a lot.

I work with real time systems and 0.0001 seconds (100 microseconds) is plenty fast for most Human to Real time systems applications. Granted JAVA is not what you want for fine-tuning your Engines performance but it's plenty fast for most applications. What makes Java so useful is you get to avoid most of the really time consuming bugs. Compare a fully functional java based multithreaded HTTP server with the C / C++ equivalent and it's going to be 1/3rd as much code. And will operate at vary close to the same speeds. In other words it's designed around applications where programmer time is worth more than machine time. We already have C so Java was built around the 95% of applications that don't need inline ASM.

I have killed BSD UNIX with buggy C networking code which is the only thing I have been unable to duplicate with good Java code. You can do bit twiddling in Java, but it's faster in C. You can have hundreds of threads doing their own thing in either but it's much easer to do that in Java than C/C++. The secret is to know enough about how Java works so that you avoid things like creating new threads that eat up a lot of time. Once you understand how things work you can use things like Thread Pooling that are extremely efficient. Instead of complaining that concatenating Strings takes so long try learning about what other tools are out there like StringBuffer.

PS: A quick look at some fast Java code. (It is a bit dated but gives you some idea what I am talking about.)

Re:Slowdown?

[JohnQPublic]

PS: A quick look at some fast Java code. (It is a bit dated but gives you some idea what I am talking about.)

Hmm ... that's a little archaic. The page complains about Java performance in JDK 1.1, and appears to be based on a site (http://www.cs.cmu.edu/~jch/java/optimization.html ) that hasn't been updated since 1998.

Re:Slowdown?

[liloldme]

It doesn't take a genius to realize that is a signficiant task in an application of a real-world size.

And anyone who's run a JVM knows about the price of this task -- yes GC takes time.

However, as I understood the article, the author was making a point that the way most C programmers manage memory tends to make the task more time consuming than is necessary. Therefore relying on a known optimized implementation rather than reinventing the wheel every time may be preferred. After all, it is just the VM implementor that needs to understand how to optimize the memory management, not the application developers. So yes, where the time is spent is shifted but also the amount of total execution time spent on memory management can be reduced -- because the task is managed differently.

As for the specific details of this paper, they're basically discussing how to determine which objects can be safely allocated from the stack, instead of heap, and therefore can be discarded without the usual book keeping required from a heap GC.

how many high performance memory intensive Java applications are there

Java is so widely used on the server side and middleware, it cannot be difficult to come up with examples -- Tomcat, J2EE app servers, etc. eBay for instance advertizes very clearly on their front page to be powered by Sun's Java technology. There are individual Java systems that manage millions of transactions daily, and there must be thousands of systems out there that do this every day with Java.

Re:Slowdown?

[eric76]

About 15 years ago I started using a technique to improve performance when doing lots and lots of very short term mallocs.

Essentially, I'd create a large ring buffer of malloced temporary buffers of some standard length. Any time a temporary buffer was needed, I'd grab the next one in the ring.

Before the buffer was provided to the function asking for it, the length would be checked. If the requested length was longer than the current length, the buffer would be freed and one of at least the proper length would be allocated. (I normally allocated by buffers in byte multiples of some fixed constant, usually 32.)

The idea was that by the time it was reused, what was already in the buffer was no longer needed. To achieve that, I'd estimate how many buffers might be needed in the worst case and then multiply that number by 10 for safety's sake.

My primary use of this was when doing enormous numbers of allocations of memory for formatting purposes. The function doing the formatting would request a buffer large enough to hold whatever it would need, write the formatted data into the buffer, and then return a pointer to the buffer. The calling function would simply use the buffer and never have to worry about freeing it.

The performance results were superb except in the very simplest cases where you allocated the buffers without ever using them.

I've never known anyone else who used this kind of approach although I've showed it to a large number of people.

Re:Slowdown?

[Dan Farina]

Many people have used this sort of approach. It is called "pooling."

Its use dates back to the first GC'd langauge, LISP, and was a common way to reduce garbage generation.

Re:Slowdown?

[Krach42]

Well the same issue exists with Copy-on-Write. COW implementations give an impression of faster copying, because they back off all the copying until the first damage to that information. Turns out that in some cases you don't need to make an actual fresh copy in every situation, and that sometimes, you just copy it to use it, don't damage it then just return. In those cases COW gains you time. Since you only copy when you must. You just have to deal with blocking during the first write to the copy. (You would have waited anyways during the copy, you just shift were it's at.)

The issue with GC vs. non-GC languages is that GC pushes all that GC stuff to the back end, but ask yourself, what if you never have to collect the garbage? The whole question lies in, if we wait until the back-end to clean everything up, will we gain any time? Considering that you spend less time on important non-transient things with GC, and potentially more time on very transient things, while you spend about the same time on both in non-GC, the question is How transient is your memory? Of course, the less transient your memory is, the less often you'll have to reallocate space for it in the non-GC case... so...

Re:Slowdown?

[eric76]

I've heard of pooling but was under the impression that you were mallocing several memory allocations at once, but that the pieces of allocated memory were assigned to particular uses.

The primary benefit of this was to perform one malloc instead of N mallocs for N different memory assignments.

I didn't know that anyone used this to automatically reuse the memory assignments whenever and wherever appropriate.

Thanks for the information.

Re:Slowdown?

[TheRaven64]

And it can seriously increase the speed of a bit of code. I recently profiled some of my code and found that it was spending around 40% of its time in malloc and free. I created a very simple memory pool (no ring buffers, just a simple linked list with insertions and removals at the same end, nice and fast) for each frequently-allocated object type, and saw a huge speed increase.

Re:Slowdown?

[crush]

I work with real time systems and 0.0001 seconds (100 microseconds)

Wouldn't that be 100 milliseconds? Ob wikipedia link

Re:Slowdown?

[Anonymous Coward]

A number of other similar changes which are too dangerous for normal software or cause too much of a slowdown are available as malloc options as described in the manual page.

And yet HP Labs built an HP/UX emulator/VM, and ran it on HP/UX, and programs generally ran faster than just running them normally.

I haven't used Java in a while, but if it has poor performance, that's because of poor design or implementation, not because safety and performance are at odds.

Re:Slowdown?

[cananian]

Andrew Appel's papers include many on this topic; the first is "Garbage Collection can be better than Stack Allocation" and the most recent seems to be "Cache Performance of Fast-Allocating Programs". Basically, the amortized time taken by fast allocation and then later garbage collection is less than the combined time to allocate and deallocate memory the "usual" way. The memory locality of the garbage collector can be made very good as well. I believe a detailed discussion in in Appels "Modern Compiler Implementation" textbook(s).

Re:Slowdown?

[ScrewMaster]

I'm waiting for the first claim that Java could not have arisen by accident.

Re:Slowdown?

[Fulcrum of Evil]

I've heard of pooling but was under the impression that you were mallocing several memory allocations at once, but that the pieces of allocated memory were assigned to particular uses.

That is basically what you just described - malloc std_length * count. Allocate out of this pool preferentially.

The primary benefit of this was to perform one malloc instead of N mallocs for N different memory assignments.

This performs 1 + N*M(1-hit_rate) allocations instead of N*M allocations, which is generally really nice.

Re:Slowdown?

[lgw]

Malloc is slow, but I almost never malloc anything in C++. Everything that possibly can lives on the stack. A different coding style is required, but it tames the resource leaks, even when exceptions are flying around, and it's as fast as memory management gets. Were it not for the limits of C++'s polymophism, I'd never need to return a pointer to a new object from any function, and as it is that's quite rare.

Re:Slowdown?

[Fulcrum of Evil]

I'm waiting for the first claim that Java could not have arisen by accident.

I'm pretty sure that Java was done on purpose.

Re:Slowdown?

[eric76]

What I described was to allocate a list of buffers and reuse them for very short termed uses without needing to check to see if what was using them before was finished with them.

In other words, you could conceivably have 1,000 or more functions sharing some fixed number of buffers (say 100 buffers) between them. When a function needs a buffer, it would just grab the least recently used buffer and reuse it.

The idea depends on the buffers being used for a short enough time that when you need a new buffer, whatever was using the least recently used buffer no longer has any need of it.

It's really a much different idea.

If you needed something for a longer term, you would allocate it normally with malloc or out of a pool as you describe.

Re:Slowdown?

[Fulcrum of Evil]

What I described was to allocate a list of buffers and reuse them for very short termed uses without needing to check to see if what was using them before was finished with them.

This can also be done by allocating a single larger memory region and maintaining a free/busy bitmap.

The idea depends on the buffers being used for a short enough time that when you need a new buffer, whatever was using the least recently used buffer no longer has any need of it.

LRU is not a requirement.

It's really a much different idea.

It's exactly the same. The linked list structure is not central to the design of your pool.

Re:Slowdown?

[Dan Farina]

What it sounds like you are doing is effectively unsafe pooling with ring buffer since you do not check for existence of an allocation by anyone (that you detail insofar). You rely on the fact that by the time you get around to using a chunk of memory again that it has since been finished with. (Which means any code that relies on this unchecked ringbuffer scheme is very brittle and the user of said code will always be suspicious if they are tripping on their own memory, but brittle is okay if you never expect it to grow, or it can occassionally crash or get wrong answers...)

This has also been fairly common trick (in some sense) with FORTRAN programs since the earlier versions of FORTRAN had no built-in mechanism for dynamic allocation of heap space (like malloc) (this is so as I have been told; I myself am not a user of old-school FORTRAN). You would have to make a big vector and basically write something like malloc (for a "general memory pool" that we nowadays call the heap, to an order of approximation) or employ collection of vectors that would each be used as a pool of memory, or both. Using a ringbuffer with a refcount for safety was one simple, effective, and common approach...and, if you needed even more speed (because whatever it was is inside a quadruple loop) you could do unchecked grabbing of stuff from the pool if you were reasonably confident your code was of high quality and that mistakes would be quickly caught.

I don't quite consider this in the same category of actions since it was compulsary (no concept of preventing memory fragmentation since it was statically allocated or 'helping' the garbage collector since there was none), but it's interesting how old knowledge like this is rediscovered.

Re:Slowdown?

[eric76]

What you are describing is for an entirely different type of problem.

What you are describing will require a search of the bitmap to locate an available buffer. It will also require that the program mark the buffers as available once they are no longer needed.

That is not at all what I was describing.

What I was describing was reusing buffers without needing to check whether or not they were being used by anything else because of the nature of the use of those buffers, i.e. by the very limited time the buffers would be used and by using the least recently used buffer, there is no need to check to see if they are being used because they would always be available.

And, of course, for that type of thing, LRU is pretty much required. I guess you could use the next to the least recently used buffer, but there wouldn't be any point of doing that.

In other words, the buffers are immediately available with no need to search for an available buffer. Just choose the next buffer (which is the least recently used).

My scheme also allows you to increase the buffer size if the available buffer size is too small.

In what I was describing, you would typically allocate N buffers with N mallocs. Then, when a buffer was too small, you just free that buffer and malloc it with a larger request.

By the way, I never claimed that a linked list structure was central. You could just as easily maintain an array of 100 pointers and an index into the array to find the location of the LRU buffer. It is O(1) either way.

The first time I ever did this, I allocated the N buffers with one malloc and had a flag bitmap that indicated whether or not each buffer was a buffer or a pointer to a larger buffer. Thus, in most cases, it performed one malloc and one free, did not require a search to find an available buffer, and temporary assignments were usually constant time.

Re:Slowdown?

[jnf]

yes, its slow, thats why most implementation act the way they do. As opposed to actually getting memory from the machine/returning it immediately on free, they tend to coalesce them into bigger chunks and stick them in a cache of sorts, the idea here is to avoid fragmentation (by coalescing them into bigger chunks), and to speed things up some.

With that said, I'd imagine this 'new' method of Theo's is even more slow and is very hurtful to performance.

Re:Slowdown?

[jnf]

As far as I know, just about every implementation of malloc uses a variation on this method, except for openbsd now. I know dlmalloc uses bins that are sorted by size.

Re:Slowdown?

[eric76]

You got it.

It would be unsafe if there was any possibility of the buffers needing to stay around very long. Like I said, I generally allocate 10x the maximum number I really think I'll need just to be safe. In general, my goal is performance, limiting heap fragmentation, or both.

I've never tried to use this where it wasn't appropriate -- i.e., where the buffer was needed for more than just a very short-term thing.

Of course, you couldn't use this in a multi-threaded program unless only one thread needed short term buffers. For a multi-threaded program, you would really need to have a separate allocation of buffers per thread or use a method of locking the buffers and not reallocating those buffers that are still locked.

Re:Slowdown?

[Anonymous Coward]

timely article. just before this article appears at a conference next month to debunk it:

http://www.cs.umass.edu/~emery/pubs/gcvsmalloc.pdf on Quantifying the Performance of Garbage Collection versus Explicit Memory Management.

very cool experiment that simulates replacing garbage collection in Java with malloc and shows (a) GC can (barely) be as fast as malloc and that (b) it takes around five times as much memory to get that performance and (c) if you have paging, you're toast. also those studies cited by the IBM guy are ancient (1993, 1994)...

Re:Slowdown?

[Johnno74]

Process explorer from SysInternals queries the performance counters of .Net apps, showing some interesting figures. A couple of the metrics shown are % time in JIT and % time in GC (relative to the total CPU time of course)

They are very interesting. Start a reasonably complex app, and do stuff for the first time. % time in JIT will probably be 1% (and keep dropping.)

The % time in GC normally sits around 1%.

Re:Slowdown?

[Anonymous Coward]

The compiler from the company everyone loves to hate, did this way back in VC++ 4.1, around 10 years ago. People like to talk about how optimized JVM's are, but they forget that compilers not only had a head start but also continue to receive optimizations. I'm sure by now all C runtimes include this.

Re:Slowdown?

[SillyNickName4me]

After all, it is just the VM implementor that needs to understand how to optimize the memory management, not the application developers.

They will still need some understanding of how it works, if they don't have that they are bound to use it in the most inefficient way possible...

So yes, where the time is spent is shifted but also the amount of total execution time spent on memory management can be reduced -- because the task is managed differently.

It is possible to do it more efficient with a GC when compared to a crappy non-GC based solution, but when you just write out what has to happen to make the GC possible, you will see that it is never the most efficient solution, and what is more, that the typical solution in C is often faster then the best optimized GC solution because it is a lot less work. Garbage collection saves work and time for the programmer and prevents a certain type of bugs. It comes at a price however. As mentioned, performance is likely worse, and what is often worse, you cannot make good predictions about where and when time is going to be spent on memory management, making it a lot more difficult to find and eliminate memory management related performance issues.

Re:Slowdown?

[Anonymous Coward]

If your paying for 10,000,000 CPU's then and 6 programmers then paying for 50c CPU's vs. 1$ CPU's is worth a lot.

"you're", "CPUs".

In the Real time world
I work with real time systems
etc.

"real-time".

Also, you are missing several commas (In the Real time world a lot, Granted JAVA is not, In other words it's, etc.).

Re:Slowdown?

[Anonymous Coward]

0.0001 seconds = 0.1 milliseconds = 100 microseconds.

Re:Slowdown?

[Anonymous Coward]

Perhaps you haven't done a lot of coding then. Memory pooling has been around for ages..
Memory pools usually also include a two phase constructor/destructor to run on the blocks. One phase will run upon real allocation/deallocation a pair of initializer/deinitilizer will run on every block returned from the pool. Things like mutex creation and other expensive operations will then only need to be done once per block, and then reused later. By the way, you can lookup slab allocator for eg.

Re:Slowdown?

[bani]

web applications are generally not time-critical, thus stuff like a couple hundred ms of GC won't necessarily hurt you. you hardly notice GC among the noise of network i/o etc.

when you get into GUI applications that's when it becomes really noticeable. the low performance and lag becomes annoying.

and when you get into the realm of realtime applications, games, etc. it's catastrophic.

i think the issue is that with C, you have control over exactly when and how the GC is done, so you can schedule it in ways it doesn't hurt your application.

Re:Slowdown?

[liloldme]

Both real-time GC and concurrent GC algorithms exist, which addresses both the points you make. The standard Sun distribution includes a concurrent GC for GUI applications.

GC algorithms are configurable on most modern JVMs.

cool

[chrisxkelley]

now unix will be more secure than all... well, again.

Re:cool

[mysqlrocks]

now unix will be more secure than all... well, again.

What do you mean? Windows Vista will be more secure than Unix. j/k

Re:cool

[miffo.swe]

"What do you mean? Windows Vista will be more secure than Unix. j/k"

Just like...
Windows 95 was more secure than Windows 3.1
Windows 98 was more secure than Windows 95
Windows NT was much more secure than Windows 98
Windows 2000 was the mother of all security
Windows XP, this time we got it right
Windows XP SP2 this time we really really got it right, promise, cross my heart and hope to die
Windows 2003, most secure server OS ever built!
Windows VISTA, even better than the worlds best system ever built, this time ill put my mother up here on this 100 fot pole and if im wrong may she fall down into that pit of crocodiles!

Until i have a real life experience of good Windows security i will tend to think back and remember all the former promises that have gone down the drain. Today you expect Microsoft to promise things that arent really true.

Re:cool

[Iriel]

Then again, while I'm not defending Microsoft, here's my take on their 'security':

Yes, plenty, and maybe even most of their promises about being a generally secure system are complete and utter rubbish. However, I'm willing to bet that each of their OSes are more secure than the last one. The problem is that they still leave plenty of holes open when they do things like (to point out the landmark example) weld the web browser to the kernel. I know that most people crack windows because it's easy, but while I may be wrong on this, I think people will continue to spend thier efforts on Windows even if their security was (competely hypothetically) top-notch only because of the bad reputation that precedes them.

I'm not saying that Windows is secure or that it ever will be. However, their security has improved, regardless of how poorly. The last reason on the list to crack Windows (in my opinion) and possibly the strongest reason is that they have a history of poor security. I think script-kiddies will pour ANY amount of effort into destroying any version of Windows just to keep that idea alive.

As much as I love Linux, I really doubt that any one distro (like RedHat for example) would be able to keep their system as secure as it is now if they were the entire world's information security scapegoat as MS is now. (PS, yes I know that MS does, mostly deserve the title they hold)

(I'm not a security expert, nor am I claiming to be, so if you think I'm wrong all I ask is that you not 'correct' me with a torch ^_^)

Re:cool

[Anonymous Coward]

You realize that "j/k" stands for just kidding, yes?

Re:cool

[resiak]

they do things like (to point out the landmark example) weld the web browser to the kernel.

For about the 35,348th time: no, no they don't. IE has nothing to do with the kernel. Go and learn what a kernel is. Hope that helps, have a nice day. :-)

Re:cool

[Anonymous Coward]

It's not that MS Windows (2000K or later) can't be secured, it's that Microsoft has strange defaults and in some cases makes it un-necessarily complex to secure the system. The defaults and the excessive features that are enabled that are security problems by themselves are well documented. Add to that the tools included with Windows are weak, incomplete, and often wrong intentionally and you have problems from the start.

None of these problems occur on any other OS. If one of the others were instantly as widely used as Windows, yes there would be more security issues discovered but not as many as are routinely discovered under Windows.

Having the OS closed source also raises security concerns as the binaries are much harder to verify as opposed to the source code available elsewhere (not just Linux source, but the BSDs and open Solaris).

Microsoft could do themselves a favor by simplifying the base OS (as they do occasionally though features creep in) and providing better default analysis tools -- hell, just buy the Sysinternals tools and bundle them!

Re:cool

[miffo.swe]

Well they do tie it into systems deep down in the OS. While its not literally in the kernel its pretty embedded in the OS. I agree with his point that Internet Explorer is tied down artificially into the core of the OS albeit not in the running kernel.

Re:cool

[grub]

It's not that MS Windows (2000K or later)

Windows 2,000,000? When was that released?

Re:cool

[mysqlrocks]

"Flamebait" - that's funny. Do I need to spell out the fact that I'm making a joke? Oh wait, I did. I guess some people don't understand that "j/k" means "just kidding". Oh well.

Re:cool

[Orgazmus]

Its just Windows 2000 with a 1 added for each bugfix

Re:cool

[neomunk]

Would it make you feel better if I said that they marry IE to the one-and-only, doing-things-most-OSs-do-in-the-kernel shell? Is that better? Oh, and screw a nice day, have an exciting month!

Re:cool

[sosume]

This was the funniest thing I read for a while on slashdot!

Re:cool

[Anonymous Coward]

"However, their security has improved, regardless of how poorly."

      MS have added all kinds of security feature certainly-- but the number of different attack vectors have also increased dramatically and need to be considered in your equation. I guess the ultimate gage would be objective statistics of the availability rate of all the world PCs. My personal experience suggests the balance of security has actual DECREASED in their systems. (Spyware alone is now causing billions in additional costs annually)

      MS is clearly a reactive company (not proactive) and it shows in that they tend to innovate remarkably very little considering the money they have. This isn't really a MS slam, that's just how they operate to keep their costs. down and profits up. It's truly ironic that the company that wishes to protect innovation with patents, is the single largest adopter of other indivduals truly unique ideas. I can't think of one thing-- not ONE-- that MS created first that changed the industry. They plonk true innovators by slightly modifying their ideas (enough to get a patent) and integrating it into their OS. If that doesn't work they then use that patent to litigate true competitors out of existence.

      If they actually chose to tap into a few billion of their profits they might actually improve the product beyond cosmetics. They need to work with industry to resolve the fact the OS and hardware remain too architecturally flawed to provide robust security. Root kits are going to become popular among script kiddies soon and this will demolish Vista just like spyware vaporized XP.

      They need to take a cue from IBMs Enterprise Systems Architecture and cleanly separate out the OS from the software subsystem and applications using far better memory partitions. Their version of protected memory just isn't close to robust enough. With the coin they have I'm surprised it's taking this long. The funny thing about this is is even though I dread the Trusted Computing platform, TC will likely be the first step to help in this endevour.

Re:cool

[Anonymous Coward]

Until i have a real life experience of good Windows security i will tend to think back and remember all the former promises that have gone down the drain.

Then you're remembering a lot promises that were never made. MS never touted superior security for upgrading from 3.1 to 95 to 98 to NT (BTW, NT was not a successor to 98, it existed back in the 3.1 days) to 2000. MS didn't feign seriousness about security until 2003 Server and XP.

But enjoy the karma ignorant moderators hand out for ignorant rants they simply agree with.

Re:cool

[Anonymous Coward]

Ballmer is that you?

new method?

[iggymanz]

other OS have had heap protection mechanisms, even one from Microsoft.

Re:new method?

[Intron]

ISTR that MS tried doing immediate free and it broke some programs that depend on the memory still being sround after being freed, so they made it optional. Sounds like OpenBSD is playing hardball here.

Re:new method?

[fitten]

ISTR that MS tried doing immediate free and it broke some programs that depend on the memory still being sround after being freed,

Sounds like those programs were broke from the start.

Re:new method?

[dougmc]

Sounds like those programs were broke from the start

Perhaps, but look at it from the user's perspective :

First, their system is working properly. Their OS is working, and their application is working. And then SP2 comes out, and they install it. And their application stops working.

Who are they likely to blame? Microsoft of the creator of their application? They'll look at what changed, and decide that Microsoft is to blame. And if this application is important, they'll even back out SP2 to make it work again.

Re:new method?

[afidel]

In the three cases we ran into when I was a consultant last year we placed the blame squarely on the shoulders of the vendor with the foobar product. They were coding in an unsafe manner to an undocumented API, it was 100% their fault. Now a naive home user with no real computer savy might blame MS, but when you look at the tradeoffs it seems like the security cleanup was good for the entire computer using public, even if a few people did have an app or two broken.

Re:new method?

[JohanV]

You mean the Data Execute Protection from Microsoft? OpenBSD has had that for a long time already, only they named it w^x.

This new feature from OpenBSD is the use of guard pages and the immediate freeing of memory. In essence this means that both bad programming and exploit attempts are much more likely to result in a core dump then some unidentifiable and non reproducible corruption or a working exploit. Many people consider that a good thing because it will result in bugs being found in userland applications that would have otherwise stayed unnoticed. So even if you don't use OpenBSD yourself this is helping your system becomming more secure and better. And if you are running OpenBSD there is o need to worry too much about the stability of this feature, it was actually enabled shortly after the 3.7 release and has been in every snapshot on the way to 3.8.

And I have to agree with the author that the best thing is that we get all the goods without ever having to switch them on!

Re:new method?

[zootm]

Unfortunately, this is a perspective that MS can't really afford to have, although they probably want to.

Re:new method?

[Taladar]

They created this user type so they have only themselves to blame.

Re:new method?

[Anonymous Coward]

Yes, they did create the user who isn't a nerd and who does not understand the workings of computer software and hardware enough to know who to blaim for problems. This is why Microsoft has a monopoly and generates billions in revenue.

Re:new method?

[zootm]

I must have forgotten that only experts were allowed to use computers, and making them accessible to people was a mistake. Sorry.

Re:new method?

[jandrese]

You're forgiven, but don't let it happen again.

Re:new method?

[ArbitraryConstant]

"You mean the Data Execute Protection from Microsoft? OpenBSD has had that for a long time already, only they named it w^x."

They also didn't need the per-page execute bit to do it. You need a fairly new machine to get the protection, but my 486 firewall has it. They also have stack protection, which is helpful because even if the heap and stack aren't executable you can overwrite return addresses or pointers to functions, and have them point to existing code that can be tricked into doing something malicious.

Re:new method?

[Anonymous Coward]

Like this bozo for example.

Re:new method?

[dougmc]

In the three cases we ran into when I was a consultant last year we placed the blame squarely on the shoulders of the vendor with the foobar product

Ok, but most Windows users don't have anything to do with consultants, and think that vendors are the guys who sell hotdogs (or the things you close to stop the wind from coming in, depending on where they live.)

Ultimately, think of Grandma. She fires up AOL, then her computer tells her that she needs SP2. She goes `ok' and many hours later, SP2 is done, and her {insert program useful to grandma here} application no longer works. She's not likely to blame this on the creator of the program -- she'll blame it on either Microsoft or AOL. (After all, grandma is no dummy. She knows that the program didn't change -- only Windows did. And perhaps AOL made it do it.)

It must really suck doing AOL tech support :)

Re:new method?

[afidel]

None of the applications which were actually broken by XP SP2 are something granny would run. On the other hand some of the applications that were affect were, but those were generally minor problems.

Re:new method?

[dougmc]

None of the applications which were actually broken by XP SP2 are something granny would run.

Have you even looked at your linked Microsoft KB items?

Your `broken' list is just a list of programs that need to have ports opened on the firewall. I wouldn't consider that broken, though it certainly could be seen as such by grandma. (And as for things that granny might run, I agree, she might not want to run Tivoli, but why couldn't she frag you in UT 2003?)

Your `affected' list is much more relevant, however, the `loss of functionalities' given includes such gems as `Program installs, but will not start.' I wouldn't call that a minor problem. (And Photoshop *is* something my grandmother has run, and that goes for a lot of other programs in the list.) In any event, about half of the problems are so serious that the program won't run or install at all.

Of course, what's my point? Not really that any of these programs are or aren't broken, or that Windows itself is broken, but ultimately grandma is going to blame the problem either on Microsoft or AOL, because the update was made to Windows and came over AOL.

Who will she call? Last I checked, Microsoft won't even talk to you on the phone without paying for a $75 or so support call, so that's out. The program vendor would probably help, and in fact they're probably the right person to call, but since it's not just one program, and it happened right after AOL installed something on her computer, she'll probably call AOL first ...

Re:new method?

[Fulcrum of Evil]

I must have forgotten that only experts were allowed to use computers, and making them accessible to people was a mistake. Sorry.

Only experts are trusted to find out why a system update broke their app and actually get it right. Sorry if this damages your precious ego.

Re:new method?

[zootm]

That's fair enough.

Re:new method?

[mgv]

ISTR that MS tried doing immediate free and it broke some programs that depend on the memory still being sround after being freed,

Sounds like those programs were broke from the start.

Yes, there were many broken programs that run under windows. I think this one was actually word perfect (or maybe lotus 123) and from memory, it simply didn't work under windows 95 due to this bug. Microsoft knew they would get the blame with the OS upgrade, even when it really wasn't their fault that a program released memory and still used it, which was possible under windows 3.11

(The exact details may be wrong but this really did happen).

In fact, M$ has a big problem of how to release a secure operating system. If they just start from scratch from say BSD, they lose compatibility. If they do this, they have to explain why you should use M$ BSD versus another version. Or why you wouldn't just port to apple, or linux.

Its the backwards compatibility that got them to be a market leader, and that is a two edged sword.

Even if you think that this lock in works for M$, it also causes them problems as if you are locked into a standard (even a proprietiary one), how can you be convinced to buy the next new upgrade? M$'s install base is now so large that they cant easily change a standard, even if they really want to. They might bring out a new version of office, but if it doesn't support the old formats, nobody will buy it. If it does support the old formats, nobody will use the new ones. Same for operating systems, if the new one isn't backward compatible, nobody will use it. If it is, everybody will write code for the compatibility mode.

So in summary, there are many things about M$ code that is borked. Some of it is actually deliberate, to compensate for what other software vendors do, and it may be a better solution to maintain backward compatibility than to junk the existing code base.

My 2c worth

Michael

Re:new method?

[innocent_white_lamb]

None of the applications which were actually broken by XP SP2 are something granny would run.
 
Don't bet on that. What about an obscure networked bridge game or something.

Re:new method?

[Shanep]

You mean the Data Execute Protection from Microsoft? OpenBSD has had that for a long time already, only they named it w^x.

I can't find it at the moment, but there was an interview with the head of Microsoft security (from memory), regarding security features in SP2. Interestingly, this fellow admitted to being a fan of OpenBSD. How interesting that one of the biggest security features of SP2 was PAE and the head security guy at MS is a fan of OpenBSD. Which had similar, yet more advanced (no NX bit requirement) functionality.

Was MS inspired by OpenBSD?

Re:new method?

[the morgawr]

Was MS inspired by OpenBSD?

For all we know they could be running OpenBSD code to do this.

Actually, now that I say it, that makes me feel BETTER. I'd rather them use the tested OBSD code as a start rather then try to redo it and screw it up...

Hope

[Anonymous Coward]

Let's hope it's not as broken as Microsoft's attempt in SP2.

But why did it take so long to implement?

Re:Hope

[slavemowgli]

Maybe the reason is precisely that they tried to make sure it's not as broken as Microsoft's attempt in SP2.

Re:Hope

[pla]

Let's hope it's not as broken as Microsoft's attempt in SP2.

In fairness, I don't know that we can really blame Microsoft for that one...

XP's memory protection works just fine (assuming the CPU supports NX pages) - The concept takes almost no thought to implement. The problem, however, arises from 99.999% of existing software not caring in the least about the separation of code and data. Usually that doesn't cause any problems, but when a "clever" (I put that in quotes as the bad-idea-of-the-day, but really, it does count as clever - Just not safe in a multitasking/multiuser/networked machine) program tries to execute something in its data, suddenly you have a problem.

As an aside, I have to wonder how functional languages such as Scheme or Tcl work at all with memory protection enabled. Not only do they tend to lack explicitly separate data and code, but they more-or-less depend on the ability to construct functions dynamically, as data, and then invoke them as code.

Re:Hope

[arkanes]

The NX flag is basically opt-in, you can request pages that aren't marked with it. In an OS that flags everthing as NX by default, you'd need to rebuild your application (although XP allows you do disable NX on a process by process basis). The point of NX isn't to prevent stuff like Scheme working, it's to prevent exploits or accidents.

Re:Hope

[Haeleth]

I have to wonder how functional languages such as Scheme or Tcl work at all with memory protection enabled. Not only do they tend to lack explicitly separate data and code, but they more-or-less depend on the ability to construct functions dynamically, as data, and then invoke them as code.

Simple - those are interpreted languages. The functions they construct dynamically and invoke as "code" are not actually machine code. The only actual machine code in the system is in the interpreter and language runtime, and those don't have to change. From the hardware's perspective, the "code" constructed and modified at runtime is quite literally just data.

For dynamically compiling runtimes that do need to construct and execute actual machine code at runtime, I belive many memory protection schemata include ways to bypass the protection. But that's not actually necessary just to use the language - it's an optimisation detail.

Re:Hope

[gmack]

Linux does it on a per application basis.. this is made easier by the fact that GCC has flagged binaries that don't need to have executable stacks (the vast majority of them) and has done that for longer than x86 has had NX so it's all pretty much automatic.

Re:Hope

[maita]

You're talking about two different things. Read a few comments above: #13704300, moron.

Re:Hope

[Krach42]

There have existed functions in Windows and Linux for a long time that mark a page as executable. Even though Linux or Windows never really enforced this to be set when executing data, it was required by specification.

Now that they're enforcing the specification, people complain that it broke.

Hey, PearPC was written before they enforced the specification, and Sebastian Biallas had the brilliant notion to actually follow the spec, and mark things as executable. Thus, when SP2 came out, PearPC worked fine.

Usually things break when moving to a newer version because some area of the spec wasn't very heavily stressed, and people writing code that just works (not as in, it works, but as in barely works,) thusly never really bothered shooting towards the spec. Then when the spec is enforced, they get all upiddy claiming that it breaks their app. Their app was broken to begin with, the previous implementations that you were relying on just didn't care.

For instance, when libc 5 (I think, don't hawk me about versions, if you know the correct versions, then please correct me, but I'm working off of a poor memory of the version numbers) came out, it enforced against passing a NULL file pointer. Before hand some people had hacked their code such that if an open failed, and returned a NULL file pointer, they didn't care or print an error message. They just kept going, since it would just waste CPU cycles, as nothing would get outputted or read from the file. It was silently gracefully failing for them, and they used that.

Then libc 5 comes out, and they break this silent graceful failure, and started reporting errors, or crashing when you passed them a NULL file pointer. People yelled and bitched, because they broke their app. But remember, THEIR APPS WERE BROKEN IN THE FIRST PLACE.

That's why I don't like people griping about "blah blah upgrade broke my app". Unless you can state that your app was built to spec from the beginning, then that upgrade didn't break your app. It was broken to begin with. The new upgrade just showed you how it was broken.

OpenBSD at the cutting edge on security

[Sv-Manowar]

Kudos to the OpenBSD folks for being at the cutting edge, in terms of implementations of these security features. Where they lead, surely others will follow and we'll be seeing this feature become commonplace. As their focus is security, its understandable that they lead more incentives in these areas than more mainstream Linux distributions.

Re:OpenBSD at the cutting edge on security

[Anonymous Coward]

I'm quite surprized to see Theo doing this.

"... in other words, it doesn't matter WHERE you shift the buggy code."

                          --Theo de Raadt

Won't this crutch actually tempt people to write sloppy memory management because "the heap manager will catch it?"

Is the performance hit really worth it?

Re:OpenBSD at the cutting edge on security

[fireboy1919]

than more mainstream Linux distributions

I know it seems strange...but OpenBSD isn't a Linux distribution at all.

I know its hard to wrap head around. Its one of those things you just have to accept. In addition:
-deep down, cows are not people too. So you can eat 'em, I guess.
-neither are cats or dogs. So don't force them to wear clothing.
-neither is information. So it doesn't care about being free or anything else.
-"Windows" is somehow both an operating system and a Window manager. You're not supposed to consider them separate things (wierd, isn't it?)
-Wearing a tampon with wings will not give you the power of flight.

Hopefully I've cleared up a few issues for you. :)

Re:OpenBSD at the cutting edge on security

['nother poster]

I read the grandparent post as saying that Linux was more mainstream than OpenBSD, not that OpenBSD was a version of linux.

Re:OpenBSD at the cutting edge on security

[richy freeway]

Wearing a tampon with wings will not give you the power of flight.

I'm no female, but a tampon with wings sounds mighty uncomfortable!

Re:OpenBSD at the cutting edge on security

[Anonymous Coward]

Depends on where they flap.

Re:OpenBSD at the cutting edge on security

[failure-man]

Won't this crutch actually tempt people to write sloppy memory management because "the heap manager will catch it?"
 
Doesn't seem so. The new malloc and mmap behavior will tend to cause buggy memory allocation code to segfault rather than allowing various sorts of stupiness or nastiness.

Re:OpenBSD at the cutting edge on security

[fireboy1919]

You can certainly read it that way if you like. You wouldn't be conveying what that sentence did, though.
As their focus is security, its understandable that they lead more incentives in these areas than more mainstream Linux distributions.

In this, the word "more" describes "Linux distributions." As it is a comparison, it also refers back to the subject of the phrase. The subject being "they," a pronoun which refers to "OpenBSD." This comparison also draws them into the same category because its comparing a single element to a collective.
To clarify, let me simplify this sentence, excluding all (mostly) superfluous modifiers:
"OpenBSD is more secure than more mainstream Linux distributions."

So...if open BSD is more secure than other, more mainstream Linux distributions, then it must be one of the lesser Linux distributions. Read more. You'll catch the subtle points more often.

Re:OpenBSD at the cutting edge on security

[dougmc]

Won't this crutch actually tempt people to write sloppy memory management because "the heap manager will catch it?"

Well, it'll catch it ... and the application will immediately crash.

I don't see how that encourages writing `sloppy memory management'. Nobody wants an application that crashes. (Granted, it's correct for an application to immediately exit when it's in danger of doing damage to something, but to an end user, they don't want their application to crash.)

Perhaps if somebody is explicitly writing their application to be secure, they could worry less about their memory management and assume that the OS will keep them honest (as it should) but ultimately, an application that crashes isn't very useful. And most end users don't care much about security anyways (though if they're using OpenBSD, odds are that they care about security more than most) and would much prefer something that doesn't crash.

Is the performance hit really worth it?

Theo seems to think so. If you disagree, either disable it (it sounds like it's at least somewhat configurable, and even if not, you do have the source), or use another OS.

Re:OpenBSD at the cutting edge on security

[ksheff]

Wearing a tampon with wings will not give you the power of flight.

Thank you. I needed that laugh on a Monday morning like this.

Re:OpenBSD at the cutting edge on security

[LurkerXXX]

You mean it will encourage sloppy coding more than the current system?

"hey, it might do something strange every once in a while, but at least it keeps running and doesn't crash, so the code is 'good-enough'".

I think Theo is doing entirely the right thing by killing badly written apps rather than letting them do bad things to the system. It's much more likely to make people fix bad code than the current system.

Re:OpenBSD at the cutting edge on security

[Anonymous Coward]

Ok. You read it your way, I'll read it mine. I read "more mainstream" as the modifier of "Linux distributions" implying that Linux is more mainstream than Open BSD, not "more" as a modifier of "mainstream Linux distributions" implying OpenBSD is Linux.

Re:OpenBSD at the cutting edge on security

[brunson]

I disagree. "more" is an adverb modifying "mainstream", which in turn modifies Linux. Try subsituting the context and it makes total sense: "The Prius is less polluting than more mainstream SUVs"; i.e. the Prius is less poluting than SUVs, which are more mainstream.

I'm not saying the sentence couldn't benefit from a little better punctuation or a slight restructuring, but I don't agree that it can only be interpretted the way you choose to.

Re:OpenBSD at the cutting edge on security

[Anonymous Coward]

I think you are correct about the original author's intent. In that case, his logic was not wrong, just the statement itself. Imagine calling Linux "more mainstream" than BSD Unix! What a moron.

Re:OpenBSD at the cutting edge on security

['nother poster]

Even in your rewriting of the text from the great grandparent post, more mainstream is still modifying Linux distributions. It's not until you re-rewrite the sentence and insert the word "other", that does not exist in the other iterations of the sentence, that it reads the way that you imagine it. Hope that helps.

P.S. Just shy of 2000 books on the shelves in the basement. Of those about 1600 have been read, 300 abandoned because the style/plot/storyline sucked too bad, and 100 await reading. Also I've been diagramming sentences with my 4th grade son.

Re:OpenBSD at the cutting edge on security

[ScrewMaster]

Regardless of all the grammatical nitpicking going on (and I read it the same way you did) ... your post was hilarious.

Re:OpenBSD at the cutting edge on security

[Anonymous Coward]

Kudos to the OpenBSD folks for being at the cutting edge, in terms of implementations of these security features.

If you had read the article, you'd have known that OpenBSD isn't the first to implement these features.

Re:OpenBSD at the cutting edge on security

[trewornan]

You should try Analysing Sentences, it's a great book and starts from the absolute basics.

Re:OpenBSD at the cutting edge on security

[lachlan76]

How about we forget about all this, and just modify the language to have a paranthesis-like structure? ;)

The Prius is less polluting than ((more mainstream) 4WDs).

As their focus is security, its understandable that they lead more incentives in these areas than (more mainstream) Linux distributions.

as opposed to

As their focus is security, its understandable that they lead more incentives in these areas than (more mainstream Linux distributions).

My solution is slower, but 100% effective

[Anonymous Coward]

When my application needs a chunk of memory, it sends a specially crafted HTTPS request to my bank, debits the account, sends a fax to the local computer shop who then sends a tech over to install the DIMMs.

When the application is finished with the memory, it sends a FAX to the local electronics recycling facility who sends out a tech to remove the DIMMs and melt them down into whatever.

Using this method of heap memory allocation (I call it "ACAlloc" for "Anonymous Coward Alloc" has been 100% effective and I have NEVER had a heap overflow exploit in any of my code.

Yes, it's slow, but I am secure.

...And I'm running the most up-to-date 80386 Linux 0.97 kernel. TDz.

Re:My solution is slower, but 100% effective

[Chemisor]

And exactly how much memory does your mighty 80386 support?

Re:My solution is slower, but 100% effective

[Anonymous Coward]

Your solution might be appealing to someone in Redmond, WA!

Re:My solution is slower, but 100% effective

[OldManAndTheC]

remove the DIMMs and melt them down into whatever.

Wow - when you do garbage collection, you don't fool around!

What's next?

[Groo Wanderer]

Ok, we start out with 'protection', then we move to 'a heap' of protection, most assuredly to be followed by 'a whole heap' of protection. I can only see this spiral continuing until Bill Gates himself gets up on stage at CES in an Elvis suit promising 'a hunka- hunka- burnin protection'. *SHUDDER* Time to take a cold shower.

              -Charlie

Re:What's next?

[OverlordQ]

No no, you have it wrong. Gates will sit there and glare, while Balmer flies around like a insane monkey on crack.

Re:What's next?

[Rufus88]

Bill Gates himself gets up on stage at CES in an Elvis suit promising 'a hunka- hunka- burnin protection'. *SHUDDER* Time to take a cold shower.

You *need* a cold shower? Hell, to me, that image *was* the equivalent of a cold shower!

Re:What's next?

[Anonymous Coward]

And you forget the native American release, which will undoubtedly have "heap big" protection.

And if you think I'm going to apologize for that you little know your man.

Re:What's next?

[Fulcrum of Evil]

I can only see this spiral continuing until Bill Gates himself gets up on stage at CES in an Elvis suit promising 'a hunka- hunka- burnin protection'.

No way could BG pull off Elvis. Have you seen his hair?!

Sacrfice useability? Nice idea , won't work.

[Viol8]

"And if we have to sacrifice a little usability on the way there, then so be it."

Nice dream , meanwhile in the real world both users and most coders
(if they dare to admit it) will NOT sacrfice usability or ease of
coding for security measures that (in their minds) are nothing to
do with their application. Unless that is they're forced to either
by company policy or restrictions in the OS. If there are restrictions
in the OS then IT tech leads might start to ask "well, I can do this in
OS A , why won't OS B let me do it? I'll use OS A". Not an ideal situation
but when all that matters is delivery times and saving $$$ then its what
will happen. Security , as ever , will come a poor second.

Apologies to the Black-Eyed Peas

[Anonymous Coward]

OpenBSD's "My Heap":

Lookin' at my heap, heap
You can look but you can't touch it.
If you touch it, I'ma start some drama.
You don't want no drama.
[...]
My heap, my heap, my heap, my heap.

Re:Apologies to the Black-Eyed Peas

[mrtroy]

Immma get get get you drunk Get you love drunk off my heap Immma make make make you scream Make you scream make you scream! [...] My heap, my heap my heap, my heap My lovely lady lumps

Re:Apologies to the Black-Eyed Peas

[Anonymous Coward]

Huh? Should we know that song? Black Eyed Peas suck.

Hm... old technique?

[archeopterix]

Ok, the article is light on technical details, but it seems that they are using guard pages. Guard pages aren't exactly shiny new. Efence has been using them since a long long time.

Re:Hm... old technique?

[Chocolate Teapot]

Guard pages aren't exactly shiny new

Shhh!! I was waiting until everyone started using them before hitting them with my patent ;)

Re:Hm... old technique?

[qwertphobia]

Ok, the article is light on technical details, but it seems that they are using guard pigs. Guard pigs aren't exactly shiny new. The Battalion has been using them since a long long time.

Re:Hm... old technique?

[Dausha]

"Shhh!! I was waiting until everyone started using them before hitting them with my patent"

I know you're being cheeky, and I see the humor. Although, this tactic can cause one's patent to be invalidated. I forget the basic rule.

Re:Hm... old technique?

[Anonymous Coward]

Guard pages aren't exactly shiny new.

Like a guard page,
touched for the very first time.
Like a gua-a-a-a-ard page,
With your safe heap,
next to mine.

Hmmm, so it was only me that started singing that on reading your post?

Re:Hm... old technique?

[Anonymous Coward]

Old technique ? yeah sure. but all throughout the system, and no way to cop out of it.

THIS is what makes the difference.

Why do you think the new malloc was two years in coming ? it was because there were numerous issues to fix all over the system before you could activate it...

Microsoft Windows?

[fernique]

Could this technology be implemented in the Microsoft Windows systems to be more secure than Linux?

Re:Microsoft Windows?

[Slashcrap]

Could this technology be implemented in the Microsoft Windows systems to be more secure than Linux?

It certainly could be implemented, but it would have the slight drawback that a huge proportion of apps would stop working.

It's going to break a lot of stuff on OpenBSD as well, but because of their audience they can get away with e.g telling you not to run Apache because it's insecure. Also, the OSS apps that break because they assume a specific memory management model will get fixed. No-one is going to be able to fix Windows apps except the developers. And if they even bother they will just expect you to buy the new version.

And you seem to be saying that implementing this one feature on Windows will make it more secure than $OtherOS. I initially thought you might be trolling, but I've decided to give you the benefit of the doubt and just assume that you either know nothing about security or are making a lame attempt at humour.

Re:Microsoft Windows?

[Anonymous Coward]

Except for that whole "it already has been implemented" issue. Does it help? Yep. Does it solve the problem? Nope. Too many stupid users and unpatched applications out there to really fix the problem in the short term. But it moves Windows towards being a less brain-dead implementation.

Re:Microsoft Windows?

[KillShill]

yes, yes it could.

and secure windows would have the benefit of not using up any electricity, since the computer wouldn't be turned on.

so it's a win-win situation (pun intended).

My Windows XP has heap protection!

[callipygian-showsyst]

Microsoft released this last year. Read the details here

As BSD is my other favorite OS (I tend to use XP for desktops and FreeBSD for servers), I'm glad to know that they've finally caught up with Microsoft in this important area.

Re:My Windows XP has heap protection!

[jcupitt65]

The MS thing is just support for no-execute: the bit that says that this page is only code and not data and you shouldn't try to run anything in here. Everyone has supported this for ages.

This is more. It looks like they are adding extra 'tripwire' pages to the heap, so if an attacker manages to write to part of the heap they shouldn't, there's a good chance they'll hit a tripwire and be detected.

Re:My Windows XP has heap protection!

[zootm]

I think they did add heap protection at the same time, although I hear it's been "broken" (although in practical terms I've no idea).

Re:My Windows XP has heap protection!

[Triumph The Insult C]

this support showed up in openbsd almost 2 years ago

microsoft's support was partial. intel's implementation is partial

openbsd's support is full. amd's implementation is full (and freely documented)

Re:My Windows XP has heap protection!

[Mr. Moose]

...that says that this page is only code and not data and you shouldn't try to run anything in here

The "only code, not data. DO NOT RUN THIS" flag could be the reason why my Windows box stops working sometimes... Hope the BSD implementation is a bit better.

Re:My Windows XP has heap protection!

[xgamer04]

I can see it now...

  Haha, I'm totally gonna piss all over the heap. This box is goin' DOWN!

BZZZZZT! Heap protection initiated

  WTF??
  HAHA PWNED

Re:My Windows XP has heap protection!

[Anonymous Coward]

The MS thing is just support for no-execute: the bit that says that this page is only code and not data and you shouldn't try to run anything in here. Everyone has supported this for ages.

The no-execute bit says that the page is not code and should not be executed. Such things are nice to have on a stack, for example. The 80386 introduced a hardware mechanism for supporting this feature but it had not garnered widespread support until more recently.

Re:My Windows XP has heap protection!

[vulcan_pupil]

M$ added more than just NX support. SP2 added a heap protection mechanism, but as some have already pointed out, it can still be exploited.

Re:My Windows XP has heap protection!

[Anonymous Coward]

Why do comments like this get moderated as "interesting"?

the comment implies that OpenBSD hasn't had non-executable flags per pages which is absolutly wrong because Microsoft took that idea from OpenBSD (and so did PAX). In fact, OpenBSD might have been the first.

I think they should change "interesting" to "uninformed"

Re:My Windows XP has heap protection!

[Anonymous Coward]

No. The 386 doesn't have the NX bit. The 386 page table entries have flags for read and write, user and system, but it doesn't have an execute bit.

x86-64 has an NX bit. 32-bit x86 does not.

Now, OpenBSD has some hack to get around it. Maybe they implement it in the page fault handler, i.e. do NX in software; I don't know.

Re:My Windows XP has heap protection!

[callipygian-showsyst]

Actually, they both were probably influenced by Intel which has had hardware support for this (in some form or another) since the 80386.

Re:My Windows XP has heap protection!

[Anonymous Coward]

AMDs is far from full.

Re:My Windows XP has heap protection!

[Jerry Coffin]

Microsoft released this last year. Read the details here

As has already been pointed out, what Microsoft released last year wasn't really quite the same (though it was certainly related.

Interestingly enough, however, Microsoft released something that was probably even more effective long before that. OS/2 1.0 (way back in 1987) used the Intel segmentation model instead of page-based memory protection. One result of this was that when you allocated a chunk of memory, the protection on that chunk could be set to exactly the size you requested. Attempting to write even one byte beyond the amount allocated would result in an immediate trap. Interestingly enough, segments also had separate bits for execute vs. read from day one, so it was trivial to say something could be read by not executed. The NX bit just added that ability to page-based protection.

It looks like OpenBSD is also trying to make the heap use address space relatively sparsely -- i.e. attempting to assure that using a more or less random address is likely to result in a trap. Again, segment-based protection went a bit further: each segment required you to load a specific value in a segment register. With segment registers, you not only got a trap if you tried to use an illegal address, but you got a trap as soon as you even tried to form the address by loading an illegal value into the segment register.

The shortcoming is that segment registers are only 16 bits, and the required privilege level is encoded into a couple of those, reducing the address space still further. The bottom line is that if you catch the interrupt from the illegal access, it's pretty trivial to step through all possible segment register values, and find which ones are legal and which aren't.

The OpenBSD approach should gain a major advantage on a 64-bit processor -- a given amount of allocated memory will use the larger address space much more sparsely, so a randomly chosen address is much less likely to be valid.

As BSD is my other favorite OS (I tend to use XP for desktops and FreeBSD for servers), I'm glad to know that they've finally caught up with Microsoft in this important area.

This is in OpenBSD, not FreeBSD. Assuming it works out well, it might someday migrate to other OSes, and FreeBSD would probably be the first/easiest target for migration, but that's the future. OTOH, unless my memory has gotten even worse than usual, FreeBSD, NetBSD and Linux all do roughly the same sort of thing as XP SP2.

--
The universe is a figment of its own imagination.

Hm... gotta reply to myself

[archeopterix]

Ok, I've posted hastily, thus creating a bit of an half-assed post. They use more techniques (random address allocation, immediate free-to-kernel), still not revolutionary, but indeed worth mentioning. My bad.

Could this help Gnome?

[MuMart]

Sounds like a heap that returns unused pages to the system like this would help the problem described by John Moser in the Gnome Memory Reduction Project here.

Is it really true that the standard GNU/Linux heap implementation holds onto pages like this when it becomes fragmented? That sounds really primitive to me.

Re:Could this help Gnome?

[Ulrich Hobelmann]

I don't know if GNU malloc uses mmap() or brk() for its allocation, but in both cases small memory chunk that the user allocates are taken from bigger, contiguous blocks of memory.

Maybe that's primitive, but it ensures that usually small memory requests are fast, and don't have to much space overhead, either.

If the memory de/allocation patterns are really bad, though, you get fragmentation with the mentioned problems. It's a tradeoff. Note that OpenBSD didn't choose its way for reducing heap usage (maybe they use even more memory, due to overheads), but for security reasons.

There would be one solution, and that's using different arenas, or memory regions for allocation. For instance every window might have its own allocation region, so when you close the window/document, the memory BLOCK is freed. No fragmentation, almost no overhead. I really wish Java apps, Cocoa apps, and other (Mozilla) would do this, as they seem to suffer from this fragmentation problem, only increasing their memory usage, even after closing all documents/windows etc.

Anyway, I love the new malloc, and kudos to the whole OBSD team!

Re:Could this help Gnome?

[joib]

I don't know if GNU malloc uses mmap() or brk() for its allocation, but in both cases small memory chunk that the user allocates are taken from bigger, contiguous blocks of memory.


It uses mmap() for big allocs (IIRC the threshold is 4 MB) and brk() for smaller ones.


There would be one solution, and that's using different arenas, or memory regions for allocation. For instance every window might have its own allocation region, so when you close the window/document, the memory BLOCK is freed.


Something like memory pools, used e.g. by apache portable runtime, gcc, and whatnot. Or the hierarchical allocator used by samba..


I really wish Java apps, Cocoa apps, and other (Mozilla) would do this, as they seem to suffer from this fragmentation problem, only increasing their memory usage, even after closing all documents/windows etc.


Actually, Java being garbage collected and not having "raw" pointers can avoid fragmentation by repacking heap memory. Of course the JVM must still use malloc() or mmap()/brk() directly to get the memory from the OS.

In related news, GCC 4.1 stack protector

[joib]

The upcoming GCC 4.1 release will include a stack protector. Basically it's a reimplementation of the old propolice patch.

Hopefully mainstream distros that have been wary of propolice will start using this new feature. And perhaps glibc malloc will borrow a few tricks from this new openbsd malloc too.

Re:In related news, GCC 4.1 stack protector

[rehabdoll]

There are patches available at http://www.trl.ibm.com/projects/security/ssp/ for 3.4.4 and 2.95.3

Re:In related news, GCC 4.1 stack protector

[Triumph The Insult C]

And perhaps glibc malloc will borrow a few tricks from this new openbsd malloc too.

maybe glibc developers should start small and put strl*() and friends in first ...

Re:In related news, GCC 4.1 stack protector

[stevey]

And binary packages for Debian's Sarge release on x86 are available here.

Re:In related news, GCC 4.1 stack protector

[Anonymous Coward]

Also in related news, OpenBSD has used ProPolice since version 3.4, relased 1 November 2003.

Is there any question that it is ahead of the curve?

Re:In related news, GCC 4.1 stack protector

[Intron]

Although strlcpy and strlcat attempt to fix problems with buffer overflows due to lack of null termination on strn* results, it is interesting to note that they still have a problem if you pass a string argument that is not null-terminated, since they return the total length. This problem could be detected by the guard page on malloc.

Re:In related news, GCC 4.1 stack protector

[Intron]

There's an inconsistency in strl* definitions, too. The size parameter DOES include space for the null, but the returned length DOES NOT include the null. So much for fixing the inconsistencies in strn*.

Re:In related news, GCC 4.1 stack protector

[Anonymous Coward]

Well sure, but that's a case of invalid input. It can also have problems if you pass it an incorrect buffer size.

Or, anything related to the reading of strings has this problem. Nobody complains about the rampant instability of strlen(), but if you supply it an unterminated buffer, of course it's likely to crash, and if it doesn't, you end up with an incorrect result. Still, this is so obvious as to not need any discussion.

Whats so new here?

[Anonymous Coward]

"innovation in Unix that talks about OpenBSD's new heap protection mechanism as a major boon for security.", surely you mean "innovation in *BSD " ?

When taking a look at some of the features of my favorite Unix environment you'll notice that among the security features its also possible to shield processes from each other and control the amount of memory they use (near the bottom, the so called "buffer overflow attack"). If you wish to study this in more detail you can check out the documentation on resource control which explains you the issue in more detail. For the impatient; controlling physical memory and such is explained in chapter 10

Sorry but I can't conclude that this is something new in the world of Unix.

call bob, bob.

[Anonymous Coward]

"well, I can do this in OS A , why won't OS B let me do it? I'll use OS A"

... and call OS A Linux and OS B OpenBSD ...

I hope i dont ever find out how to get memory usage on openbsd, sure as hell ain't "free"...

Already in Microsoft DEP

[Henry V .009]

DEP from Microsoft is only enabled by default on some system binaries. Here is how to enable it for everything: http://www.microsoft.com/technet/security/prodtech /windowsxp/depcnfxp.mspx

My CPU doesn't support DEP in hardware, so I imagine the software-based method of doing this will create quite a speed hit. Anybody have any experience with turning on DEP for all programs?

Re:Already in Microsoft DEP

[Henry V .009]

Well, I've turned it on now. No noticeable performance hit, and no mysterious application failures. Just compiled a Visual C++ program and it ran fine. Same for G++ on CYGWIN.

Also, I have not been hacked anytime in the last ~5 minutes, whatever that's worth (but would I know?).

As an aside, I read the paper on the Microsoft DEP flaw a few months ago, and wasn't that impressed. It looks very hard to exploit. And since DEP is a added protection mechanism, the existence of a small, hard-to-expliot flaw isn't that big of a deal. (In simple terms, with DEP on, a hacker would have to exploit and DEP flaw and a normal overrun flaw to hack the system.)

Re:Already in Microsoft DEP

[Anonymous Coward]

The NX bit (DEP) isn't the same as heap protection.

Intron == heap protection

[hey]

I don't know if it was on purpose but your sig -- which mentions Intron (aka Junk DNA) is very apt.
OSes can put Intron between Exon (useful DNA -- useful stuff on the heap) to
detect badly behaving apps!

In other words, so-called "Junk DNA" may actually have a use...
HEAP PROTECTION ;-)

Re:Intron == heap protection

[SilverspurG]

In some cases introns serve a known purpose in indexing for the enzymes which work with DNA. Junk DNA is the stuff with no known purpose at all.

You're still correct. It is heap protection in an evolutionary way. Heap protection on computers seeks to safeguard the data as it is. Junk DNA accepts that things are going to get corrupted and seeks to make it statistically less likely for the important parts to get corrupted.

Thanks for that train of thought. :)

Re:Intron == heap protection

[Jah-Wren Ryel]

You're still correct. It is heap protection in an evolutionary way. Heap protection on computers seeks to safeguard the data as it is. Junk DNA accepts that things are going to get corrupted and seeks to make it statistically less likely for the important parts to get corrupted.

Just being a "dummy target" seems like an inefficient use for extra DNA. I would have expected something along the lines of ECC like reed-solomon coding to have evolved. Kinda shoots down the "intelligent design" theory too, unless you can accept that God is an amateur at this stuff.

Re:Intron == heap protection

[beanlover]

Kinda shoots down the "intelligent design" theory too, unless you can accept that God is an amateur at this stuff.

Unless God put that stuff there to keep you guessing. :)

Re:Intron == heap protection

[Anonymous Coward]

Circular Argument, (n)
See Circular Argument.

Re:Intron == heap protection

[Anonymous Coward]

Just being a "dummy target" seems like an inefficient use for extra DNA. I would have expected something along the lines of ECC like reed-solomon coding to have evolved.

http://ds9a.nl/amazing-dna/#rss

Re:Intron == heap protection

[Anonymous Coward]

Just being a "dummy target" seems like an inefficient use for extra DNA. I would have expected something along the lines of ECC like reed-solomon coding to have evolved. Kinda shoots down the "intelligent design" theory too, unless you can accept that God is an amateur at this stuff.

Dummy DNA is one of many mechanisms that function as the cellular security model. Don't get all high and mighty on yourself. Dummy DNA in no way shoots down intelligent design.

Re:Intron == heap protection

[Anonymous Coward]

To an ID believer, nothing shoots down ID -- ID is a matter of *faith*, and is not meant to be a testable theory.

Totally wrong

[Anonymous Coward]

What they restrict here, is extremely shitty programming. What's the point of allowing accessing memory areas out-of-bounds? Fewer crashes, because the system does not notice them? This is a very bad idea. If you don't know how to handle arrays and you don't know any other means to restrict yourself from doing something stupid, OpenBSD shows the best way to do it (as last instance).

Wrong solution for solving heap problems.

[master_p]

From the kerneltrap.org post:

He explains that for over a decade efforts have been made to find and fix buffer overflows, and more recently bugs have been found in which software is reading before the start of a buffer, or beyond the end of the buffer.

The solution that the kerneltrap.org refers to against buffer overflows is to:

1. unmap memory as soon as it is freed, so as that to cause a SIGSEV when illegaly accessed.
2. have some free 'guard' space between allocated blocks.

My opinion is that #1 will slow software down, although it will make it indeed more secure. #2 will make it more difficult to exploit buffer overflows, since the space between two allocated heap blocks will be random (and thus the attacker may not know where to overwrite data).

Unless I haven't understood well, these solutions will not offer any real solution to the buffer overflow problem. For example, stack-based exploits can still be used for attacks. The solution shown does not mention usage of the NX bit (which is i86 specific). It is a purely software solution that can be applied to all BSD-supported architectures.

Since all the problems relating to buffers (overflow and underflow) that have costed billions of dollars to the IT industly is the result of using C, doesn't anyone think that it is time to stop using C? there are C-compatible languages that allow bit manipulation but don't allow buffer overflows; e.g. Cyclone.

Re:Wrong solution for solving heap problems.

[cerelib]

Amen to that last comment. The widespread use of C is probably one of the biggest contributors to unstable and unsecure systems. I am not saying that C does not have its uses, but some things should be reconsidered.

Re:Wrong solution for solving heap problems.

[Anonymous Coward]

1. OpenBSD already makes use of NX-bit protection (they call it W^X).
2. OpenBSD previously implemented stack protection (propolice + other stuff).
3. OpenBSD even implements W^X on non-NX-bit i386 through other techniques.

Re:Wrong solution for solving heap problems.

[Anonymous Coward]

Furthermore, you can unmap only allocations greater than a page size. For objects less than page size (which is the majority of the allocations) it still allows dangling accesses after a free.

Even for objects greater than page size, what happens if you run out of the address space? Correct me if I am wrong but, you will have to reuse the previously unmapped pages ... so its a not a fool proof solution.
atlease for dangling references.

Similarly if your buffer overflow is not off by one but by a page size, then a guard page will not help.

Nevertheless its a good step forward.

Re:Wrong solution for solving heap problems.

[Homology]

1. OpenBSD already makes use of NX-bit protection (they call it W^X).
2. OpenBSD previously implemented stack protection (propolice + other stuff).
3. OpenBSD even implements W^X on non-NX-bit i386 through other techniques.

The last point is important since most i386 machines don't have the NX-bit, and that will be the case for many years to come.

Re:Wrong solution for solving heap problems.

[Homology]

Indeed, its a good step forward. Quite a few bugs in aplications in the OpenBSD ports tree was fixed because of this. A benefit for non-OpenBSD users as well.

Re:Wrong solution for solving heap problems.

[Anonymous Coward]

OpenBSD already has had W^X for at least 2 years. Further, sure, returning memory to the kernel might slightly slow things down, but you'd also end up with lighter memory consumption, and thus less swapping. For example someone in another thread above has pointed out that GNOME is trying to reimplement malloc with mmap, too, for this reason.

Besides... This solution is much faster than the current ones that people use to get the same result. Look at valgrind. That emulates a 386 just to trap these sorts of problems! Why do that, when you can have the MMU do most of the work? This makes malloc debugging faster than, well, using a malloc debugger.

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

C is actually the most secure language currently, AFAICT. Languages with higher level intrinsics (C++, Java, Basic, Mono, Objective-C, etc) have a more complex implementation that may allow different exploit vectors; while languages with real-time interpretation or runtime code generation (Java or Mono with JIT) will wind up disabling things like strong memory protection policies (strict Data/Code separation -- code is data when generated at runtime) and may not in their own code create a backup buffer overflow protection.

In the event of a screw-up on the part of the JIT or runtime programmer for any language, every program is instantly vulnerable, and all of this generic proactive security stuff is disabled because this "secure language" doesn't work in an "inherantly secure" environment, only a much weakened one. C's runtime is rather basic (and it's still huge), as is its language; people still screw that up once in a while, but rarely.

While these "shiney new secure languages" may boast "immunity to buffer overflows," their runtimes are still designed around other concepts that may leave holes. Look at this memory allocator and think about a bug in the allocator that smashes up its own memory before it gets everything set up; because the new protections aren't yet set in place, it'd be totally vulnerable at that point (no practical exploit of course). A bug that forgets to add guard pages (generates 0 guard pages every time) might occur too in one update. Now add to that something like Java or Mono-- interpreted or not, you're running on a whole -platform- instead of just a runtime. C++ instruments loads of back-end object orientation.

So in short, C is a very basic language that has easily quantifiable attack vectors, and thus the system can be targeted around these for security. Several such enhancements exist, see PaX, GrSecurity, W^X, security in heap allocators, SELinux, Exec Shield, ProPolice. Higher level languages like C++ implement back-end instrumentation that ramps up complexity and may open new, unprotected attack vectors that are harder to quantify. Very high end languages on their own platform, like Java and Mono, not only implement massive complexity, but rely on a back-end that may lose its security due to bugs. Platform languages may also be interpreted or runtime generated, in which case they may require certain protections like PaX' strict NX policy to vanish; in some cases these models (as an implementation flaw) also don't work well with strict mandatory access control policies under systems like SELinux.

Face it. C is the best language all around for speed, security, portability, and maintainability. Assembly only brings excessive speed at the cost of all else; and higher level languages sacrifice both speed and real security (despite their handwaving claims of built-in security) at varying degrees for portability, speed of coding, and maintainability. Even script languages working inside a real tightly secured system would more easily fall victim to cross-site scripting, the injection of script into the interpretation line; under such a system, any similar attack is impossible in a C program.

On a side note, I'd love to see a RAD for C. Think Visual Basic 6.0, but open source, using C/GTK+. Glade comes close. . . .

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

Face it. C is the best language all around for speed, security, portability, and maintainability. Assembly only brings excessive speed at the cost of all else; and higher level languages sacrifice both speed and real security (despite their handwaving claims of built-in security) at varying degrees for readability , speed of coding, and maintainability. Even script languages working inside a real tightly secured system would more easily fall victim to cross-site scripting, the injection of script into the interpretation line; under such a system, any similar attack is impossible in a C program.

Higher level languages aren't typically more or less portable. . . . (even Java half the time needs modifications to run on various platforms; look at Limewire or anything else big and java)

Re:Wrong solution for solving heap problems.

[csirac]

The solution shown does not mention usage of the NX bit (which is i86 specific).

Actually, x86 is one of the last into town with "NX bit" functionality. POWER (and PPC I guess), PA-RISC, Sparc, Alpha, etc. on the big-iron has had this feature as a standard part of their architecture (along with the OSes that run on them) for bloody ages now... on those CPUs, even Linux has had hardware support since before whenever x86 got NX support.

http://en.wikipedia.org/wiki/NX_bit

Although this can stop execution of arbitrary code in areas of memory that aren't meant for it, you're still going to be vulnerable to corruption of memory and make the software behave in unintended ways. The problem is that the Operating System can't tell the difference between a memory pointer that has stepped outside the malloc'd space it was intended to address and into an adjacent one, or two legitimate pointers behaving nicely.

Adding the guard space will definately help, as will unmapping memory. OpenBSD hasn't exactly been known for its high performance; people use it for security and peace of mind. Additionally, they made mmap() return random addresses, which is very cool indeed.

doesn't anyone think that it is time to stop using C?
C is not used for building web apps, so what are you talking about?

Everybody knows C is the right tool for some jobs, Java/Perl/Ruby/etc for others.

In the right hands C can be just as secure and has the potential to be much faster than most of the alternatives.

Re:Wrong solution for solving heap problems.

[pammon]

Welding without goggles is actually the safest way to weld, AFAICT. Welding goggles open up additional hazard vectors - for example, you might choke on them.

While these "shiny new goggles" may boast "immunity to getting molten metal in your eyes," their approach is still designed around other concepts that may leave holes. Look at their strap and think about it wrapping around your neck. Or maybe you'll have an allergic reaction to the plastic.

So in short, welding without goggles has easily quantifiable hazard vectors, and thus the system can be targeted around those for security. Several such enhancements exist, see "Welding Without Goggles Using Goggles," or "Blindfolded Welding."

Face it. Welding without goggles is the best approach all around for speed, join strength, and safety. Even velcro would more easily fall victim to a "pull apart" type attack; any similar attack is impossible if you weld.

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

Actually, the correct analogy would be:

Permenantly dark goggles -- Java, etc

Goggles that tint when bright light happens -- C

The argument "C is insecure, we should use [insert odd language here]" is like saying, "Goggles that auto-tint are bad because they're not dark all the time, and you could see a bright flash."

The argument I made would be more like, "The auto-tint goggles are better because they take the basic need -- you have to see -- and allow that, but still protect you from the blinding flash of welding light when needed by cutting your vision off at that moment, a necessary evil some of the time."

I suggest you don't make retorts to peoples' comments unless you actually know what you're talking about.

Re:Wrong solution for solving heap problems.

[barryman_5000]

I hope you are not talking about swt or functions outside of java like windows api. You can't simply expect java to do everything under the sun(pun). On note to the grandparent, C is only more secure in that a very adept programmer can control any aspect of the program he wants to. Even still you rely on libraries many times creating even a simple program. I would bet that a java program would be more secure than a C program in that fact that C just leaves you out there. Java has security features built in whether or not you use them.

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

Another interesting thing I should point out is that some of the proactive measures that prevent attacks that may still apply to these other languages are in some cases not totally effective anymore, or must be disabled in cases. Heap corruption protections could for example be less effective in C++ due to high level class control; while Java may need data-code separation disabled. So in that sense your argument would be more fit to say that Java is like taking the goggles off, because we're using a slightly lower power welder that doesn't create as bright a flash; the goggles have to come off because the flash isn't bright enough to let you see what you're doing with the goggles on.

Re:Wrong solution for solving heap problems.

[pammon]

Actually, the correct analogy would be:

Permenantly dark goggles -- Java, etc

Goggles that tint when bright light happens -- C

So there you're arguing that C has fewer limitations than, say, Java? Sure, but I wasn't arguing that; I was addressing your claim that "C is actually the most secure language currently."

The argument "C is insecure, we should use [insert odd language here]" is like saying, "Goggles that auto-tint are bad because they're not dark all the time, and you could see a bright flash."

This seems to be a separate point from the one you made previously. You argued that C is the most secure language because it has the slimmest runtime and so the fewest things that can go wrong. I satirized that point. In places that demand C's flexibility, by all means, use C.

I suggest you don't make retorts to peoples' comments unless you actually know what you're talking about

Gladly, but you "what you're talking about" seems to have changed. Beam in your eye first.

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

The added complexity for Java I was talking about was the java platform-- Java programs don't see the whole system, they see a platform. This includes at least a garbage collector and a high-level object infrastructure (like C++ has); but can in many systems go into a runtime interpretor or Just-in-Time compiler.

As for inherant security, compiler and OS mods make C quite secure even when you screw up (ProPolice, PaX, GrSecurity, PIE, etc); Java's platform (and a few others) may forbid these systems or mitigate their usefulness. For example, Just-in-Time compiled code might not be position independent, so it may be somewhere predictable; and it probably won't be stack smash protected, as we believe that the stack cannot be smashed. An error in the Java platform's bounds checking code might actually allow a stack smash; the difference between this and a C stack smash protector is of course that the Java one will probably resize the buffer (on the heap) or just warn you, while in C the program will crash and the programmer is -forced- -to- -fix- -it-.

On my end, I prefer a language that takes full advantage of the OS security. Script languages open their own holes; look at Hardened PHP and its strict handling of include() to try feebily to avoid XSS, something that occurs because it's a script language and the OS can't prevent runtime script injection (just runtime native code injection). The thing also brings along some heap protections for PHP's special memory manager, which is similar to Java, C, Mono, and Python having their own memory managers operating overtop of or in place of the system's malloc() implementation. But at least in these cases, I can say that every precaution taken is still in effect. Without runtime code generation, for example, a full memory protection policy and stack smash protector still afford their protection; although the heap protections from glibc don't protect the higher level allocators (weakened), and interpreted languages can find XSS bugs (weakened).

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

You argued that C is the most secure language because it has the slimmest runtime and so the fewest things that can go wrong. I satirized that point.

What you're responding to is C as a language prone to buffer overflows, where runtime code execution is possible and thus code injection (the binary equivalent of cross site scripting) can happen.

What I was talking about as C as a language where such things are handled by a compiler that sets up guard objects that make the program abort on stack buffer overflows; where heap allocators actually protect against heap smashes; where an operating system level data/code separation prevents anything from being write+execute at the same time, or moving from !execute (data) to execute (code); and where such an operating system allows the administrator to reduce privileges to the needed set and proactively prevent things like temporary file race attacks and named pipe spoofing/leaking.

The second such system is marginally more processor intensive than the first, and may also use some VM space in the case of heap protections. In my experience, the full force of hardened systems is a raise of CPU usage by approximately 1% (give or take 0.25%), discarding mandatory access control systems; that is, if you use 1% of the CPU normally then you'll use 1.01%, and if you'd use 90% then you'd use 90.9%. Full CPU utilization is at what is typically 98% now. SELinux, however, has been benched at up to a 7% load increase on its own; but again remember even on a low-end system a typical set of real-time tasks like word processing or instant messaging should be under 5% CPU usage. This is somewhere around the cost of C++ (depending on C++ features used; pure virtuals can cause nasty overhead, but there are worse runtime hits for other tools in C++), and still well below Java/Mono.

As for security, in the first type system something like Java or Mono may hold the key to a bit better setup. You have bounds checking, although a bug in the interpreter can fail; but a bug in the kernel or compiler can cause a fail in the other protections anyway. When you look at the second system, however, you find that there's not only no need; but the added complexity only introduces more chances for bugs. Some new methods of attack arise that are either out of scope for specific protections like compiler-level stack smash protection from this; other times, the method of implementation can simply require that one of the protections the system supplies is not used for functional reasons. In the end, you have two implementations trying to solve the same problem, and they can't really both apply to a given situation.

Re:Wrong solution for solving heap problems.

[barryman_5000]

Well done response but it only tends to suggest that if java itself isn't coded well then a bad program can cause stack smashing. If java allows a programmer to cause a problem then its java's fault. Everything is an object and I don't believe that a java program should ever reach (or can possibly) out of the jvm's memory range. In fact it can't address memory outright at all. Its all done via relationships of objects. This will limit a java programs ability to leak memory from another program.

If java is coded well (which means that its coded well in C by good programmers) and the fact you can't grab memory absolutely or anything else except relationship-wise, then java seems pretty self contained. In fact a java program shouldn't be able to touch another java's memory(hopefully).

The only thing I think could make a java program insecure is if I C program attacks a jvm specific problem. ie. java itself wasn't coded well.

Re:Wrong solution for solving heap problems.

[master_p]

Actually, x86 is one of the last into town with "NX bit" functionality.

I am aware of that fact, as well as the fact that protection that is 100% equal to the NX-bit can be achieved using segments on 80x86: the flat 32-bit memory model can co-exist with CS and DS pointing to different memory space. Self-modifying native code is something that is rarely been needed, and virtual machines could be modified overnight (they are not that many anyway) to request new 'executable' memory through a new 'malloc' subroutine.

The 80x86 CPUs are the dominant CPUs though, and most of them lack the NX bit, so I don't see a point in bringing other CPUs in the discussion.

The problem is that the Operating System can't tell the difference between a memory pointer that has stepped outside the malloc'd space it was intended to address and into an adjacent one, or two legitimate pointers behaving nicely.

Programming languages have advanced to the point that they can do some of the job of the operating system, 'proving' through their type system when a pointer is invalid or not or restricting pointer usage to the bare essentials (i.e. no wild tricks like pointer arithmetic and such).

Adding the guard space will definately help, as will unmapping memory.

Indeed, by why shouldn't the problem be solved once and for all?

C is not used for building web apps, so what are you talking about?

C may be not used directly, but it is used indirectly. All of the tools that are used for building web apps, and indeed any kind of app, including the O/S, is made with C. So a bug or invulnerability of the host environment will affect the web app, and the bug would have been made easier to make by using C.

Everybody knows C is the right tool for some jobs, Java/Perl/Ruby/etc for others.

Having the ability to manipulate data at bit level does not go against being safe. The problem is that the C programming language offers bit-level manipulation without providing the relevant safety and without actually provide a binary standard (bit-level manipulation tricks that C can do are often non-portable). There are lots of other programming languages that offer the same kind of bit-level management but are much safer (ADA, Cyclone, Eiffel, Erlang some of the languages that come to mind).

In the right hands C can be just as secure and has the potential to be much faster than most of the alternatives.

In reality, nobody has the programming ability to make 100% safe C applications...so some help should be provided by the language. And your comment about performance is misleading: ADA applications (for example) are just as fast, since ADA is, in reality, C disguised in better 'more safe' clothing: lots of errors can be eliminated at compile time, through the better type system.

Re:Wrong solution for solving heap problems.

[master_p]

C is actually the most secure language currently, AFAICT. Languages with higher level intrinsics (C++, Java, Basic, Mono, Objective-C, etc) have a more complex implementation that may allow different exploit vectors; while languages with real-time interpretation or runtime code generation (Java or Mono with JIT) will wind up disabling things like strong memory protection policies (strict Data/Code separation -- code is data when generated at runtime) and may not in their own code create a backup buffer overflow protection.

C++ is like C in the area of safety: there is no 'enforced' protection through the language; C++ leaves safety at the discretion of the programmer: if the programmer uses the STL containers properly, then safety is guarranteed. But that is not always the case: when under pressure or little experienced, it is easy to forget using the method 'at' (which throws an exception when the index is out of bounds) and use the operator [] which does not check the index, and thus compromise safety. In reality, C++ programs are never as safe as C++ can be, because C++ allows mixed-mode programming: C++ and C in the same program.

The other bunch of programming languages are safer than C, because they check the code they execute, and they can not write outside of their buffers, due to language constraints.

In the event of a screw-up on the part of the JIT or runtime programmer for any language, every program is instantly vulnerable, and all of this generic proactive security stuff is disabled because this "secure language" doesn't work in an "inherantly secure" environment, only a much weakened one. C's runtime is rather basic (and it's still huge), as is its language; people still screw that up once in a while, but rarely.

Reality proves you wrong: 99% of buffer overflows exploits are in C-based applications. And your quote that 'In the event of a screw-up on the part of the JIT or runtime' only proves that C is a dangerous language that should not be used! JITs and runtime-environments for sandboxed programming languages are made with ...C.

Look at this memory allocator and think about a bug in the allocator that smashes up its own memory before it gets everything set up

Indeed. C-based complex code like memory allocation algorithms are a liability. Managed code allocation where memory is garbage-collected and defragmented is much better: not only data do not remain in the same position in memory (thus making it difficult to exploit things), but garbage-collected applications run faster than those with manual memory allocation.

Several such enhancements exist, see PaX, GrSecurity, W^X, security in heap allocators, SELinux, Exec Shield, ProPolice. Higher level languages like C++ implement back-end instrumentation that ramps up complexity and may open new, unprotected attack vectors that are harder to quantify. Very high end languages on their own platform, like Java and Mono, not only implement massive complexity, but rely on a back-end that may lose its security due to bugs

Your argument that 'C is simpler, therefore there are fewer ways to exploit it' is good, and I agree with that. But all the security measurements you mention have failed to fix the problems, simply because it is too easy to have a security bug in C-based applications. Since CPUs do not enforce a strict separation (CPUs are badly designed, but I've got to live with that) of data and code, no security measure like the one you mention can guarrantee 100% safety.

Face it.

What I am facing, as a long time C programmer (which knows and uses Java, Ada and VB, and knows ML, Haskell and other non-commercial languages), is:

• programs that are nearly impossible to debug, since pointers can go crazy.
• thousands of time spent hunting obscure bugs.
• lots of irritation trying to code complex algorithms and master pointers and manual memory allocation, at the same time.
• excessive t

Re:Wrong solution for solving heap problems.

[LizardKing]

Wow, a Scott Nudds posting.

(this an impossibly obscure comp.lang.c reference that no one will get).

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

C++ is like C in the area of safety: there is no 'enforced' protection through the language

C++ has considerations beyond C that are not addressed in security enhancements used in secure computer information systems.

The other bunch of programming languages are safer than C, because they check the code they execute, and they can not write outside of their buffers, due to language constraints.

Built with ProPolice (which was basically merged with GCC 4.1), C cannot overflow a stack buffer in a protected function; it can miswrite if passed a bad pointer that starts past or before a buffer. With a proper memory protection policy like PaX gives, C programs can't execute data (stack, heap, anonymous mmap() segments, etc) as code anyway.

But all the security measurements you mention have failed to fix the problems, simply because it is too easy to have a security bug in C-based applications. Since CPUs do not enforce a strict separation (CPUs are badly designed, but I've got to live with that) of data and code, no security measure like the one you mention can guarrantee 100% safety.

PaX does a damn good job at stopping code injection attacks, and return-to-libc, due to its strict data-code separation. PaX uses the same method as ExecShield and W^X on IA-32 to fake non-executable high memory addresses; and in the low addresses it sets the supervisor bit on non-executable pages so it can interfere when a program tries to access them and make sure it's right. These protections are 100% guaranteed to do what they say; however, they don't protect against all attacks, nor does using something like Java.

C is not secure: we would not have this conversation if C was secure.

C is more secure in a 'secure' system that implements transparent measures like ProPolice, PaX, GrSecurity, ExecShield, W^X, OpenBSD's new heap protections, and other similar things than many other languages in the same system, due to the effectiveness of the protections and the demands and previously unpredicted problems that arise in some other lanugages.

The Java example easily proves you wrong: Java programs are 100% portable, coding is much faster since no memory management issues exist, and more maintaineable since there are no header/implementation file tricks, nor there are symbol collision issues due to packages.

Technically Java programs are not portable; Java programs run on the Java platform, which interfaces with the files on the system, networking, and display. The Java Platform is like its own system. Furthermore, sometimes Java programmers like to assume that you have "C:\Windows" paths, other times they assume you have "/home/james" paths; you look at Java programs like Limewire or anything else that's more than a few thousand lines of code and you'll see there's a version for Windows, Linux, and Mac. Finally, writing programs in Java means you have to either use only facilities there already; create your own bindings that have to be rewritten/rebuilt per system to interface with C libraries; or rewrite C libraries in Java. This has an effect when you want to use a special graphics or sound library, for example if you want Theora or Speex in Java.

I hope I cleared your confusion up. As for your points about maintainability or ease of initial coding, I'll agree that it can be tedious to initially lay out a C program; although most of that tedium to me is design considerations. Beyond that, you're probably just not a good C programmer, albeit you may be a good enough problem solver to still do amazing things with C.

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

Exactly my point. If the JVM/JIT is miscoded, you have a problem. My point with this was of course that even a well-behaved JVM that functions within parameters of all added security enhancements on a secure system brings with it new considerations and possibly new ways to attack it that may not be covered, all for the sake of implementing a 'secure' programming language on top of a system that's already 'secure'. Solving the same problem in the same system twice leaves you open to screwing up twice, instead of once.

Re:Wrong solution for solving heap problems.

[master_p]

C++ has considerations beyond C that are not addressed in security enhancements used in secure computer information systems.

Nope. All its problems are derived from C.

Built with ProPolice (which was basically merged with GCC 4.1), C cannot overflow a stack buffer in a protected function;

PaX does a damn good job at stopping code injection attacks

These programs are not a C standard yet...so you are talking about another language, not C.

There are also other C compilers beyond GCC.

And using all the programs you mention does not make C safer: the harm is already done.

C is more secure in a 'secure' system that implements transparent measures like ProPolice, PaX, GrSecurity, ExecShield, W^X, OpenBSD's new heap protections, and other similar things than many other languages in the same system, due to the effectiveness of the protections and the demands and previously unpredicted problems that arise in some other lanugages.

These programs are not part of C. They offer protection for assembly, ADA and all applications at the C level.

Technically Java programs are not portable; Java programs run on the Java platform, which interfaces with the files on the system, networking, and display. The Java Platform is like its own system.

But the Java platform is portable, and part of the specification...therefore, Java programs are portable. And having programmed several Java apps that run in windows/linux/mac/solaris as is, I can definitely say your argument is really really bad.

Furthermore, sometimes Java programmers like to assume that you have "C:\Windows" paths, other times they assume you have "/home/james" paths; you look at Java programs like Limewire or anything else that's more than a few thousand lines of code and you'll see there's a version for Windows, Linux, and Mac.

It's because people don't use the available Java libraries for managing system resources. Java has a whole preferences package for just that job.

Finally, writing programs in Java means you have to either use only facilities there already

oh come on...it's called software reuse.

create your own bindings that have to be rewritten/rebuilt per system to interface with C libraries

that's only if you need something Java does not offer...which is hardly the case.

And how is that different than C? don't tell me you have a magic wand that makes your C code portable (and if you have, it can be applied in Java as well).

This has an effect when you want to use a special graphics or sound library, for example if you want Theora or Speex in Java.

Java offers a complete multimedia solution that's not in alpha or beta state.

I hope I cleared your confusion up.

I was not confused. You made my confusion up, just to justify your argument that 'C is the safest language', which is obviously a good joke.

although most of that tedium to me is design considerations

The issues are real, and not design considerations.

Beyond that, you're probably just not a good C programmer

Aha..second conclusion about me...do you know me? do I know you? it's amazing how low one can go in order to 'prove' some argument.

albeit you may be a good enough problem solver to still do amazing things with C

your last sentence does not make sense. If one does amazing things with C, then he is a good programmer. Otherwise, he isn't. Beyond the contradiction though, you can't be serious that C is the safest language around. It's the ultimate joke...the joke of jokes. I will not forward your post to my programmer colleagues and friends, because they have work to do, and they can't go around all day laughing their asses out.

Re:Wrong solution for solving heap problems.

[bluefoxlucid]

|| C++ has considerations beyond C that are not addressed in security enhancements used in secure computer information systems.

| Nope. All its problems are derived from C.

So for example getting a mucked-up pointer to alter an exception handler is derived from C? (can be used with non-safeSEH Windows programs to defeat DEP, in theory) Oh wait, C doesn't have a standard exception feature. Guess not. It also doesn't deal with vector tables and other class object data.

| These programs are not a C standard yet...so you are talking about another language, not C.

Not programs. ProPolice is a modification to the compiler, similar to a compiler optimization (without these, btw, C is a thousand times slower than Java); MSVC also now has a similar but weaker protection. PaX and GrSecurity are operating system modifications. I'm talking about C on a specifically hardened system; the language is identical.

| And using all the programs you mention does not make C safer: the harm is already done.

Not really. The attacker introduces his attack with payload into the program; at the precise moment that the first instruction is about to be executed, the security systems are triggered. This delays the execution of the first instruction in the attack. The security systems determine quickly that there's something most definitely wrong, and take the program off the CPU and out of memory immediately, close all file handles, etc. Gone. Attack never happens. A crash happens instead.

| These programs are not part of C. They offer protection for assembly, ADA and all applications at the C level.

Right, they're part of the operating system. Notably though, if the program relies on certain attributes of memory layout or policy, the protections must be disabled. For example, last I looked (JRE 1.3), the Java platform had issues with changing memory layouts; if you randomly placed the stack/heap/mmap/executable base of Java in memory, it would crash. By design, you have to disable the strict data/code separations for a JIT or JVM, although precompiled native binaries from gcj aren't affected by this (except maybe as a bug). Of course also by design, Java doesn't concern itself with overflows and double frees, so in the unlikely event of a borked JIT or JVM you're screwed (back to relying on the C programmer who wrote the JVM to get it right, but you can't use the added security enhancements available "just in case"); this is mitigated to a degree with native binaries like those from gcj, because you can turn back on the data/code separation and the random address space and let that dredge out any code injection or execution flow changes.

| that's only if you need something Java does not offer...which is hardly the case.

GTK+ (so your Java apps don't look ugly), the new whatever-video-codec-just-got-invented-last-week, a new peer to peer protocol with a library for a reference implementation (like libpng is a reference implementation of PNG), anything available right on your system.

| And how is that different than C? don't tell me you have a magic wand that makes your C code portable (and if you have, it can be applied in Java as well).

Alright, I guess that was off topic; I still make the portability arguments with Java because people constantly brag about how you "don't have to recompile Java programs" because they're instantly binary portable, which they're not really.

Java offers a complete multimedia solution that's not in alpha or beta state.

Does it support my RSTDMIF movie format for rolling state translation differential moving image format that I just made up 10 seconds ago? Will it when my reference implementation is finished and working? No? What about on platforms with the C library? You'll then be able to get to it from C, C++, Objective-C . . . but not java? Not mono (C#) either? So then what's it go

Re:Sacrfice useability? Nice idea , won't work.

[Daniel_Staal]

Actually, for OpenBSD, it will work. And has.

The reason is very simple: There will always be some applications where the security of the system is a paramount point. Where it does have to do with the application. OpenBSD caters directly to those people, and those applications.

Now, you are right, this means OpenBSD is likely to never get as large a following as Linux or even FreeBSD, but they honestly don't care. They are making a system that fits their goals, and security is among the top goals.

This actually allows security to spread: Once these changes are on a 'major' system, applications start to be ported to work with them, which means the changes can be ported to other systems.

OpenBSD is a security testing ground. If it's features get in your way, you use a different system. This won't be the first time that advice will have been applicable.

Re:Have you ever noticed..

[ankarbass]

So it seems, we are birds of a feather.

Re:Sacrfice useability? Nice idea , won't work.

[Anonymous Coward]

I understand the sentiment, but ... consider the influence of laws like Sarbanes-Oxley. When CxOs go to jail because of their developers' bad code ... security might become more important.

Unnecessary when using languages that solve this p

[putko]

Languages like Lisp, Haskell, Scheme and ML allow you to avoid buffer and heap overflows (assuming the language implementation is correct).

It is therefore, in my opinion, less optimal (from a security perspective) to use something like "C" for a complicated app like sendmail, web server or secure shell daemon (sshd) than it is to use a language like "C".

Finally Locking the Door

[Doc Ruby]

I'm surprised that modern heaps still put writable data segments adjacent to executable code segments. Self-modifying code is rare enough that all code should be in read-only (except by privileged processes, like the kernel which sets it up and tears it down) segments, except when a process has privilege to write to its own code segment. Then its code segment should be a data segment, with other security features applied (that are too expensive to apply to every code segment). Generally then, segments are either writeable or executable. Data segments could still get overwritten, which could put unsafe values in unexpected variables (like "write to that file" instead of this file). But at least those insecure operations are limited to the operations programmed into the code, not arbitrary new code inserted into executables by buffer overflows.

After decades of these problems, and known techniques already applied in Computer Science, its surprising that we're only now seeing these techniques deployed in popular OS'es like OpenBSD. Hopefully the open nature of OpenBSD and other OSS OS'es will see them tested for winning strategies quickly, and widely adopted.

Re:Finally Locking the Door

[Anonymous Coward]

"Generally then, segments are either writeable or executable. Data segments could still get overwritten, which could put unsafe values in unexpected variables (like "write to that file" instead of this file). But at least those insecure operations are limited to the operations programmed into the code"

Such a solution would not protect against the following case: the program running is an interpreter, and the attacker changes the program residing in a data segment, e.g. by changing "rm junk" into "rm -r *"

Or are interpreters such as sh, perl and python already mapping script files read-only?

Re:Finally Locking the Door

[ep385]

Self-modifying code is rare enough that all code should be in read-only (except by privileged processes, like the kernel which sets it up and tears it down) segments,


You're forgetting about systems that do code generation on the fly such as Lisp systems. In such systems code is data and thus executable code is also just data. It's stored in the heap and must be executable.

Let's not restrict our operating systems to support only the lowest common denominator (the C programming language).

Re:Finally Locking the Door

[Doc Ruby]

Your example is one of the few cases of self-modifying code (or permission to do so). Old programming languages used to declare some "variables" as "constants". I don't know why we discarded that mode.

Re:Finally Locking the Door

[Doc Ruby]

I didn't say that the OS should prohibit the rare case of self-modifying code (like the tiny amount of Lisp). Just that they will require other techniques outside the scope of my suggestion, which targets the large majority of "static code" processes.

Re:Finally Locking the Door

[Bruce Perens]

You're forgetting about systems that do code generation on the fly such as Lisp systems.

Lisp? Forget about Lisp! Java does it.

But well-writen systems on good hardware (meaning not using i386 page-tables) use mprotect() to make code-pages executable and not writable, and everything else is not executable. So, systems with self-modifying code can be well-protected from this sort of attack. This is available from the OS on all POSIX systems where the hardware page-table-entry has the appropriate bits. One of our problems is that i386 does not. So, our main hardware platform won't protect us without kludges like Solar Designer's use of segmentation registers to establish a protected stack.

Did they fix this in X86_64? Can you make it available to a 32-bit address space without changes visible in user-mode? That would be worth working on.

Bruce

Re:Finally Locking the Door

[Bruce Perens]

Did they fix this in X86_64?

Yes. It's used in the NX patch for Linux from Ingo Molnar. Was merged into the kernel with 2.6.8 .

Bruce

Re:Finally Locking the Door

[AJWM]

It's not quite that easy.

Generally then, segments are either writeable or executable. Data segments could still get overwritten, which could put unsafe values in unexpected variables (like "write to that file" instead of this file).

The method of attack would then be to write to that file where that file is an executable binary. I.e., the attack takes place on disk, instead of in memory. (Oh sure, it'll be more complicated than that, but that's the effect.)

The Burroughs MCP used addition tag bits in memory so that executable code was unmodifiable at the hardware level. It got around the disk attack by enforcing that only an application tagged as a compiler could mark a disk file as executable, and only the kernel could tag an application as a compiler (and then, normally only by a command entered from the console). Writing to an executable file would immediately clear its executable bit. I still came up with two different attacks on that, but they were very tricky and one involved the use of a doctored (on a non-Burroughs system) backup tape.

Still, anything that helps prevent the easy exploits is worth looking at, and probably implementing.

Re: Finally Locking the Door

[gidds]

An interesting system. Still, in the modern Unix environment, how would it handle shell scripts? (Or Perl, awk, Python, Ruby, whatever.)

Presumably, another method of attack on such systems would be to write out source code, and then arrange to call the compiler on it to get your executable...?

Re: Finally Locking the Door

[AJWM]

Hypothetically, yes. The Burroughs OS put a lot of restrictions on languages that used pointers as such. The general programming language was Extended Algol (though PL/I, Cobol, Fortran, APL etc were available), which limited how you access the hardware. For systems level programming there was DCAlgol (Device Control Algol), which included a built-in command to issue commands as though they came from the system console, and Espol, an Algol-like language which included a built-in array called "memory" -- which is exactly what it sounds like. Both the DCAlgol and Espol compilers were restricted to the equivalent of the "root" user.

None of which is particularly relevant to today's Unix-like, paged memory systems. I've toyed with the idea of writing an emulator for the B6700 hardware (48-bit word plus 3 tag bits, would work fine -- with bits to spare -- on an AMD64) but unlike OS/360, none of the Burroughs system software (AFAIK) was ever made public. (Unisys has their own emulator, of course.)

Re:Finally Locking the Door

[Homology]

I don't understand why you undermine OpenBSD improvements in security. They made their improvements painlessly available for their users, and it works. What's so bad about that? It's not perfect, but sure beats "not used at all".

Re:Finally Locking the Door

[Bruce Perens]

I am not undermining the improvements. I am discussing the claim. The claim is that the software is 1) more protective than it really would be - because it doesn't protect small allocations and that it 2) outperforms Electric Fence, which it turns out it does only by (again) turning off protection for small allocations.

With security software, it's essential that you understand what's really going on.

Bruce

Pax?

[Anonymous Coward]

I thought that pax was a tar lookalike that nobody uses.

Re:Pax?

[MrDoh1]

Funny, I thougth PAX was a TV station no one watched.

Re:Pax?

[zyche]

I thought that pax was a tar lookalike that nobody uses.

...except OpenBSD. (Since it's BSD-licensed.)

Also Worth Mentioning

[RAMMS EIN]

This presentation (by Theo de Raadt) gives a good overview of the security features in OpenBSD (beyond what's already outlined on the OpenBSD security page). It covers W^X, random stack displacements, random canaries to detect stack smashing, random library base addresses, random addresses for mmap and malloc operations, guard pages, privilege revocation, and privilege separation. One thing it doesn't cover is systrace.

Heap protection?

[justforaday]

I call my heap protection mechanism "bumpers" : p

Re:Sacrfice useability? Nice idea , won't work.

[Anonymous Coward]

Um. What Theo means is that incorrectly written programs will fail to work. So, Linux will let you access a free()'d buffer, but OpenBSD will not. Since more programs get tested on Linux than on OpenBSD, the bug is never manifested and no one cares.

So, let's say your favorite software package is one of those programs. You won't be able to use it until someone comes up with a patch that fixes the allocation bugs.

Correctly written programs are unaffected. If you never access what hasn't been allocated, and don't touch what you've told the libc you want freed, you are fine. Can't say that about Linux.

Linux Had A Spec For This Ages Ago

[Anonymous Coward]

But Linus doesn't like specs so it got dropped !

Kerneltrap link

[Anonymous Coward]

In case you wanted to read the Kerneltrap article mentioned but not linked http://kerneltrap.org/node/5584

Skinny Elvis

[Anonymous Coward]

Gates may pull off a skinny Elvis, but for good ole fat Elvis... "Monkeyboy" Balmer got that all sewn up!

Posted Anonymous to preserve lame hope of a career at Microsoft. *wink*

Bring it to the PSP

[Anonymous Coward]

Let's hope Sony will bring it to the PSP, so it'll be safe from those dangerous .tif images attacks....

Not the same thing

[Anonymous Coward]

What you are talking about is the no-execute page protection. Unix has had this for most of it's existence. Until the last few years it didn't work very well on Intel systems due to hardware limitations (if you can read a page it is executable). The workaround involves using segmentation but it was very ugly. Recent processor updates have included the ability to set a no-execute bit which works even if the page is readable. Both OpenBSD and Windows (and Linux) support this already.

The _heap_ protection is additional. There are various attacks based on malloc()ed memory overflows which are thrwarted by use of non-readable memory barriers and better protection of the internal malloc datastructures. The no-execute stuff isn't really related, though I believe OpenBSD already disallows execution of malloc()ed memory by default.

Re:Not the same thing

[Henry V .009]

Windows hardware-DEP isn't quite the same as its software-DEP (the NX-bit stuff). It also does SafeSEH checks.

Re:Not the same thing

[csirac]

He was trying to say that windows's "DEP" stuff isn't anything new. It's been around on the big iron in the form of IBM's POWER, HP's PA-RISC, MIPS, Sun's Sparc, DEC's Alpha etc. CPUs for a very long time (as a hardware, not software implementation) and hence has been supported by most OSes for those architectures for an equally long time as well (including Linux).

http://en.wikipedia.org/wiki/NX_bit

Heap Protection vs. Managed Code

[hey]

I wonder of all these creative heap protection mechanisms (GCC 4.1, NX, etc) will largely remove the need for managed code (ie Java and C#). If you can write in C++ and not worry about double-free()s that pretty neat. Maybe even a "disruptive technology".

Re:Heap Protection vs. Managed Code

[Henry V .009]

If you are writing in C++, you should

1. Not be using free in the first place. It's an error to code C++ as if it were C—use new and delete
2. Not be doing explicit memory management. Use RAII (Resource Acquisition Is Initialization) practices. Not using RAII in C++ is an error. (Beyond resource handling nightmares, you aren't being exception safe without RAII.)

Re:Heap Protection vs. Managed Code

[Anonymous Coward]

Well, its not a "error" just bad practice.

Managed code wasn't invented for good and careful programmers, it was invented so
their bosses could be sure that no matter how bad the programmers the application won't have a leak, bound overflow, etc.

Managed code makes that promise and now (or soon) these heap protection thingies will too for conventional languages.

Re:Heap Protection vs. Managed Code

[Henry V .009]

Well, its not a "error" just bad practice.

And building a bridge from sub-standard materials is just "bad practice" too, I guess?

Re:Heap Protection vs. Managed Code

[0xABADC0DA]

I think the more pertinent question is, with all this cruft added on to C/C++/asm to make it somewhat secure and programs a little bit less buggy, why not just write in a safe language like Java or 'managed' C# instead?

Individual memory allocations in Java were already much faster than plain C heaps before adding extra overhead from many more virtual memory maps and immediate free's. I mean a simple "free(malloc(8k));" is going to run maybe two or even three orders of magnitude slower than a safe language's "new char[8k];". And it's still going to be unsafe and buggy. So what's the actual advantage to old-school coding?

If the kernel was written in a safe language the MMU could handle the write barrier used by the garbage collector (gc) to reap short-lived objects. So when an object is collected, instead of scanning the whole memory to see if a reference was taken to it or doing lots extra work when any references are taken, the hardware just tells it which pages were written to. So if you allocate 1gb in N objects but only modify one page of memory, only one page of memory is scanned for references. Basically memory allocation/deallocation overhead in that best-case example would take about N + 0.000004 instructions (in general it would take more time, but still far less than non-GC based heaps). With unsafe malloc/free this would be more like 100*N and with this openbsd immediate-release it could easily be 1000*N or more. Hell if the kernel just had any help for safe languages at all this would be possible (unfortunately kernel developers are notoriously against this sort of thing on purely ideological grounds).

Re:Heap Protection vs. Managed Code

[Anonymous Coward]

Build this kernel, then we'll talk. MMU rearrangement is slow by the nature of the hardware.

Re:Heap Protection vs. Managed Code

[0xABADC0DA]

You don't "rearrange" the MMU, you just check/clear the dirty bit in the page table. This bit is set by the hardware regardless so setting it is zero overhead. Clearing these bits is still only 0.006% the time of scanning all of memory. So unless you write to 32k unique pages per GC you get a huge benefit from this.

You could probably also use the MMU to reduce pauses in the gc... you determine what objects are unreachable in the background and using the dirty bit you can tell which pages may have references into the set, so instead of starting over from scratch again you just weed out the now-reachable objects from the set, which seems like it should be a much more tractable problem.

If I wasn't working I would do this. Check out jnode or jxos for example.

Re:Heap Protection vs. Managed Code

[Anonymous Coward]

I do not understand how a pure* "free(malloc(8k));" is different from a pure* "new char[8k];" in a "safe" language. Especially when most "safe" languages are implemented in C. Are you smoking pot ?

* by pure I mean no caches, whatever..

For Real Security

[RAMMS EIN]

For real security, don't use C. I've advocated this many times before, and have recently written a new essay about how better programming languages can lead to better software. Although this essay isn't entirely focused on security, it does demonstrate that some of the most common security bugs simply cannot occur using, in this case, Common Lisp instead of C.

Re:For Real Security

[zCyl]

Sure, you can find demonstrations of things that fail in C which won't fail in Common Lisp. But Common Lisp is not exactly known for the human readability of its syntax, and human readability is the key to reducing algorithmic bugs.

Re:For Real Security

[Jamu]

From the examples in the essay, you could easily make the case that for real security use good code.

Re:For Real Security

[Tyler Eaves]

Uh, lisp is EXTREMLY readable, assuming yoour're using a lisp-aware editor.

Re:For Real Security

[NullProg]

Just curious,

Do you also advocate four times the memory usage and double the speed? Do you blame the language or the speaker when they can't formulate a proper sentence construct? How about we just teach the programmers better instead of bitching about the tool they use.

If half the C coders out there knew the differences between stack, heap, and namespaces, we would not even be debating this issue. Don't blame the coders, blame the universities.

Enjoy,

Re:For Real Security

[RAMMS EIN]

``Do you also advocate four times the memory usage and double the speed?''

Depends on the situation. In my essay, I am speaking in general terms. It probably depends on the situation which language is best to use.

``Do you blame the language or the speaker when they can't formulate a proper sentence construct?''

In a way, I do. My girlfriend does not natively speak the language many of my friends and relatives speak. When she misplaces stress on a word, or gets the gender of a word wrong, I don't hold that against her. Her native language always puts stress on the first syllable and there is no grammatical gender, so you can't make these mistakes, just like you can't make mistakes that lead to buffer overflows or memory leaks in Common Lisp.

``How about we just teach the programmers better instead of bitching about the tool they use.''

Because that doesn't work. The pitfalls I mention in my essay, and how to avoid them, have been known for over a decade. Still, they constitute the majority of bugs found in software. Using a language in which these pitfalls do not occur is guaranteed to solve these problems.

``If half the C coders out there knew the differences between stack, heap, and namespaces, we would not even be debating this issue. Don't blame the coders, blame the universities.''

Eh? How many people learn how to code in university? How many of these universities teach people to code in C? Universities I've been to teach coding in Java, Scheme, or Haskell; languages that don't suffer from many of the weaknesses outlined in my essay. All these universities teach techniques to improve code quality, and some explicitly cover the pitfalls in my essay. I don't see how universities deserve any blame for this.

Re:For Real Security

[zCyl]

Syntax highlighting and auto-indentation can only take you so far. Lisp itself is designed the way a computer thinks, not the way a human thinks. Sure, we can train ourselves to process Lisp code, but this will never compare to the debugging ability from that corresponding level of training in a language designed more along the lines of human thought.

Re:For Real Security

[Sebastopol]

That article you posted was pathetically weak. If it wasn't for your resume, I'd have assumed you were ~15 years old.

You're going to need more experience before you can back up that claim. There's an excellent post in this discussion that exactly counters your statement, with far more intelligent proof.

Re:For Real Security

[NullProg]

``Do you also advocate four times the memory usage and double the speed?''
Depends on the situation. In my essay, I am speaking in general terms. It probably depends on the situation which language is best to use.

In what situation? In my current and previous jobs, my programs have had to be fast, portable, and easy to maintain.

Eh? How many people learn how to code in university? How many of these universities teach people to code in C? Universities I've been to teach coding in Java, Scheme, or Haskell; languages that don't suffer from many of the weaknesses outlined in my essay. All these universities teach techniques to improve code quality, and some explicitly cover the pitfalls in my essay. I don't see how universities deserve any blame for this.

Universities deserve fault because they are not teaching micro-processors. These days they teach mostly VM's. Eventually in a programmers life he or she will have to talk to the CPU. All the new programmers I've met in the last five years are woefully ill-prepared. How are you going to tell a recent graduate that his program needs to run on a x86, x64, POWER5 and a CELL Processor? How are you going to teach them to program (the eventual) quantum computer? Programming safely is a mindset, reguardless of the language. It doesn't matter if the programmer is using VB/Java/Haskell/C++ whatever.

Thanks for the response.
Enjoy,

Re:For Real Security

[RAMMS EIN]

``In what situation? In my current and previous jobs, my programs have had to be fast, portable, and easy to maintain.''

Common Lisp offers all of these (although for speed you might need a commercial implementation). What could be troublesome is real-time and/or embedded systems. Lisp, like any garbage collected language, runs better when given more memory (even when that memory isn't in active use), and real-time requirements conflict with high-performance garbage collection algorithms (at least, they did when I last read up on them; things may be different now). I'm not saying Common Lisp would be unsuitable for these environments, just that I can see problems there.

``Universities deserve fault because they are not teaching micro-processors. These days they teach mostly VM's.''

Indeed, the courses I took that related to real CPUs were not compulsory. Still, VMs are usually very comparable to real CPUs when it comes to things like memory layout (and ways to exploit that) and paralellism (and the troubles that leads to).

``All the new programmers I've met in the last five years are woefully ill-prepared.''

Maybe. But it's not the universities' job to train programmers. Universities train researchers. For research purposes, VMs are just as valid as real CPUs, and possibly more interesting, because it's easier to build and experiment with a VM than with a real silicon CPU.

``How are you going to tell a recent graduate that his program needs to run on a x86, x64, POWER5 and a CELL Processor?''

Well, if you write your program in a portable language (which both C and Common Lisp are), there should be no problem. Sure, you can gain some benefits from platform-specific optimizations, but you'll rarely need both those and portability, and if you do, you just have to be prepared to spend a lot more on development and maintenance.

``Programming safely is a mindset, reguardless of the language.''

Ah, finally we get back to security. Yes, what you say is absolutely true. No language can protect you against all possible mistakes. You will note that my essay doesn't address issues like temporary file creation, which is another major source of vulnerabilities. It's possible to write code that generates predictable temporary filenames in any language, so people will keep making that mistake.

However, don't make the mistake of thinking that just teaching people how to write secure code obviates the need for safe languages. There will always be people who don't have the security mindset, and people will always make mistakes. Given that the most common vulnerabilities found in software are possible in C, but not in Common Lisp, I claim that using Common Lisp instead of C constitutes a definite step forward in terms of software security. The fact that these vulnerabilities have been the most common for as long as I've read security lists shows, to me, that you can't rely on programmers to not write insecure code.

Re:For Real Security

[RAMMS EIN]

While I have a feeling it's pointless to argue this with you, I'm still going to try.

``Lisp itself is designed the way a computer thinks, not the way a human thinks.''

That's right, and it's a great advantage. Where other languages have two different idiosyncratic syntaxes for invoking functions and doing arithmetic, Lisp has only one syntax for both of them, and it's much more regular. This makes Lisp code very easy to read and write programmatically (also for humans, if you can get over the fact that you got taught the idiosyncratic syntaxes in school).

Lisp makes great use of the simplicity of its syntax by offering you powerful mechanisms for program transformation (macros, but they are vastly more useful than macros in other languages). There are also functions for reading and writing Lisp expressions (which can be data or code), which makes it very easy to implement things that operate on Lisp code (e.g. syntax highlighters), but also configuration files.

It's worth noting that although all these features could be implemented in languages with different syntaxes, there do not seem to be any languages that have these features, besides Lisp. Also, although several attempts have been made to introduce a more canonical syntax for Lisp, programmers have prefered Lisps s-expressions every time.

Lisp syntax takes a while to get used to (but so does any syntax!), but then it has great advantages over the available alternatives. My only gripe with it is the outer set of parentheses, which I see as inhibiting interactive usage (think shell).

Re:For Real Security

[RAMMS EIN]

``That article you posted was pathetically weak.''

Care to elaborate?

``You're going to need more experience before you can back up that claim. There's an excellent post in this discussion that exactly counters your statement, with far more intelligent proof.''

Which post is that?

As for the experience argument, I find it's usually the people who oppose my view who lack the experience or sense of reality. YMMV, of course, but the complete lack of anything substantial in your post makes it hard for me to come up with sensible arguments or evidence.

OpenBSD Goals

[RAMMS EIN]

Yep. After all, the goal of the OpenBSD project is not simply to be the most secure operating system ever. The goal is to provide the best security, along with a number of other goals, such as running Unix software, achieving good performance, and providing a good (according tho the stated goals even "the best") development platform.

Re:Unnecessary when using languages that solve thi

[Listen Up]

ALL buffer and heap overflows in individual programs are the fault of bad programming, not bad programming languages. If a programmer is not competent enough to test their code, and then blame the programming language when there is a problem, should not be coding. It is simply irresponsible.

You forgot to add Java to your list of programming languages which goes out of its way to assist in preventing and avoiding both buffer and heap overflows/errors (i.e. bounds checking and similiar technology).

This is how Electric Fence works.

[Bruce Perens]

Electric Fence explicitly allocates dead pages at the end (or configurably, the beginning) of buffers. It can also protect the memory immediately as it is freed. I think it was first published in 1987.

It may be a legitimate invention - it is cited as prior art in an ATT patent. This is also the first known example of a prior Open Source publication causing a patent filer to cite. ATT also removed a claim from the patent that my work invalidated. Just search for "Perens" in the U.S. patent database to find the patent.

We don't run it on production programs because of its overhead. To do this sort of protection exhaustively, it requires minimum two pages of the address space per allocation: one dead page between allocations and one page allocated as actual memory. This is a high overhead of page table entries, translation lookaside buffers, and completely destroys locality-of-reference in your application. Thus, expect slower programs and more disk-rattling as applications page heavily. If you are to allocate and release memory through mmap, you get a high system call overhead too, and probably a TLB flush with every allocation and free.

Yes, it makes it more difficult to inject a virus. Removing page-execute permission does most of that at a much lower cost - it will prevent injection of executable code but not interpreter scripts.

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

Bruce

Re:This is how Electric Fence works.

[Anonymous Coward]

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence."

Yes, I do wonder how many open source coders have tested their app with Electric Fence. In many cases I do wonder if there was any testing at all. OpenBSD seems to be one of those rare F/OSS projects which has actually designated time to auditing and improving core code and removing bloat. There are some very high quality apps out there, but I doubt many have gone to the bother of running their code through proper verification checks etc.

Re:This is how Electric Fence works.

[LurkerXXX]

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

I think that's exactly the problem though. There are users out there installing applications they found somewhere, by somebody who may or may not have bothered to use a good debugger. This will prevent those unknown bad apps from fouling up the system.

I'm still on OpenBSD 3.7. I haven't tried the 3.8 builds, but I'm hoping the overhead from this won't be too bad.

Re:This is how Electric Fence works.

[Homology]

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

Actually, several bugs in various ports was found because of this new allocator, as a search in the ports@ will show you. This does not imply, of course, that running Electric Fence will find even more bugs.

Re:This is how Electric Fence works.

[Homology]

Bugger, there is a missing "not" in my post. Here is the correct one:

Actually, several bugs in various ports was found because of this new allocator, as a search in the ports@ will show you. This does not imply, of course, that running Electric Fence will not find even more bugs.

Re:This is how Electric Fence works.

[Bruce Perens]

On shared-library systems, you can set an environment variable that tells the linker to load Electric Fence every time. Anyone with the patience to try that might be rewarded with some core dumps for which productive bug reports can be made.

Bruce

Re:This is how Electric Fence works.

[Anonymous Coward]

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

The OpenBSD allocator is on all the time, no need for re-linking.

This means the (many) end-users will test more code paths than the (few) developers using libefence.

Re:This is how Electric Fence works.

[Bruce Perens]

It would generally find more because it exhaustively protects allocations and the new BSD allocator only does so when the allocation is greater than a page in size (8K?). Electric Fence positions the allocation at the end of the page rather than the start (unless you are testing for underruns), I wonder if the BSD allocator does this.

Image processing performance on some architectures can be horrible if all buffers begin on a page boundary. Some processors only have 2-way set-associative caches. This reveals worst-case behavior with a two-operand image processing operation and one output buffer. You can end up refilling a cache line with every operation. Many image processing libraries assign a random pad at the beginning of each buffer to get around this.

I think the iAPX 432 may have been the best approach to this problem so far, as it put every function in its own privilege ring. It's surprising that nobody is thinking that way today.

Bruce

Re:This is how Electric Fence works.

[stab]

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

The OpenBSD allocator already has revealed a number of software bugs (in X11, in ports, they lurk everywhere). Some of the bugs found were years old. Thats the point of the testing process in the OpenBSD release cycle.

I think you're missing the point behind the integration of these technologies into OpenBSD. The idea is that they are always on, with a performance hit that is acceptable that your day-to-day programs can be protected, and most crucially, used under them. Not sitting in a debug environment getting limited regressions and unit tests that the particular programmer felt like writing (and if I want that, I run it under Valgrind, which has a near-miraculous tendency to find lurking bugs).

And considering them in isolation is also dangerous. When you combine the address randomization, W^X, heap protection, propolice canaries and local variable re-ordering, you're left with a system that has accepted a reasonable performance hit in return for a large amount of protection against badly written code. Sprinkle in regular audits, timely releases every 6 months to keep our users up-to-date on stable systems, and a 'grep culture' to hunt down related bugs in the source tree when a bug does strike.

As others have pointed out, other "hero projects" have stuck bits and bobs into their respective distributions. But how many have had the discipline to follow through, maintain and integrate their patches, test the fallout and release a complete OS with thousands of third-party packages year after year... probably only Microsoft, but the first thing they do at the sign of incompatibility is to turn the protection off. Oh well :)

Re:This is how Electric Fence works.

[Bruce Perens]

Well, unfortunately, it appears that the way to get performance is to only allocate dead pages around allocations that are one page in size or greater. What's that, 4K? This is larger than almost all strings one would care to read and most C++ objects, for example. So, it doesn't surprise me that the examples found are regarding large (image-sized, in the case of X) buffers.

It would be nice if we could get to the point that it would be possible to run the maximum protection in a day-to-day environment. That is more than we can ask of the hardware at present.

Bruce

Re:This is how Electric Fence works.

[stab]

The guards pages are only practical for larger allocations due to hardware limitations, as you say. Strings are protected by different means such as Propolice; in order to minimise the overhead of Propolice, it "detects" strings (as opposed to byte buffers) and specifically protects them with canaries to try and find overflows that would smash the stack (the local variable re-arrangement tries to put these buffers as close to the canary as it can at compile time). The string detection is a heuristic as gcc doesn't maintain quite enough type information when it reaches the code generation parts which Propolice touches. Also, there is a simple bounds checker built into gcc which looks for incorrect use of statically allocated buffers with some standard functions such as strncpy or sscanf (you'd be amazed how many people specify the buffer size wrong to a bounded function).

None of these are perfect of course, but each of the techniques has found bugs (hundreds in the case of the two mentioned above) in our source and ports trees. It's also great to see projects like CCured being developed at Berkeley; although the overhead is just slightly too high to be used "out of the box" right now, it still works great with select applications such as Apache. The underlying tool, CIL can compile most of the OpenBSD source tree (including the kernel) now, and the result even boots when using a null source-to-source transform.

Re:This is how Electric Fence works.

[labratuk]

To quote Theo from the link in TFA:

To some of you, this will sound like what the Electric Fence toolkit used to be for. But these features are enabled by default. Electric Fence was also very slow. It took nearly 3 years to write these OpenBSD changes since performance was a serious consideration. (Early versions caused a nearly 50% slowdown)

Re:This is how Electric Fence works.

[Bruce Perens]

Hm. One way to do the little ones, another for the big ones, and another for the static ones. My feeling for the past decade or so has been that this entire problem just cries out for hardware to address it. In some ways, we were better off with segment registers, since what we are really trying to do here is implement a non-contiguous address space.

As I have written elsewhere in this thread, we had the iAPX 432 sometime in the 80's, which could actually be made to run fast if done today, and which kept every function in its own privilege ring.

I don't understand why we do not see more innovative hardware architecture around the issue of security in general.

Bruce

Re:This is how Electric Fence works.

[Bruce Perens]

But it seems that the performance increase depends on turning the feature off for the vast majority of allocations - those under a page in size. This should not be taken as a comment on the implementation of Electric Fence and the superiority of the BSD implementation. It's an observation that the hardware does not run efficiently when you use this technique.

Bruce

Re:This is how Electric Fence works.

[stab]

It's a real struggle between the typical programmer's desire to loosely specify what they want in quick-and-dirty code, vs the operating system's desire to enforce this moving "security" target we keep talking about. And all sandwiched around the limitations of current hardware. Researchers wanting to play in the hardware space have it tough as well; the hardware/software co-design needed (to hack the FPGA breadboards together, get kernels booting, figure out why the VHDL compiler is segfaulting on valid code, user-space gcc hacks to emit new instructions, etc) are pretty demanding.

There's been huge amount of work on the rigorous software theory side in the last few decades, which eliminates the need for hardware to dynamically enforce things that can be statically proven away. Peter Sewell and his team recently successfully formally specified the socket API and machine-proved it. Projects like Microsoft's Singularity, or Cornell's Typed Assembly Language are looking at making practical, high-performance type-safe operating systems. I'd expect that, in the next few years at least, we'll see one of these approaches come to fruition before fine-grained hardware enforcement. And as a long-term goal, isn't it more desirable to have higher quality software than more paranoid hardware?

Re:This is how Electric Fence works.

[Anonymous Coward]

You're right.

But yet, since they commited this protection on cvs HEAD, OpenBSD's devs have found many bugs on many heavily used components (including x.org, kde, gnome).
For sure, those bugs should have been caugth by Efence: theyre may still be lazy devs, around there, that don't take full advantage of Efence checks. So here my conclusions:

• Some devs weren't using Efence at all (yes, they should have, but ...). Having a similar protection on by default on OpenBSD helps to finish their work, since the app is exposed to the protection.
  
• Some devs weren't using Efence deeply / long time enough to have checked all code path. Maybe because of the slowdown that make it harder to use beside debugging situations. This implementation doesn't put a very noticable slowdown. The system remains usuable on a daily basis, so people (I mean, even end users) have there a better opportunity to have the softwares checked in depth (not only during short debug sessions).
  

The keywords here are "system wide, by default, all the time", "no huge slowdown" and "more exposure".

So yes, Efence does this, and better. And we are all are very thankful to you for this great lib. But having more code (all the code running on OpenBSD) exposed to a similar protection, and having it exposed for a longer time (as long as it's used on OpenBSD, basically) isn't a bad thing.

People shouldn't think of OpenBSD protections and Efences as competitors, but rather complementary tools: Efence has more granularity, people should continue to use it. OpenBSD gives more exposure. At the end of the day, having both available results on having better softwares on the OSS world. All the world is pink !

VBLinux

[Frankie70]

  For real security, don't use C.


I am rewriting Linux in Visual Basic 6.0.
I am going to call the distro VBLinux.

Re:VBLinux

[Anonymous Coward]

GOTO hell!

Re:VBLinux

[i_finally_got_an_acc]

You laugh, but why doesn't someone compile a linux kernel with cyclone? Why not attempt to make a D or even OCaml port? These languages are all safer than C but still compile to machine code.

Re:VBLinux

[Anonymous Coward]

Thats VBGNU/Linux you insensitive clod. -RMS

Re:VBLinux

[Clover_Kicker]

Probably because it would take a lot of work.

Re:OpenBSD does not matter

[Ulrich Hobelmann]

Who is "not giving back?"
Feel free to download all of their source code before complaining.
If you don't like it, don't use it, even though I'm sure you too benefited from some security fixes that originated with OpenBSD.

/me can't resist

[SilverspurG]

From the article:

" The more hurdles that one has to jump through for good security, the less likely people will go through the trouble. OpenBSD allows even the most inexperienced users to take advantage of these technologies without any effort. "

Can they fix the government, too? It sounds like the same problem.

Performance?

[cardpuncher]

I'd be interested to know what the performance impact of this is - exactly what counts as "too much of a slowdown".

Application heap allocation has "traditionally" been fairly inexpensive unless the heap has to be grown (update a couple of free block pointers/sizes) and the cost of growing the heap (which requires extending the virtual address space and therefore fiddling with page tables which would on a typical CPU require a mode change) is mitigated by allocating more virtual address space than is immediately needed.

If free space is always unmapped then each block allocation will require an alteration to the page tables, as will each unallocation. Not to mention that could cause the page-translation hardware to operate sub-optimally since the range of addresses comprising the working set will constantly change.

If most allocations are significantly less than a page size, then the performance impact may be minimal since whole pages will rarely become free, but if allocations typically exceed a page size, that would no longer be true. If the result is that some applications simply implement their own heap management to avoid the overhead, then you've simply increased the number of places that bugs can occur.

Re:Performance?

[Anonymous Coward]

If you're really interested in knowing what the performance impact is, why don't you install OpenBSD 3.7 and time your apps, then install OpenBSD 3.8 snapshot and time your apps. Oh, wait, that would be *you* doing the work, not reading from slashdot.

Re:Unnecessary when using languages that solve thi

[Ulrich Hobelmann]

I like Lisp and ML, really. But the issue of automatic memory management is (or should be) orthogonal to the choice of language. There could be better languages than C with manual memory management. In fact, IMHO the world could use just such a language, as "managed" languages aren't appropriate for everything.

Given good libraries, a mail daemon or web server isn't a problem in C. But while I used to like C, I'd prefer to use Java, or better - Lisp, today, because it's tedious to write all that support code in C that the others have built-in. A C with better syntax and abstraction facilities, and more extensive and non-braindead standard libraries could do a great job for me.

Re:Unnecessary when using languages that solve thi

[putko]

Although you can blame the programmer for writing a faulty application, I don't think that is relevant.

You are a taxi driver. Your job is to deliver people from point A to point B, safely and quickly.

You can choose from two cars.

Car A is such that if you make an error when driving, it will do something bad -- perhaps even blow up, killing the passengers. Furthermore, a driver can accidentally drive the car into an obstacle, ruining everything.

Car B is the same as the first, except there's nothing the driver can do to make the car blow up. If the driver purposefully tries to drive the car into a wall, it stops before it hits the wall.

It seems like you want to pick car A, and say that anyone who drives car A badly, is at fault.
I'm saying just pick car B, there will be far fewer accidents, and there is no reason to point your finger at the "bad driver".

If a driver is given the choice between the two, and picks car A, and makes a mistake, it would seem that he has been grossly negligent. He could have chosen the safe option, but he chose the unsafe one, and the passengers got killed. So I think we agree the programmer has been terribly irresponsible.

Re:Unnecessary when using languages that solve thi

[Rycross]

All languages allow you to avoid buffer and heap overflows if the language implementation is correct. Buffer and heap overflows are programming errors, not problems with a language. Some languages just have better facilities to force well-behaved code. You can add Java and C# to your list there by the way.

Re:Unnecessary when using languages that solve thi

[putko]

I don't understand how you'll do a language like ML or lisp without automatic storage management.

The problem has to do with returning closures with bound variables -- the nice thing about having a GC is that they will get freed, eventually.

If you do manual memory management, will you manage this yourself (for closures)? Or will you ban closures? If you are banning closures, you've lost one of the key benefits of using Lisp or ML --- you just got rid of lambda (in its most general form), and you'll just have supercombinators (functions without bound variables), like in "C". Or will you rely on some static analysis to find things that can be freed automatically? These are all pretty tough -- just using GC is probably the way to go.

Re:Unnecessary when using languages that solve thi

[hackstraw]

Languages like Lisp, Haskell, Scheme and ML allow you to avoid buffer and heap overflows (assuming the language implementation is correct).

It is therefore, in my opinion, less optimal (from a security perspective) to use something like "C" for a complicated app like sendmail, web server or secure shell daemon (sshd) than it is to use a language like "C".

The problem is for one reason (or randomness) or another, no mainstream daemons or operating systems or client applications are written in Lisp, Haskell, Scheme, or ML. Plenty are written in C, C++, and Objective C, and I have yet to of heard of a rewrite into another language in order to alleviate the security risks because of the current language.

I cannot explain why the other languages are not used. I have not used any of them personally, but somebody should be able to shed light on the subject.

Not really unnecessary

[dereference]

ALL buffer and heap overflows in individual programs are the fault of bad programming, not bad programming languages.

I agree, and I'd go one step further to say that we're starting to get into a situation where programmers are learning and using languages (such as Java) that don't allow this particular kind of sloppy coding. The problem is that many of these programmers aren't even aware of the concept of a buffer overflow, let alone how to actively detect or prevent it.

I happen to be a fan of Java, but I get very concerned when I hear folks suggest that we can solve all our security woes by simply using these "safe" programming languages. We can't totally ignore the fact that coders need to understand these issues, whether the programming language (or operating system) protects them or not. It's rather like suggesting that drivers don't need to learn to drive safely any more, since we all have seat belts and air bags.

ElectricFence

[Mike McTernan]

Didn't Bruce Perens use a similar method of page protection to implement Electric Fence?

http://perens.com/FreeSoftware/ElectricFence/

If this is the case, it's great that such a feature should go into an OS by default. I personally love anything that gives me confidence in the implementation of any applications I write, especially as this type of technique makes debugging much easier.

Re:ElectricFence

[Nimrangul]

Same concept yes, but it's been built from scratch over the past 3 years to not be such a massive slow-down to the system. You'd have known this had you read the linked KernelTrap article.

Re:ElectricFence

[Mike McTernan]

Oh. I only read the article linked from the story and not the thread linked from the story linked from the article...

Anyway, Perens himself comments on the original story mentioning efence, so I guess my post is redundant in any case. Ho hum.

Re:Unnecessary when using languages that solve thi

[mypalmike]

I like your analogy. I'd change it so that the two cars are:

- Car A is such that if the taxi driver makes an error, the car explodes, killing everyone in it instantly.

- Car "C" is such that the taxi driver is a serial killer who picks up passengers, hunts down, tortures, and kills their immediate families, and then slowly, painfully kills the passengers.

Neither is particularly desirable, but I'd still take car A. :)

Re:Unnecessary when using languages that solve thi

[Anonymous Coward]

Languages like Lisp, Haskell, Scheme and ML allow you to avoid buffer and heap overflows (assuming the language implementation is correct).

C allows you to avoid buffer and heap overflows. C++ does too, it provides even more tools.

Would a programmer incapable of writing code without buffer overflows be capable of writing complex client-server apps in those languages?

This is great. Just super

[ckaminski]

Now what I'd like to see is thread-protected heaps. Where malloc()d memory is accessible only to the thread that created it. Yes, some mechanism would have to be created where a thread could delegate access to it's heap, I suppose. There are times where I'd want to start a series of threads from a message dispatcher, and allow the children to only access the parent's message queue, and not allow the parent access to the child's message queue (Yes, I understand there are better mechanisms for this like pipes, but it's a simple example).

Some platforms just don't support the fork() mechanism, and I'd like one more tool to help make threaded system design a little more robust.

Re:Unnecessary when using languages that solve thi

[Anonymous Coward]

Legacy code. Conservatism. Efficiency (among implementations of equivalent algorithms, the one coded in lower-level language will 99% jf the time be faster/lighter on memory).

Re:Unnecessary when using languages that solve thi

[Todd Knarr]

All well and good. But suppose that car B, with all it's other advantages, is limited to a top speed of 45mph and can't go onto the freeways while car A has no problem with freeways. Most of your customers will want to go places where the fastest way there involves taking a freeway. As a taxi driver, which do you choose? For me it'd be easy: car B isn't an option because it doesn't meet minimum requirements, so car A's the only choice I have.

I ran into this situation when having to choose between Java and C++ for a program. The absolute requirement was consistent high performance (on the order of 500 transactions/second). It would've been easier to write this program in Java, frankly, and it'd've been theoretically able to meet the performance criteria. But. Yes, there's always a but. In this case it was the garbage collection. At random unpredictable and uncontrollable intervals the whole program would stop dead while the JVM did garbage collection. We tried multiple JVMs, we tried every option we could think of, we called in the vendors and had their techs try their hand at the problem. No joy. In C++ we had to manage memory ourselves and take care to avoid leaks, but at least we could at the same time slice deallocation up so it was a steady small overhead we could manage and account for in the performance budget.

Scheme

[RAMMS EIN]

``Not only do they tend to lack explicitly separate data and code''

Where do you get that idea from? True, you can store functions in variables and pass them as arguments, but that doesn't mean that you don't know what's code and what's data anymore. The function body is code, the pointer to that body is data.

``but they more-or-less depend on the ability to construct functions dynamically, as data, and then invoke them as code.''

They don't really depend on that. It's a possibility, but most often functions constructed at run time will use code that is already known at compile time (and thus would already have been generated). In the cases where you really construct new code at run-time, you'll have to do some trickery; but you still don't need to have any part of memory be both writable (data) and executably (code) at the same time.

Also, arguably, Scheme doesn't need this kind of protection, because the sort of bugs that this heap protection scheme protects against simply aren't possible in Scheme programs. C and C++ are actually among the few languages in popular use that suffer from these vulnerabilities. See also my essay about the subject. Although the examples it gives use C and Common Lisp, they are almost equally applicable if you substitute C++ and Scheme.

Re:Sacrfice useability? Nice idea , won't work.

[xgamer04]

Yeah, just like nobody used any of those other security projects from OpenBSD, like OpenSSH...jeez, what do we expect, people to start taking advantage of security measures? (oh yeah, read all the AC replies, they're far more eloquent than I.)

linux had this for years

[Anonymous Coward]

http://pax.grsecurity.net/
Protection Against eXecution.

and integrate it with something decent like http://www.rsbac.org/

OpenBSD is about security as a priority

[Anonymous Coward]

OpenBSD focuses on security, and this latest addition is not just a security measure slapped on to just another Unix clone. The developers encourage an environment of secure design as the top priority. Many other operating systems choose to focus on performance or a great desktop experience.

Clearly a conspiracy

[Anonymous Coward]

It almost seems like a tit for tat as OpenBSD leads the OSS community into adopting bloatware in the name of "security" in response to noise from Redmond about how they justify bloat in the name of "security" (much like the White House justifies... er...). Meanwhile, the astute /.er will recognize the spector of Andy Grove is steadily pushing all we rates closer to the edge of buying more and faster processors. Surely, there must be at least ONE Intel employee working with OpenBSD to solve this 'security issue'. Kurzweil was wrong!

ownage is still eminent

[y0-w00t]

nuff said

Re:This is great. Just super

[Paul Rose]

Memory protection between threads?

Good idea, let's call it "processes"

Re:cutting edge?

[Anonymous Coward]

I have a Hardened Gentoo Linux system (http://www.gentoo.org/proj/en/hardened/primer.xml ) that already incorporates similar technologies. It uses the PaX kernel patch for randomly arranging the memory layout of applications. It also compiles in stack smashing protection into every program.

Re:Unnecessary when using languages that solve thi

[putko]

Here's something from guys who do lots of transactions in a safe language -- perhaps you've heard of Orbitz? If you'll notice, they manage memory so that they avoid GCs whenever possible.

My original point was more about things like sshd -- there's just not a good reason for that to be written in an unsafe language. It defeats the whole purpose of your "secure" shell if a buffer/heap overflow allows someone to compromise your whole box.

Normally the code that needs optimizing is a small part -- you make that go fast and you can achieve your goals.

MS did that a while back

[YesIAmAScript]

And updated it in Service Pack 2.

There was a big story on slashdot on how it had been worked around by programs that wanted to still be malicious.

MS' support works by just checksumming the structures between blocks. If the checksum doesn't match, someone has overwritten the structure. This can be worked around if you wish to be nasty.

Sounds like BSD is putting unwriteable pages between blocks in the heap. This is impractical and/or stupid. It means you have to waste the remainder of a page (real memory) and the next page (only address space) for each block you wish to protect. In addition, unless the block is a near multiple of your page size (4K) you have to decide whether to butt it up against a block behind it (to detect overruns) or in front of it (to detect underruns).

You can beat both of these limitations by making portions of pages unwriteable instead of the whole thing. This requires making the page unwriteable in hardware and then on every write to that page, bounds checking the pointer and either doing the write at privileged level (where it will succeed) or failing, depending on the result you want. This is terribly slow, it will destroy throughput.

All in all, I think I like MS' solution the best, in terms of practicality.

Either way, it must come as a huge shock to hardcore slashdotters to hear MS beat them to a security feature. I'm sure they'll manage to block this out of their minds and return to the party line in short order.

Re:MS did that a while back

[FrostedChaos]

Better re-read the OpenBSD implementation, and probably some background material too, because you got it completely wrong.

Patent applied for...

[It doesn't come easy]

Key word search "Heap-o-Protect"...

This is why I couldn't use OpenBSD exclusively...

[ArbitraryConstant]

They're willing to break things in order to improve security. That's commendable, and I can't see myself using anything else for the firewall, but I simply cannot do without some software and some of it is binary-only. Does it break for me? I don't know, but from what Theo has said breakage isn't uncommon for large applications. I haven't checked because Java is poorly supported for unrelated reasons, and this rules out OpenBSD without me having to validate all my other software.

Also, the OpenBSD crowd is very quick to say that performance should be good enough for most purposes, but that's a copout. They have no idea what any particular person needs to do. The double-halt bug is a good example of how this is an issue. If they don't pay enough attention to performance to catch such a major issue quickly, they aren't going to be catching up to Linux or FreeBSD anytime soon.

I guess in debug mode...

[Anonymous Coward]

Electric Fence for Linux and Xcode's Gaurd Malloc library do this. You could do it in production if you really wanted to but *why*? There is a heavy performance hit and an attacker can avoid (albiet not easily) the gaurd pages anyway.

Individual application owning in a dedicated VM OS

[Anonymous Coward]

Eventually, thanks to processor virtualization each application will run in a dedicated VM with an OS instance dedicated to it. Not sure if this will cause a slowdown and memory hogging.. but I think we're moving towards that given the reduction in RAM prices and the quest for improved security and reliability. Like it or not.

Re:Unnecessary when using languages that solve thi

[Ulrich Hobelmann]

I'm not talking of something like Lisp, just C with more syntax for abstraction mechanisms, like real closures, maybe a type system that wouldn't get in the way with tuples and higher-order functions (or no type system).

Basically closures can be managed like objects, and the non-GCed C dialects use objects/structs routinely.

I've never had a big problem with freeing memory in C. Of course making closures more convenient will mean that there's more memory to manage. But something like unwind-protect in Lisp, or Region-Based memory management could make this really manageable, IMHO.

Of course for many problems, a language with GC like Lisp is more appropriate. You have to choose when to use it, and when to use the lower-level language. For stuff like file systems or databases or OS kernels, I think a more powerful language than C would result in more sanity, less duplicated features (that look different, because the abstraction level is too low) and more stability and performance.

yea, but..

[jnf]

whats the performance hit of immediately free'ing everything instead of sticking the free'd memory into bins? Of having to get the memory from the machine everytime with no caching bins/etc? How many wasted pages of memory are there for so called guard pages?
It's not so much that they are cutting edge, its more that other OS's will sacrifice some security for speed and tell the programmers 'program correctly!'.
or at least thats my take.

Re:yea, but..

[mrogers]

How many wasted pages of memory are there for so called guard pages?

AFAIK guard pages don't waste any physical memory - a guard page is a virtual memory page with no corresponding entry in the page table, so a page fault occurs when the guard page is accessed.

Re:yea, but..

[jnf]

Ah yes, that wouldn't surprise me-- okay strike that comment from the record ;]

The reason most malloc implementations do what they do is in the name of speed, and in fact I seem to remember when I first read about this that this has been Theo's pet project for quite some time and they haven't pushed it because performance was so incredibly bad. Really its a catch-22 situation, and I don't think this will fix it, but I suppose if you value security over speed, then this is right up your alley-- although if you are a person like that, you should probably be auditing the code you run.

At any rate, good call, thanks for the more correct information.

Re:This is why I couldn't use OpenBSD exclusively.

[Nimrangul]

But they don't care, they're not trying to be FreeBSD or a Linux distribution, they're trying to be OpenBSD and a part of that is not letting people's perception of optimum performance get in the way of doing what is right by them.

You gotta remember, the project doesn't do it for outsiders, what they do is for themselves. They want security and are willing to pay performance and ease of use to get it, it's like a mantra for them, never take the path of least resistance.

If this looses like 5 or 10 percent of it's performance on my machines I won't mind, it's another layer of protection and I like having it and am fine with the cost, faster hardware isn't that expensive. If something I run crashes, I will report to the people that wrote it, telling them that I found a problem that was found by OpenBSD's malloc, maybe they'll even devote an old test box to checking for bugs on it.

If OpenBSD was trying to be a Linux distribution then we'd not have most of the good stuff that makes OpenBSD unique.

App specific security policy

[CustomDesigned]

Microsoft should have handled broken apps like SysV did with null pointer derefence. Make the default policy to do immediate free. But provide a flag, settable by application, that can disable immediate free for that app only. Require the users to add a registry entry to work around the broken app. Be sure to name the registry key something that makes it clear to an end user that the app is broken, and the OS is just working around the problem.

Re:App specific security policy

[Hal_Porter]

Actually freed blocks go on a per application list, and on Win2k an up they go at the top of the list so they will be allocated next time you malloc

http://msdn.microsoft.com/library/default.asp?url= /library/en-us/apcompat/apcompat/do_not_read_or_wr ite_from_freed_memory_blocks.asp

This exposed bugs where applications touch a freed block, so they came up with app verifier

http://msdn.microsoft.com/library/default.asp?url= /library/en-us/dnappcom/html/AppVerifier.asp

This marks all the pages on the per app free list as invalid, so you are guaranteed to get clobbered if you touch them. I'd bet there is a AppCompatibility flag in the registy which does the opposite somewhere, i.e. it keeps freed blocks valid for longer (for ever?). That way, if they find some 'commercially important' application would break with the new heap allocation algorithm when they test it, they can set the flag and no one will see a problem. Except the company who wrote it, who presumably get an error report.

Re:Unnecessary when using languages that solve thi

[Clover_Kicker]

You're totally right, dude.

Let me know when you release your Haskell version of Sendmail, and I'll switch over immediately.

Re:This is why I couldn't use OpenBSD exclusively.

[Anonymous Coward]

Big apps still work after this change. And we've had native Java for a while now.

oneupsmanship

[Anonymous Coward]

I'm re-writing it in Perl and Java.

Re:Sacrfice useability? Nice idea , won't work.

[DrSkwid]

We OpenBSD users *already* sacrifice heaps of usability.

I --force it on my team and they live with it.

CPU is cheap, downtime is expensive and usually really annoying.

It Is Official

[Anonymous Coward]

It is official; NetCraft has confirmed: OpenBSD 3.7 has been hacked by a rogue internet group less than 24 hours after release.

"We can't believe how easy this one was to crack. There are 3 exploits you can do over the internet right out of the box, and I think we're going to find more," said ZeroC00L, a leader of the X0r h@X0rs. The group claims responsibility for demonstrating exploits in the past 5 OpenBSD releases.

"I think the main reason that people think OpenBSD is 'secure' is because Theo [de Raadt, leader of the OpenBSD 'project'] says it is. The truth is about the opposite; we can't find a single exploit in the latest RedHat, but OpenBSD is OpenSwissCheese. All that crap legacy code from fucking Berkeley hippies, you know."

Theo de Raadt has apparently refused to comment.

Re:It Is Official

[Anonymous Coward]

Theo de Raadt has apparently refused to comment.

If Theo isn't responding then it obviously means he has something to hide. Zerocool is probably telling the truth.

Whatever happened to segmentation?

[Sloppy]

You know, the memory model that the 8086 did stupidly and the 80286 did ok, which everyone put behind them when the 80386 gave us paging and a "flat model" that consisted of more address space (4GB) than anyone would ever need. You know, the one that was totally a pain in the ass to work with on platforms like MS-DOS. You know, the memory model everyone called "brain dead."

It is now very clear that everone wants it. You allocate a segment, tell it exactly how many bytes long it needs to be, and any overrun or underrun causes an MMU exception. No sentinels to check, no waiting for an overrun to reach a page boundary.

Just something to think about, even though it's not portable to other archs. i386 can still do it. Does x86-64 have it?

Re:Whatever happened to segmentation?

[Sloppy]

Holy crap, if my 1987 self could hear what I'm saying in 2005... "You senile old bastard! I'll fucking kill you for this!" would probably be his first reaction.

2005 self would counter with, "Yeah, the pointers will be bigger than they used to be, but you progam in high-level languages now, so you don't ever worry about that. It's the compiler's problem."

1987 Sloppy would say, "But I'm going go write a compiler!"

2005 Sloppy would say, "You fuckwit, you never got anywhere on that project. You barely even started it. Too much time fucking around with graphics and genetics."

1987 Sloppy would say, "But, but, it's not fair! Segmentation is an x86 thing. Everyone knows that in the future, we'll all be using 68k. 68k doesn't do segmentation."

2005 Sloppy would sigh.

1987 Sloppy would say, "Oh come on. There's no way people are still using x86 in the 21st century, or even in the 1990s. No fucking way."

2005 Sloppy would just shrug. There's nothing to do in a situation like this. There's nothing you can say. They'll never believe you.

Re:Whatever happened to segmentation?

[Duhavid]

I could be misremembering, but...

A: You didnt allocate a segment, segmentation was a hack to allow access to more memory without having the register size change to *really* support it. Two registers were used to point to a given memory location, one was offset ( shifted left ) by ( I think ) 4 bit positions so that you got a 20 bit address from 16 bit registers. Any combinations in the two registers that resulted in a given address were OK to use to get that address.

B: I dont think the MMU or any other feature gave you any overrun or underrun exceptions on violations. And the segment size was 64k IIRC.

Re:Whatever happened to segmentation?

[AJWM]

I don't remember x86 segmentation working this way.

You seem to be describing how Burroughs Large Systems (B[567][5789]00 models -- and possibly the current A series, though I'm not very familiar with those) handled memory. Yeah, segments were any size you want, accessed through a descriptor (which had hardware-recognizable tag bits) which described the position and length of the segment. Try to access out of those bounds and the hardware wouldn't let you. The descriptor also carried info as to whether the segment was executable or not, etc.

Re:Whatever happened to segmentation?

[Sloppy]

What you're describing (in both points A and B) is how the 8086/8088/80186 worked. I'm talking about the 80286 and later, which supported true segmentation, with an MMu and everything. (Really! I'm not making this up. :-)

In fact, that's how OS/2 version 1.x handled its virtual memory. It did its swapping by segments instead of by pages (written for the '286, it didn't even know what a page is).

Re:Whatever happened to segmentation?

[Sloppy]

Yeah! That's what I mean. 286 "protected mode" works that way (it was the only way to implement virtual memory on that chip) and the 386 can do it even better. Everyone just uses paging on the 386 nowdays, but the hardware can do segmenting too. (And apparently I'm the only person on Slashdot that remembers this legacy. Oh well. I guess "1987 Sloppy" would be pleased.)

Re:Whatever happened to segmentation?

[Duhavid]

Please excuse my ignorance!

(-: unfortunately, I have a lot of it... :-)

Re:Whatever happened to segmentation?

[Anonymous Coward]

The problem with segmentation (on the 386) is twofold: Descriptors, and the number of descriptors.

  Since descriptors were implemented in hardware to be fast, there were limitations on them, Specifically, you couldn't get more than 8192 global descriptors or 8192 "local" descriptors, where "local" was usually tied to a process. The problem then became one of granularity. Too many algorithms require individual instances of structures to be allocated with only a pointer reference to them, so one would be limited by the number of descriptors, or one wouldn't use descriptors at all. I went back and forth about descriptors in my childhood playing around with my 486 (I must be younger than you), but decided that it really wouldn't be of much use outside of toy systems, or would require very careful programming to efficiently use descriptors, and even then large arrays would have to live inside a single descriptor, which would mean that intra-member corruption could still occur. It's possible that if the 386 had extended the number of segment descriptors past the 16-bit boundary, they would have been useful, but that would require redoing all the segment descriptors, which would require redoing the opcodes for all the far jump/call instructions and some stack stuff, and more importantly would have broken backward compatibility.

The other problem with descriptors is that you only had 4 segment registers to hold them (2 on the 286). That really limited what you could do, because generally algorithms pull from more than 4 places at once, and loading descriptors from memory into the segment registers was slow.

Back then I decided that the only real option was just to formally prove everything at the machine code level. Look for "proof carrying code" on the Google and note that it's essentially at the toy problem stage right now, but it's probably the only route to go for very secure systems that also need to be fast and reasonably small. Arbitrary packet filtering is one of the examples they do with PCC, and prove both a time bound and memory safety for the packet filter, which gets checked by the OS before the code runs. The problem is getting everyone to write proofs for their code, or writing compilers that can do it without too much additional proof generation in the high level language. It's a lot easier with functional languages, of course, but I think functional languages are greeted about the same way as formal proofs are by too many people.

Re:Whatever happened to segmentation?

[Eli Gottlieb]

I remember too, because I've recently written operating system code that put me in segmentation/paging-combination Hell!

Truth be told, though, segmentation is a brilliant scheme for designating how memory regions should be protected, and an elegant segmentation scheme on top of paging hardware (to let you mismatch virtual/physical addresses but still have the same logical address with segmentation used) is a beautiful thing.

NOW IF ONLY MORE HARDWARE WOULD SUPPORT IT!

Just Wondering

[skubeedooo]

When program X exits, does it's deallocated heap memory become available to a lucky program who asks for some space and gets given the same address? Does does the OS overwrite it before passing it on to the next program or is this the task of the writer of program X?

Re:Just Wondering

[Intron]

According to the C standard, malloc does not initialize the memory it gives you. Its program X's responsibility to clear confidential info before calling free (or exiting - same thing). A program that doesn't care should not be penalized. Its pretty much like you have to shred confidential info before throwing it in the trash.

Re:Just Wondering

[Anonymous Coward]

yes it the task of the program X. Program X must memset all memory that
contain sensitive data. and optionaly prevent that block to be swaped out.
this is done into programs that get password input for exemple.

-Bob

Re:Just Wondering

[OttoM]

Pages that are returned to the system are cleared before handed out to another program.

Still, it is a good think to clear confidential data when done with it. Because if the page gets reused within the same program, the data may leak.

Swapping isn't a problem on OpenBSD since the swap space is encrypted by default.

Re:This is great. Just super

[ckaminski]

Methinks you doth misseth the point:

<quote>
Some platforms just don't support the fork() mechanism...
</end-quote>

And multitasking with processes sucks when fork() is not available. 90% of OSes treat processes as threads these days (if they support threading, that is) and provide memory protection to processes. Why cannot they do the same on a threadly basis? Hmm?

Threads aren't the end-all be-all of multi-processing, but they sure are useful.

Some Emacses are written in LISP

[billstewart]

There are a variety of Emacs versions and imitators out there, and most of them are either written in some version of LISP or else interpret some version of LISP for many of their library functions. Usually it's Emacs's ELISP as opposed to Common Lisp or other major dialects.

Perhaps there aren't other popular counterexamples, but there's at least that one. I do remember the old Winterp XLISP-based window package - one of my friends used it for fast prototyping, and found that the prototypes usually had better performance than the evenual production code, as well as often having more features...

Re:This is why I couldn't use OpenBSD exclusively.

[TheRaven64]

I don't think OpenBSD breaks things in the name of security. I think OpenBSD allows things that are already broken to crash rather than being compromised. And this is why I do use OpenBSD - it's a lot easier to restart a program than clean up a compromised machine.

Re:Unnecessary when using languages that solve thi

[Bob The Cowboy]

It always surprises me when something like this gets modded up on /.

This I believe is one of the key differences between programmers and computer scientists. I would simply love to see something with the complexity of sshd or sendmail written in Haskell, Lisp, Scheme, or whatever the current interpreted language is en vogue. Seriously, show me the code! Benchmark some enterprise-level (as in, used in businesses, not the buzzword) non-trivial app (like the aforementioned) written in C/C++ compared to it's interpreted counterpart. I would be incredibly surprised if Apache, Mozilla, Word, whatever would run with anywhere near the speed or responsiveness.

Your opinion (which you are of course entitled to) ignores things like Operating Systems, embedded devices, and even the runtimes/interpreters/virtual machines themselves. And then who is going to maintain this code? Haskell and friends don't have anywhere near the mindshare that C/C++/Java have. You're right.. in theory, using C is terrible, but in reality the other options are worse for different reasons.

Bill

Re:Unnecessary when using languages that solve thi

[pyrrhonist]

At random unpredictable and uncontrollable intervals the whole program would stop dead while the JVM did garbage collection.

This indicates to me that you have a memory leak in your application.

I helped write a telephony server in Java, which had to be both long-running and fast. Anytime the JVM paused, it was always due to a memory leak. When we first encountered it, we weren't sure what it was either because the symptom was that we were dropping telephone calls due to unhandled events (the events would get to the queue, but the event loop wouldn't handle them due to gc kicking in, and then the ISUP timers would expire causing a missed call). We originally thought the problem was with the event loop, and we steadily improved it. However, one day, we finally ran out of memory after hours of processing calls (nobody was around to witness the pauses, though). It turns out, nobody (including me) had bothered to look at the memory size for the application (typical Java myopia), which was steadily growing as the application ran. We immediately ran the basic Java heap analysis tools and found out that we were misusing the heap something fierce. We were able to reproduce the problem quickly by limiting the heap size, and we noticed the JVM pausing as the gc struggled to free up some memory. After that, we ordered OptimizeIt, and I spent a lot of time tracking down exactly which objects weren't being deallocated properly.

Before we checked for leaks, our server would run for days until it eventually ran out of memory. After we found all the leaks, our server would run forever with the minimum allowed heap size, and we never had any gc pauses after that.

Re:This is great. Just super

[Paul Rose]

Methinks you doth misseth the point:

That has been known to happen. I don't think so in this case, but I'll will admit I was overly flip about it.

The issue is that memory protection is nice, but in some OS's (like NT and OS/X) launching a new process is more expenisve than launching a new thread.

But Isn't one of the things that makes threads cheaper than processes on these systems the fact that you don't have to set up as much VM stuff when launching a new thread? If so wouldn't threads become more expensive?

No it wouldn't

[crush]

I can't read what's in front of me. Apologies.

Re:This is great. Just super

[ckaminski]

It's certainly nothing I'd want enabled on a system wide basis. I mean, the simple process of walking a bitmap or list of thread handles could increase the time required to touch/access a page O(1), which would suck in a system of lots of threads (I've only seen one design that actually was better off with 600 threads than 60 threads with 10 WFMO/select() handles per).

I guess what I want is a fast fork() on WIN32. :-/ AFAIK, it's the only commonly available OS at the moment without one. Java doesn't quite have that feature either, and with Java, the usefulness of such a feature (thread protected memory) is open to debate.

Steel-Toed Boots and Shooting Your Foot

[billstewart]

C: You Shoot Yourself In The Foot

C is still my favorite practical programming language. It's lightweight, elegant, terse, does exactly what you tell it, doesn't hide the machine from you, and people should stop using it for significant projects because too many people can't use it safely 100% of the time. Not only do they shoot themselves in the foot, but they're getting blood all over the floor and other people are slipping on it.

All of these mechanisms are providing steel-toed boots so that usually when you shoot yourself in the foot, the bullets ricochet harmlessly, making enough noise to wake you up, though sometimes the bullet goes through and nails your boot to the floor, but in neither case did you hit the target you actually wanted. It's there to reduce collateral damage, and to make programs crash loudly enough that you can debug them.

Re:Steel-Toed Boots and Shooting Your Foot

[Eli Gottlieb]

Pascal: The Compiler Won't Let You Shoot Yourself in the Foot

And that is why I use FPC.

Rob Pike on Threads

[billstewart]

Rob Pike (or maybe Ken Thompson} once gave a Plan 9 talk where he said that threads are for operating systems that can't do fork() very well...

Slower and 0% effective

[Fujisawa Sensei]

The solution is 0% effective because 386 used SIMMS not DIMMS. :P

immediate free is new?

[Anonymous Coward]

I made a plugin to http://fidolook.org/ in Borland Delphi.
To make DLL tiny i removed most of Borland RTL, including their heap manager - and replaced it with WinAPI calls. And it crashed. It appeared that the host app freed buffer before using it. And WinAPI (indeed there are 3 APIs, i can't remember which certain was used) wiped out at least the very beginning of the buffer. It was somewhen around Win2k sp2 if i remember right.

Re:This is why I couldn't use OpenBSD exclusively.

[ArbitraryConstant]

"it's a lot easier to restart a program than clean up a compromised machine."

That depends on what it is. If it crashes in such a way that the program isn't usable, then OpenBSD simply isn't an option.

Re:This is why I couldn't use OpenBSD exclusively.

[Anonymous Coward]

But it's not 5-10% is it? It's orders of magnitude for some things. They haven't even fixed the double-halt bug in STABLE that cuts I/O performance in half in some cases.

Gimme a break... hlt hlt was fixed in both OpenBSD 3.7 and 3.6 stable branches.

Re:This is why I couldn't use OpenBSD exclusively.

[Anonymous Coward]

But it's not 5-10% is it? It's orders of magnitude for some things.

Yet users will take a hit by running a virus scanner without complaint because they're used to the slowdown.

Re:Unnecessary when using languages that solve thi

[Todd Knarr]

No, no memory leak, it just did a lot of object creation and destruction, which meant a lot of garbage to be cleaned up when the GC finally triggered. We thought it might be a leak, but heap analysis showed nothing was allocated after GC that shouldn't have been. A typical GC pass would free up about 1gig of memory, which was about right for the amount of object creation that'd been done. The behavior's the big strength of garbage-collected systems, and their big weakness (depending on how your program works).

Re:This is why I couldn't use OpenBSD exclusively.

[ArbitraryConstant]

Ah, sorry. Didn't make the errata page.

Re:This is why I couldn't use OpenBSD exclusively.

[ArbitraryConstant]

"Yet users will take a hit by running a virus scanner without complaint because they're used to the slowdown."

Sure. Windows users.

Re:This is why I couldn't use OpenBSD exclusively.

[Anonymous Coward]

Yup, most users.

Just pointing out that unless a performance difference is glaring and sudden people tend not to mind or even notice. And I've seen crap Unix software and drivers too that users accepted as normal.

Re:This is why I couldn't use OpenBSD exclusively.

[ArbitraryConstant]

"Just pointing out that unless a performance difference is glaring and sudden people tend not to mind or even notice. And I've seen crap Unix software and drivers too that users accepted as normal."

That depends on what you're doing. If it were an across-the-board 10% I probably wouldn't care, but it's not.

Linux 2.6 kernels are absoloutely outstanding at responsiveness, the only comparable experience I've had was with BeOS. It's pretty easy to tell whether or not (for example) your music skips under high load. Frankly, I've gotten used to being able to throw almost anything at my UP Linux machine without GUI or music being impacted perceptibly. Provided I'm not swapping, I haven't seen a slowdown under any load.

Nothing else out there at the moment seems to be able to do it. Windows can't do it, Mac can't do it, FreeBSD can't do it (even with ULE...), and OpenBSD certainly can't do it (my SSH sessions to my OpenBSD box get laggy when it's under high load).

I will probably get modded as flamebait..

[jnf]

But what is new about this method exactly? From what I read the guardpages are not truly guardpages, but more of a side product of having gaps in the addresses. And whats new about immediately free'ing memory and returning it? Isn't that the way things were originally done until everyone realized just how poor on performance it was?

This is kinda MS innovation at play here.

VBLinux - Victoria Bitter

[sr180]

There is a beer in australia called VB. I believe it has the same quality that your VBLinux is going to have.

Re:Unnecessary when using languages that solve thi

[Anonymous Coward]

http://postmaster.cryp.to/

Re:Unnecessary when using languages that solve thi

[Clover_Kicker]

Wow, who knew :)

Although I'd probably describe it as an MTA construction kit rather than a drop-in sendmail replacement.

could you explain how please?

[YesIAmAScript]

TFA doesn't explain what will be done very well. And what it says is consistent with the first of the two explanations I gave. "Guard pages". Perhaps they are unreadable as well as unwriteable? That refinement doesn't change the problems with either of my two methods.

Could you explain perhaps how I am wrong here? I'm thirsty for information.

Solaris has had this for YEARS

[sethmeisterg]

Check out libumem -- much more powerful than this and has been around in Solaris for years (granted, the manpage should be better to reveal the powerful features -- but a quick look at the source at opensolaris.org reveals how to use the most advanced features).

Re:cutting edge?

[Anonymous Coward]

I have a hardened penis.

Re:This is why I couldn't use OpenBSD exclusively.

[ArbitraryConstant]

1.4 was unsupported when I gave up on OpenBSD as my dev workstation. That's spiffy, it'll help to have that available on the OpenBSD machine. Thanks.

Re:This is why I couldn't use OpenBSD exclusively.

[Anonymous Coward]

I haven't checked because Java is poorly supported for unrelated reasons, and this rules out OpenBSD without me having to validate all my other software.

Actually, I have Eclipse running and I started JBoss right from the installation the other day. JRE1.5 is in 3.8 so I wouldn't say OpenBSD has poor support for Java.

Is it as fast as it can be? Probably not, but then again, Java on Windows isn't blazingly fast either...

Juste a reminder COW may also be a loss

[renoX]

I think that you know it, but COW can sometimes be a loss: if you happen to copy the content of the memory, doing it a page at a time instead of all the page at once may incurr a loss.

So COW can be a big win if no copy occurs, a small win if some copy occur but also a small loss if many copy occur.

Re:Juste a reminder COW may also be a loss

[Krach42]

Yeah, exactly. You're basicly driven by trade-offs. As a programmer you always need to be asking, "What am I losing by going to this? What are my inputs, and initial state going to be?" etc.

When Linux and other free *nixes switched to CoW for program spawning, they new that it would slow down fork()s that would continue executing. But they also knew that the most common thing after a fork() was execv(), which would wipe out their memory map anyways. Thus, over all CoW for program spawning is a win.

But if you have a program load up a lot of data, then fork() and do its process on the two sets of data, then fork() and so on to maximize CoW losses, then you're definitely worse off.

Actually, I worked this into a program that I wrote that was designed to cripple a computer. Basically, it allocated a huge chunk of memory did a while(fork()); (which would be a standard fork bomb, but computers are capable of dealing with these now) then after that, it would do reads randomly from the huge memory buffer, and then a write randomly into the huge memory buffer. This not only destroys cache performance, but also attacks the Virtual Memory system, and also increases CoW losses.

That program can take out a great many computers such to the point that they're so busy dealing with page faults that they can't even handle keyboard interrupts in a reasonable time frame. My friend had an OpenBSD system, which is where this program took on its biggest teeth, and at one point he had high double digit, or triple digit load avg. Everytime he came out with something to take teeth off of my program, I added in something else.

Eventually, he had to lock down the computer to the point that I could barely even run gcc, and at that point, it actually was able to cope with my program reasonably well.

dumb question re: malloc

[cyclomedia]

um, surely all os's should do this:

1. prog calls malloc (or equivalent)
2. os hands memory to prog
3. prog tries to read or write out of the range it was given
4. BANG!

seems a bit bloody simple to me.

unless it's something to do with the bios/hardware just dumbly doing what it's asked by the code. which is just as stupid as a prog requesting a handle to the File System and magically getting read/write access to every file everywhere instead of just it's local folder and "c:/my documents"

JITs need write and execute simultaneously

[p3d0]

You can't write a high-performing JIT compiler with pages using memory that's not writable and executable at the same time. The dynamic nature of Java necessitates continual patching of the generated code for a number of reasons. Even if there were no page-boundary problem (ie. making one page unexecutable could affect any number of methods) and no threading problem (ie. making a page unexecutable would affect every thread executing code on that page) it would still be infeasible to do all those extra system calls every time the generated code needs modifying.

Extrapolating from a sample of one?

[p3d0]

Nice dream , meanwhile in the real world both users and most coders (if they dare to admit it) will NOT sacrfice usability or ease of coding for security measures that (in their minds) are nothing to do with their application.

Really? And I guess you have asked "most coders" for their opinion on this?

Re:Unnecessary when using languages that solve thi

[bluGill]

What is the point in using a garbage collected language if you can still get memory leaks? If I have to run my programs through valgrind or the like to find leaks, why don't I just write them in C++ to start with?

I know there are other advantages to Java, C++ has some large ones too, and if you use the STL and RAII like you should, you end up simulating the advantages of java. (Plus you don't have to worry about the JVM license)

Re:Unnecessary when using languages that solve thi

[pyrrhonist]

What is the point in using a garbage collected language if you can still get memory leaks? If I have to run my programs through valgrind or the like to find leaks, why don't I just write them in C++ to start with?

You can't have memory leaks in the classical sense (i.e you cannot allocate memory directly in Java). We were actually talking about object leaks, which you can also have in C++, Perl, Python, Ruby, and pretty much any other language that has objects. The problem illustrated above has to do with figuring out exactly how long your objects should live. In a program that is long-lived and processes a lot of data very quickly, it's very important to break your references as soon as possible or dilligently reuse objects, because you will run out of memory fast. Regardless of the language, the heap analysis software is used in this case to pinpoint which objects can be deallocated sooner and references that were never broken.

I know there are other advantages to Java, C++ has some large ones too, and if you use the STL and RAII like you should, you end up simulating the advantages of java.

As far as resource (e.g. socket handle) leaks go, you'd be better off in C++, because there is no way for Java to do RAII (i.e. no explicit destructors). Java does have container classes, iterators, and parametric types, however. In C++ you will also need Boost, OpenSSL, and Xerces to simulate the rest of Java's standard API.

(Plus you don't have to worry about the JVM license)

The only license that I've ever worried about when doing commercial software development is the GPL.