As an aside: the default payment level for AutoMan is US minimum wage, and there is no built-in provision for differentiating wages based on the country of the worker.
Relevant - a related but different approach: AutoMan, a language for programming with people: http://www.automan-lang.org/
AutoMan is a platform for integrating human-based and digital computation. It allows programmers to "program with people", which appear to the programmer to be ordinary function calls. AutoMan automatically handles details like quality control, payment, and task scheduling. It is currently implemented as a domain-specific language embedded in Scala (a language that runs on any machine with a Java Virtual Machine), and uses Amazon's Mechanical Turk as a backend.
You might be interested in the other best paper award winner (in the shameless self-promotion department): TFS: A Transparent File System for Contributory Storage, by Jim Cipar, Mark Corner, and Emery Berger (Dept. of Computer Science, University of Massachusetts Amherst). Briefly, it describes how you can make all the empty space on your disk available for others to use, without affecting your own use of the disk (no performance impact, and you can still use the space if you need it).
Enjoy!
--
Emery Berger
Dept. of Computer Science
University of Massachusetts Amherst
If you encounter a problem using DieHard, I'd appreciate hearing some details (send me mail). Obviously, the system works fine for me and others. I'm using it now, and have been doing so for months without incident.
No, that's what OpenBSD does. Sorry for any ambiguity -- that whole paragraph was about OpenBSD.
-- Emery Berger
Re:Vista already doing some of this
on
DieHard, the Software
·
· Score: 5, Informative
Hi,
Here's a more detailed answer -- I'll add it to the FAQ.
OpenBSD (a variant of PHKmalloc) does some of what DieHard's allocator does, but DieHard does much more. On the security side, DieHard adds much more "entropy"; on the reliability side, it mathematically reduces the risk that a programmer bug will have any impact on program execution.
OpenBSD randomly locates pages of memory and allocates small objects from these pages. It improves security by avoiding the effect of certain errors. Like DieHard, it is resilient to double and invalid frees. It places guard pages around large chunks and frees such large chunks back to the OS (causing later references through dangling pointers to fail unless the chunk is reused). It attempts to block some buffer overflows by using page protection. Finally, it shuffles some allocated objects around on a page, randomizing their location within a page.
DieHard goes much further. First, it completely segregates heap metadata from the heap, making heap corruption (and hijack attacks) nearly impossible. On OpenBSD, a large-enough underflow on OpenBSD can overwrite the page directory or local page info struct (at the beginning of each page), hijacking the allocator. This presentation describes several ways OpenBSD's allocator can be attacked. By contrast, none of DieHard's metadata is located in the allocated object space.
Second, DieHard randomizes the placement of objects across the entire heap. This has numerous advantages. On the security side, it makes brute-force attempts to locate adjacent objects nearly impossible -- in OpenBSD, knowing the allocation sequence determines which pages objects will land on (see the presentation pointed to above).
DieHard's complete randomization is key to provably avoiding a range of errors with high probability. It reduces the worst-case odds that a buffer overflow has any impact to 50%. The actual likelihood is even lower when the heap is not full. DieHard also avoids dangling pointer errors with very high probability (e.g., 99.999%), making it nearly impervious to such mistakes. You can read our PLDI paper for more details and formulae.
-- Emery Berger
Re:Vista already doing some of this
on
DieHard, the Software
·
· Score: 5, Informative
Hi Slashdot readers,
DieHard's randomization is very different from what OpenBSD does, not to mention Vista's address-space randomization. I've added a note to the FAQs that explains the difference in some detail, and answers several other questions, but in short: "address-space randomization" randomizes the base address of the heap and also mmapped-chunks of memory, leaving the relative position of objects intact. By contrast, DieHard randomizes the location of every single object across the entire heap. It also goes further in that it prevents a wide range of memory errors automatically, like double frees and illegal frees, and effectively eliminates heap corruption.
Here's a transactional file system (& more) developed at UMass Amherst that sounds like it might fit your needs. I've boldfaced the key line.
Recent gains in energy-efficiency of new-generation NAND flash storage have strengthened the case for in-network storage by data-centric sensor network applications. We argue that a simple file system abstraction is inadequate for realizing the full benefits of high-capacity low-power NAND flash storage in data-centric applications. Instead we advocate a rich object storage abstraction to support flexible use of the storage system for a variety of application needs and one that is specifically optimized for memory and energy-constrained sensor platforms. We propose Capsule, an energy-optimized log-structured object storage system for flash memories that enables sensor applications to exploit storage resources in amultitude of ways.
Capsule features:
Capsule provides the abstraction of typed storage objects to applications; supported object types include streams, indexes, stacks and queues. A novel aspect of Capsule is that it also allows composition of objects for instance, a stream and index object can be composed to archive sensed data. The objects expose a data structure-like interface, allowing applications to easily manipulate data.
Capsule seamlessly supports storage on a multitude of platforms (both NAND and NOR flashes) with the help of a hardware abstraction layer; the same application can be used on any supported platform. Currently we support storage on the following platforms: Mica2, MicaZ, Mica2 Dot, Telos and our custom NAND flash board. However, the use of an abstraction layer allows supporting a new platform a breeze.
Capsule supports checkpointing and rollback of storage objects, allowing the application to tolerate software faults and device failures. The application state is automatically restored when the device restarts.
We provide our own file system implementation on Capsule that allows applications using Matchbox and ELF to be ported to Capsule.
Flash offers a finite storage medium; once it fills up, free space needs to be created before any further data can be stored on it. Basic Capsule objects perform memory reclamation by supporting compaction.
Click here for the Capsule site. It's been deployed on a number of very small platforms (including turtles (!)).
-- Emery Berger, Dept. of Computer Science, UMass Amherst
Some idiot modded the above as a troll, which is way off. But I'll elaborate anyway.
I wrote an application that performs a number of Monte Carlo simulations (if you're interested in why, see DieHard for some idea). Anyway, I chose to write it in Python. The simulator makes intensive use of associative arrays and random number generation. The latter (RNG) is from the standard library, and as for the former (assoc arrays), one would think these would have to be implemented in more or less the same way both in CPython and Iron Python.
Nonetheless, the program runs 1.77 times slower in Iron Python than in CPython! That's not good. It suggests that there is some deep inefficiency either in the interpreter, or in the CLR, or both. Worse, when I use psyco, I get a further speedup of 3x over the original. Where does that leave us? CPython + psyco = roughly 6x faster than Iron Python.
To me, this result substantially undermines the claim that Iron Python is competitve with CPython. A nearly 2X slowdown is not competitive. Moreover, anyone looking to achieve higher performance is going to be using something like psyco, and then we're talking about potentially taking a 6X hit when using Iron Python.
While I'm sure that the.NET framework adds plenty of interesting functionality for some people, the claim that its performance is competitive strikes me as disingenuous.
But perhaps I'm just trolling...
-- Emery Berger
University of Massachusetts Amherst
I ran it on one of my Python apps and it actually slowed it down by a factor of 1.77. Lots of caveats here, but this is the opposite of what I expected to see.
We recently developed a programming language system called Flux that we presented at USENIX this June that addresses exactly this problem. Flux is a domain-specific language for building server applications from serial C and C++ components. The compiler then generates event-driven or multithreaded code -- it's just a compiler flag and a different runtime system. You as a programmer do not deal with the nitty-gritty details of managing concurrency. Moreover, Flux can optionally generate a simulator that lets you load-test your app before deployment in order to isolate bottlenecks. We've built four servers so far in Flux - including the web server hosting Flux, and a BitTorrent seed - and its performance matches or exceeds that of hand-tuned high-performance servers. It's still a prototype system, but worth checking out.
-- Emery Berger
-- Assistant Professor
-- Department of Computer Science
-- University of Massachusetts Amherst
If what you want is stability, you must replace the memory manager. This requires no change to your code; there are several plug-in replacements. You can use a conservative garbage collector to get rid of memory leaks. Another possibility -- in the self-plug department -- is to use DieHard, which eliminates a wide range of memory errors and even defends against heap overflows and other heap corruption. Pick whichever works best for your needs, but pick one. You will not get stability and reliability with C++ if you stick with the default memory manager.
You may find DieHard useful, if heap errors are also a problem -- it's a plug-in replacement for malloc/free that provides probabilistic guarantees of correct execution in the face of errors.
Thanks for your comments. I agree that the core result -- GC requires a substantial amount of space to reduce the frequency of collections, otherwise performance suffers -- shouldn't be surprising, but the existence of articles like the IBM one suggests otherwise...
I do have a few comments about your comments, though. First, I don't understand your objection to the insertion of frees. This just simulates the effect of the programmer invoking free. We examine two cases: (1) where the free calls are as early as possible -- the last use of an object and (2) where the calls are as late as possible -- the last time the objects are reachable. This brackets the range of programmer behavior. We measure both cases. Both choices of when to free objects turn out to have surprisingly similar performance results, and in any event, all of the garbage collectors require a lot of space to reduce GC frequency (roughly 5X before runtime performance matches malloc/free).
As for the tools used -- you work with the best you have available. That said, this is a pretty comprehensive study on a reasonably solid platform, though all the collectors are stop-the-world (all threads stop during collection). Concurrent GC is slower than non-concurrent GC, so including concurrent collectors wouldn't alter our results. However, parallel stop-the-world collectors would speed tracing (computing the root set is not the problem). We couldn't test such collectors because of infrastructural limitations, but we definitely want to revisit them later.
One note: you shouldn't read our study to suggest anything about compile-time garbage collection beyond the fact that it basically changes the time-space tradeoff (it won't make programs faster when running in plentiful RAM). The oracles used to guide free calls are derived from profiling, not from static analysis. Escape analysis is great for reducing contention in the presence of multiple threads, but it's likely to be of limited value when you have generational collectors (insight: it only captures short-lived objects, precisely what generational collectors already handle superbly).
Emery Berger
Asst. Professor, Dept. of Computer Science
University of Massachusetts Amherst
While I find it difficult to argue with people who disagree with GC's software engineering benefits - which seem obvious to me - I have to say that I think the author of this article is missing the point. GC might well squeeze a few less instructions out of the allocation sequence, though this is not necessarily true for some well-tuned allocators.
But the allocation sequence isn't the bottleneck --
it's all about the actual garbage collection itself. GC can be costly in terms of cycles and, if you're unfortunate enough to be running in low physical memory, page faults. The reorganization of memory may improve application locality, but that may not be enough to buy back the performance penalty.
Rather than try to argue the case with hand-waving, my student Matthew Hertz and I performed experiments on an infrastructure that lets us replace garbage collection with malloc/free in Java programs so we can quantitiatively compare the cost of garbage collection. Our paper, to appear in OOPSLA 2005, shows that a good, generational copying garbage collector can match or slightly beat the performance of malloc/free -- but this requires 5X as much memory as the allocator. It's a classic space-time tradeoff. On the other hand, if you don't have enough RAM to hold the entire heap, though, garbage collection runs tens or hundreds of times slower.
-- Emery Berger
Asst. Professor, Dept. of Computer Science
University of Massachusetts Amherst
Pools have problems (was Re:apr_pool_t)
on
APR 1.0.0 Goes Gold
·
· Score: 5, Interesting
Ugh. Although pools are plenty useful, they have lots of problems. The inability to free individual objects can be a real deal-breaker. Since you can't free objects within a pool until all of them are no longer in use, dead memory can just accumulate. For lots of reasonable situations, this means unbounded memory consumption. It also makes it very hard to incorporate existing code that uses malloc/free.
I wrote a paper about this, and problems with custom (special-purpose) memory allocators in general, called
Reconsidering Custom Memory Allocation (OOPSLA 2002). In it, I also describe a new memory allocation scheme, called reaps. This is a hybrid of regions (pools) and heaps that acts just like pools except you can still free individual objects.
In fact, for a case study, I put reaps into Apache (adding a ap_pfree call), and showed how using reaps made it simple to incorporate a piece of existing C code (namely, bc) into Apache. Without reaps, an invocation consumed 7.4MB of memory (since every free had to be disabled). With reaps, 240KB.
I did send a message to the Apache folks about this a while back, but they balked because the implementation is in C++, rather than C (developed with my Heap Layers infrastructure)...
-- emery
Emery Berger
Assistant Professor
Dept. of Computer Science
University of Massachusetts, Amherst
You make one horrible suggestion and one great one. The great one is to address allocation performance problems by substituting a better allocator (e.g., Doug Lea's). This is a good idea for lots of reasons. It takes almost no time and involves no source code changes, and leverages heavily-used robust code.
The horrible suggestion is to roll your own memory manager. This approach causes numerous software engineering problems -- consider what happens if you inadvertently call free on one of your custom-allocated objects. Worse, it often yields no performance improvement compared to DLmalloc. See my OOPSLA 2002 paper Reconsidering Custom Memory Allocation for a full analysis. One of the morals of the story is that you should use DLmalloc instead of freelists, which yield neither cycle-level nor locality performance advantages.
The shell-like scripting languages are actually a lot closer fit to their problem domain (managing programs, I/O, and files) than more recent scripting languages like Perl and Python. There are a number of reasons for their decline in popularity, but it is hard to argue that explicit management of files and their contents (open, close, iteration) is easier or better than piping and redirection. It is also possible to make shell-like languages both faster and safer than other scripting languages (this is work in progress - I hope to make a prototype release available soon). Although it may surprise many Slashdot readers to hear that bash is easily 10 times faster than Perl for many text processing tasks...
I would be interested to know what is motivating the shift exactly. I gather that the poster is not intending to use bash to write lengthy programs. On the other hand, it sounds like some other posters do use some variant of shell to write long programs. For what exactly? Is it really more convenient than using another scripting language? Part of the motivation for this question is that I have developed a little prototype optimizing interpreter for a shell-language subset (basically sh) and am looking for applications, especially where performance is an issue.
read the comments from the horse's mouth
on
Grid Processing
·
· Score: 4, Informative
This story already appeared, but was posted by someone who was not confused by the use of the term "grid"... Doug Burger, one of the two key profs on this project (and no relation!), answered lots of questions, which you can see here.
-- emery berger, dept. of cs, univ. of massachusetts
We're doing OS research here (UMass) with Linux, and unless you have very specialized needs, you're far better off using User-Mode Linux. By running Linux as a user-process, you don't have to worry about bringing down your machine and you can use gdb to do your debugging.
In the field of computer science, this problem is probably less pronounced. Several resources make it very easy to read papers, so access is not much of an issue. These resources include CiteSeer and the ACM Digital Library, which contain vast databases of computer science publications in electronic form.
You have to be careful when you say that something is "accepted at OOPSLA." I happen to have a paper that's in the technical track of OOPSLA this year (Reconsidering Custom Memory Allocation). That's where the real computer science is happening. Then there are the other sessions that are, shall we say, not held to the same standards...
Slashdot Mirror
As an aside: the default payment level for AutoMan is US minimum wage, and there is no built-in provision for differentiating wages based on the country of the worker.
Relevant - a related but different approach: AutoMan, a language for programming with people: http://www.automan-lang.org/
AutoMan is a platform for integrating human-based and digital computation. It allows programmers to "program with people", which appear to the programmer to be ordinary function calls. AutoMan automatically handles details like quality control, payment, and task scheduling. It is currently implemented as a domain-specific language embedded in Scala (a language that runs on any machine with a Java Virtual Machine), and uses Amazon's Mechanical Turk as a backend.
Technical paper at http://www.cs.umass.edu/~emery/pubs/AutoMan-UMass-CS-TR-2012-013.pdf, to appear at OOPSLA 2012.
(Disclaimer: I am one of the authors.)
You might be interested in the other best paper award winner (in the shameless self-promotion department): TFS: A Transparent File System for Contributory Storage , by Jim Cipar, Mark Corner, and Emery Berger (Dept. of Computer Science, University of Massachusetts Amherst). Briefly, it describes how you can make all the empty space on your disk available for others to use, without affecting your own use of the disk (no performance impact, and you can still use the space if you need it).
Enjoy!
--
Emery Berger
Dept. of Computer Science
University of Massachusetts Amherst
If you encounter a problem using DieHard, I'd appreciate hearing some details (send me mail). Obviously, the system works fine for me and others. I'm using it now, and have been doing so for months without incident.
Thanks,
-- Emery Berger
No, that's what OpenBSD does. Sorry for any ambiguity -- that whole paragraph was about OpenBSD.
-- Emery Berger
Here's a more detailed answer -- I'll add it to the FAQ.
OpenBSD (a variant of PHKmalloc) does some of what DieHard's allocator does, but DieHard does much more. On the security side, DieHard adds much more "entropy"; on the reliability side, it mathematically reduces the risk that a programmer bug will have any impact on program execution.
OpenBSD randomly locates pages of memory and allocates small objects from these pages. It improves security by avoiding the effect of certain errors. Like DieHard, it is resilient to double and invalid frees. It places guard pages around large chunks and frees such large chunks back to the OS (causing later references through dangling pointers to fail unless the chunk is reused). It attempts to block some buffer overflows by using page protection. Finally, it shuffles some allocated objects around on a page, randomizing their location within a page.
DieHard goes much further. First, it completely segregates heap metadata from the heap, making heap corruption (and hijack attacks) nearly impossible. On OpenBSD, a large-enough underflow on OpenBSD can overwrite the page directory or local page info struct (at the beginning of each page), hijacking the allocator. This presentation describes several ways OpenBSD's allocator can be attacked. By contrast, none of DieHard's metadata is located in the allocated object space.
Second, DieHard randomizes the placement of objects across the entire heap. This has numerous advantages. On the security side, it makes brute-force attempts to locate adjacent objects nearly impossible -- in OpenBSD, knowing the allocation sequence determines which pages objects will land on (see the presentation pointed to above).
DieHard's complete randomization is key to provably avoiding a range of errors with high probability. It reduces the worst-case odds that a buffer overflow has any impact to 50%. The actual likelihood is even lower when the heap is not full. DieHard also avoids dangling pointer errors with very high probability (e.g., 99.999%), making it nearly impervious to such mistakes. You can read our PLDI paper for more details and formulae.
-- Emery Berger
Hi Slashdot readers,
DieHard's randomization is very different from what OpenBSD does, not to mention Vista's address-space randomization. I've added a note to the FAQs that explains the difference in some detail, and answers several other questions, but in short: "address-space randomization" randomizes the base address of the heap and also mmapped-chunks of memory, leaving the relative position of objects intact. By contrast, DieHard randomizes the location of every single object across the entire heap. It also goes further in that it prevents a wide range of memory errors automatically, like double frees and illegal frees, and effectively eliminates heap corruption.
-- Emery Berger
-- Emery Berger, Dept. of Computer Science, UMass Amherst
I wrote an application that performs a number of Monte Carlo simulations (if you're interested in why, see DieHard for some idea). Anyway, I chose to write it in Python. The simulator makes intensive use of associative arrays and random number generation. The latter (RNG) is from the standard library, and as for the former (assoc arrays), one would think these would have to be implemented in more or less the same way both in CPython and Iron Python.
Nonetheless, the program runs 1.77 times slower in Iron Python than in CPython! That's not good. It suggests that there is some deep inefficiency either in the interpreter, or in the CLR, or both. Worse, when I use psyco, I get a further speedup of 3x over the original. Where does that leave us? CPython + psyco = roughly 6x faster than Iron Python.
To me, this result substantially undermines the claim that Iron Python is competitve with CPython. A nearly 2X slowdown is not competitive. Moreover, anyone looking to achieve higher performance is going to be using something like psyco, and then we're talking about potentially taking a 6X hit when using Iron Python.
While I'm sure that the .NET framework adds plenty of interesting functionality for some people, the claim that its performance is competitive strikes me as disingenuous.
But perhaps I'm just trolling...
-- Emery Berger
University of Massachusetts Amherst
I ran it on one of my Python apps and it actually slowed it down by a factor of 1.77. Lots of caveats here, but this is the opposite of what I expected to see.
We recently developed a programming language system called Flux that we presented at USENIX this June that addresses exactly this problem. Flux is a domain-specific language for building server applications from serial C and C++ components. The compiler then generates event-driven or multithreaded code -- it's just a compiler flag and a different runtime system. You as a programmer do not deal with the nitty-gritty details of managing concurrency. Moreover, Flux can optionally generate a simulator that lets you load-test your app before deployment in order to isolate bottlenecks. We've built four servers so far in Flux - including the web server hosting Flux, and a BitTorrent seed - and its performance matches or exceeds that of hand-tuned high-performance servers. It's still a prototype system, but worth checking out.
-- Emery Berger
-- Assistant Professor
-- Department of Computer Science
-- University of Massachusetts Amherst
If what you want is stability, you must replace the memory manager. This requires no change to your code; there are several plug-in replacements. You can use a conservative garbage collector to get rid of memory leaks. Another possibility -- in the self-plug department -- is to use DieHard, which eliminates a wide range of memory errors and even defends against heap overflows and other heap corruption. Pick whichever works best for your needs, but pick one. You will not get stability and reliability with C++ if you stick with the default memory manager.
You may find DieHard useful, if heap errors are also a problem -- it's a plug-in replacement for malloc/free that provides probabilistic guarantees of correct execution in the face of errors.
http://www.cs.umass.edu/~emery/diehard
-- emery
I do have a few comments about your comments, though. First, I don't understand your objection to the insertion of frees. This just simulates the effect of the programmer invoking free. We examine two cases: (1) where the free calls are as early as possible -- the last use of an object and (2) where the calls are as late as possible -- the last time the objects are reachable. This brackets the range of programmer behavior. We measure both cases. Both choices of when to free objects turn out to have surprisingly similar performance results, and in any event, all of the garbage collectors require a lot of space to reduce GC frequency (roughly 5X before runtime performance matches malloc/free).
As for the tools used -- you work with the best you have available. That said, this is a pretty comprehensive study on a reasonably solid platform, though all the collectors are stop-the-world (all threads stop during collection). Concurrent GC is slower than non-concurrent GC, so including concurrent collectors wouldn't alter our results. However, parallel stop-the-world collectors would speed tracing (computing the root set is not the problem). We couldn't test such collectors because of infrastructural limitations, but we definitely want to revisit them later.
One note: you shouldn't read our study to suggest anything about compile-time garbage collection beyond the fact that it basically changes the time-space tradeoff (it won't make programs faster when running in plentiful RAM). The oracles used to guide free calls are derived from profiling, not from static analysis. Escape analysis is great for reducing contention in the presence of multiple threads, but it's likely to be of limited value when you have generational collectors (insight: it only captures short-lived objects, precisely what generational collectors already handle superbly).
Emery Berger
Asst. Professor, Dept. of Computer Science
University of Massachusetts Amherst
But the allocation sequence isn't the bottleneck -- it's all about the actual garbage collection itself. GC can be costly in terms of cycles and, if you're unfortunate enough to be running in low physical memory, page faults. The reorganization of memory may improve application locality, but that may not be enough to buy back the performance penalty.
Rather than try to argue the case with hand-waving, my student Matthew Hertz and I performed experiments on an infrastructure that lets us replace garbage collection with malloc/free in Java programs so we can quantitiatively compare the cost of garbage collection. Our paper, to appear in OOPSLA 2005, shows that a good, generational copying garbage collector can match or slightly beat the performance of malloc/free -- but this requires 5X as much memory as the allocator. It's a classic space-time tradeoff. On the other hand, if you don't have enough RAM to hold the entire heap, though, garbage collection runs tens or hundreds of times slower.
--
Emery Berger
Asst. Professor, Dept. of Computer Science
University of Massachusetts Amherst
The same thing is happening in western Massachusetts to connect The Five Colleges (Amherst College, Hampshire College, Mount Holyoke, Smith, and the University of Massachusetts, Amherst).
I wrote a paper about this, and problems with custom (special-purpose) memory allocators in general, called Reconsidering Custom Memory Allocation (OOPSLA 2002). In it, I also describe a new memory allocation scheme, called reaps . This is a hybrid of regions (pools) and heaps that acts just like pools except you can still free individual objects.
In fact, for a case study, I put reaps into Apache (adding a ap_pfree call), and showed how using reaps made it simple to incorporate a piece of existing C code (namely, bc) into Apache. Without reaps, an invocation consumed 7.4MB of memory (since every free had to be disabled). With reaps, 240KB.
I did send a message to the Apache folks about this a while back, but they balked because the implementation is in C++, rather than C (developed with my Heap Layers infrastructure)...
-- emery
Emery Berger
Assistant Professor
Dept. of Computer Science
University of Massachusetts, Amherst
You make one horrible suggestion and one great one. The great one is to address allocation performance problems by substituting a better allocator (e.g., Doug Lea's). This is a good idea for lots of reasons. It takes almost no time and involves no source code changes, and leverages heavily-used robust code.
The horrible suggestion is to roll your own memory manager. This approach causes numerous software engineering problems -- consider what happens if you inadvertently call free on one of your custom-allocated objects. Worse, it often yields no performance improvement compared to DLmalloc. See my OOPSLA 2002 paper Reconsidering Custom Memory Allocation for a full analysis. One of the morals of the story is that you should use DLmalloc instead of freelists, which yield neither cycle-level nor locality performance advantages.
The shell-like scripting languages are actually a lot closer fit to their problem domain (managing programs, I/O, and files) than more recent scripting languages like Perl and Python. There are a number of reasons for their decline in popularity, but it is hard to argue that explicit management of files and their contents (open, close, iteration) is easier or better than piping and redirection. It is also possible to make shell-like languages both faster and safer than other scripting languages (this is work in progress - I hope to make a prototype release available soon). Although it may surprise many Slashdot readers to hear that bash is easily 10 times faster than Perl for many text processing tasks...
I would be interested to know what is motivating the shift exactly. I gather that the poster is not intending to use bash to write lengthy programs. On the other hand, it sounds like some other posters do use some variant of shell to write long programs. For what exactly? Is it really more convenient than using another scripting language? Part of the motivation for this question is that I have developed a little prototype optimizing interpreter for a shell-language subset (basically sh) and am looking for applications, especially where performance is an issue.
This story already appeared, but was posted by someone who was not confused by the use of the term "grid"... Doug Burger, one of the two key profs on this project (and no relation!), answered lots of questions, which you can see here.
-- emery berger, dept. of cs, univ. of massachusetts
We're doing OS research here (UMass) with Linux, and unless you have very specialized needs, you're far better off using User-Mode Linux. By running Linux as a user-process, you don't have to worry about bringing down your machine and you can use gdb to do your debugging.
In the field of computer science, this problem is probably less pronounced. Several resources make it very easy to read papers, so access is not much of an issue. These resources include CiteSeer and the ACM Digital Library, which contain vast databases of computer science publications in electronic form.
You have to be careful when you say that something is "accepted at OOPSLA." I happen to have a paper that's in the technical track of OOPSLA this year (Reconsidering Custom Memory Allocation). That's where the real computer science is happening. Then there are the other sessions that are, shall we say, not held to the same standards...
So is there any chance of you making your smart pointer class available? I'd certainly be interested in seeing it.