Slashdot Mirror

News for nerds writes "It was just 3 weeks ago that we learned IBM's BlueGene/L with 36.01 TFlops edged out NEC's Earth Simulator, but today NEC announces a new SX-8 supercomputer with a peak processing performance of 65 TFlops (press release). It may be available in the U.S. as Cray's OEM like SX-6."

securitas writes "The New York Times Technology has an excellent feature by Ken Belson about the coming battle over the next-generation DVD format that consumer electronics and technology giants are already preparing for. The article covers the (high-definition) HD DVD group, led by Toshiba and NEC, and the Blu-ray Group, led by Sony and Matsushita (Panasonic/JVC). Mass production is expected to begin in 2005, but both sides are expected to show prototypes and aggresively pursue partners at the Consumer Electronics Show (CES) in Las Vegas next week. Add to the mix a nine-company Chinese faction that says it will develop its own DVD format because - fearing their technology could be used by Chinese rivals - the Japanese manufacturers haven't shared much information, even within the DVD Forum. Finally, Disney, Microsoft, IBM and Intel have yet to weigh in. The worst thing that could happen would be another Betamax/VHS-type war. In that case, 'Everyone is a loser, particularly Hollywood studios, the retailer community and, most importantly, the consumer,' says Warren N. Lieberfarb, developer of the original DVD format."

jschauma writes "While it's no Indigo Espresso or a VAX Bar (though, of course, there is NetBSD/sgimips and NetBSD/vax), at least you can log in on a Mr. Coffee. And while the JavaStation has been running NetBSD for a while, full support is now completely in-tree: NetBSD's Martin Husemann announced today that he has fixed all outstanding issues with JavaStation support. This means, that you can now run your JavaStation with a stock distribution of NetBSD/sparc. The JavaStation-NC is a network computer class machine built on the microSPARC-IIep processor. More information about the JavaStation can be found in the JavaStation HOWTO, Martin's email to the port-sparc mailing list and Valeriy E. Ushakov's paper 'Porting NetBSD to JavaStation-NC.'"

slevin writes "Recently I listened to a talk by Alan Kay who mentioned that many 'new' software ideas had already been discovered decades earlier by computer scientists - but 'nobody reads these great papers anymore.' Over the years I have had the opportunity to read some really great and thought-provoking academic papers in Computer Science and would like to read more, but there are just too many to sort through. I'm wondering what great or seminal papers others have encountered. Since Google has no answers, perhaps we can come up with a list for the rest of the world?"

securitas writes "The New York Times is carrying a Reuters story about Linux as the software of choice for consumer electronics. At the world's largest consumer electronics show, the IFA trade fair 'the first Linux products are already on show and more will come soon, companies said.' The reason? Linux is freely available, widely embraced and profit margins in the consumer electronics business are one or two percent at best. The math is simple. The industry push comes from the members of the Consumer Electronics Linux Forum (CELF), that includes Sony, Philips, Matsushita/Panasonic, Hitachi, Sharp, Samsung, NEC, IBM, LG, Thomson/RCA and Toshiba. The CELF was previously discussed on Slashdot. Mirrors at Silicon.com and CNet News."

Knacklappen asks: "On August 5th, vascoda, Germany's new central access point for comprehensive scientific information, goes online. It will incorporate 23 virtual libraries and 4 scientific information networks, and offer these information for free. For the paying customer, there will also be access to electronic journals. What freely accessible scientific information portals do you use? I usually turn to the following when searching for articles: arxiv.org, AVEL, CiteSeer, dissonline.de, DOE Information Bridge, DSpace, ETD, NDLTD, OAIster, OPUS, TheO. Are there any others that you can recommend?"

rocannon writes "Cellular-news reports that NEC has announced a technical cooperation with the Japanese toy manufacturer, Konami to enable its range of MICROiR toys cars to be controlled from the DoCoMo N504iS handset or the J-N51 handset sold by J-Phone. More details on Cellular News."

An anonymous reader writes "To distinguish images derived from living vs. non-living sources, USC and NASA JPL researchers report today using the standard gzip compression utility. As a measure of overall pattern complexity, they find that the inherent pixel content of biologically generated fossils produces higher image compression ratios [more data redundancy], compared to their non-biological counterparts. The more the file shrinks, the more likely it is that a living process was involved. A test is live online here. This extends the simple, but powerful, uses of gzip to biogenic fossil detectors, in addition to spam cop filters, DNA sequence comparisons, digital camera image crunchers, etc. In nine months, the two Mars rovers will send back the first microscopic-scale images of Mars rocks, which should be amenable to some of these same techniques: thus gzipping is apparently pretty zippy."

lastberserker writes "Call them all you want - generics, parametrized types, thingamagic mumbojumbo - but (tada!) Java gets templates in 1.5 release. Nice landing after 5+ years of dancing around a bush. Competition is good, pardon my pun."

lastberserker writes "Call them all you want - generics, parametrized types, thingamagic mumbojumbo - but (tada!) Java gets templates in 1.5 release. Nice landing after 5+ years of dancing around a bush. Competition is good, pardon my pun."

lastberserker writes "Call them all you want - generics, parametrized types, thingamagic mumbojumbo - but (tada!) Java gets templates in 1.5 release. Nice landing after 5+ years of dancing around a bush. Competition is good, pardon my pun."

Here are 10+ plus answers to Slashdot questions from motion picture digital effects expert Thad Beier. He chose the additional questions himself. (Yes, he's on Slashdot almost every day; we asked him to do the interview after reading many intelligent comments he's posted.) Anyway, there's some fine insight into the intersection of moviemaking, graphic arts, and computer science here, brought to you by an award-winning member of the film industry who just happens to be a fellow Slashdot reader.

Are 'FX programming' days numbered?
by Anonvmous Coward
Every year, 3D packages get more and more sophsticated. Not just in terms of rendering effects, but in their scripting capabilities as well. Do you see a day where the artist will be able to handle the rendering features and the scripting of a 3D prog so well that it'll no longer be necessary to have a dedicated programmer on board?

Is there a particular type of problem that will always need a programmer?

Thad:
First, I feel that the difference between 'scripting' and 'programming' is nonexistent; both are programming, albeit in different languages with different development environments. People can, and do, write thousand-line MEL scripts for Maya -- which are every bit as complex as anything written in C. With each new animation system, the scripting languages become more powerful, and subsume larger modules as primitives within the language -- this should allow non-programmers (or, more realistically, people who don't consider themselves programmers) to create significant custom systems with reasonable short scripts.

Secondly, though, I feel that there will always be a need in movies for people who are predominantly programmers. Films have to compete with each other and with the library of pre-existing films, and one way that is done is by continually pushing the state of the art. A consistent request from filmmakers is for 'something nobody has seen before'. Often that means creating custom tools; or building scripts and shaders far beyond the capabilities of non-programmers.

It is true that as time has gone on, the percentage of people on visual effects teams who consider themselves primarily programmers has fallen. One reason is that when doing 500 shots of a mouse for Stuart Little you only have to create the mouse once, but you still need hundreds of people to do the artistic tasks of animation, lighting, and compositing. That doesn't mean that the programmers aren't important, they are the key to ensuring that the artists can be productive.

Shaders
by f00Dave
How much overlap is there between the programable graphics processing units (AKA "shaders") found on modern game platforms and the software/hardware used in the special effects industry? Would programming skills for one translate to the other?

BTW, I realize that special effects are half artistry, half mathematics and half sweaty work: kudos from a 'GL hacker... [;-)]

Thad:
I note that some slashdotters have criticized your math, but you have hit upon a fundamental truth of visual effects, that the work takes far more than the available time.

While it is conceivable that there is overlap possible between programming of games hardware and writing shaders for visual effects, I haven't seen too many people making the move from games into FX; mostly it is the other way around. Certainly many people in the games business are clamoring for visual effects and other film artists to help bring cinematic ideas and qualities to the games world.

The interesting new wrinkle in this is the Cg language from Nvidia. It's a new, high-lvel language for writing shaders. Cg is then compiled down to microcode run on the graphics hardware in the machine. While I had been skeptical, now I think that this might dramatically change the way that rendering is done. The work of the visual effects and game shader-writers could be exactly the same. It wouldn't surprise me if future software renderers use graphics hardware to speed up the process.

Cost
by Fembot
When films are labled as "100$ Million on special effects" where does most of that money go? On rendering hardware or what?

Thad:
I don't think that any movies have had $100 Million in special effects, yet -- unless you count Dinosaur or Final Fantasy -- which are animated (as opposed to FX) films. That said, the overwhelming cost on any films for effects at this point goes to the creative people. Especially today, the hardware is virtually free. (In some cases, the hardware is literally free as a company will donate machines in return for good PR.)

A reasonable estimate for the cost is 75% for artists, and 25% for everything else. This has changed dramatically over the digital visual effects era which started around 1990 -- back then it was probably exactly the opposite. But machines have gotten much cheaper and animators have become more expensive, and that trend will probably continue. It's interesting that people talk about how much cheaper Linux PCs are compared to SGI machines (say), but truly both machines have almost the same cost (zero) compared to the cost of the animator who is using the machine. The choice of workstation should be entirely based on what makes the artist most productive.

Directors approach?
by FurryFeet
I'm guessing you get to work pretty closely to directors. If so, can you tell us what is their approach to the new tools technology has given them? Are they still "thinking celluloid" made cheaper by rendering it digitally, or do they really seek to break the mold and make shots that were previously impossible?

Thad:
The job of a movie director is to harness the skills of hundreds of talented, unique, possibly difficult people to create his vision and tell his story. In our experience, directors always request the ideas and proposals from his creative team; and they listen to that advice. The FX team is hired to help make the movie, and are trusted to help make the decisions. In most cases, the director will work very closely with the FX supervisor when shooting the shots that will have effects, asking for help and comment on all aspects of the shot. After the ability to get the most out of his team, though, the most important quality of a director is decisiveness -- once all of the input has been gathered, everybody has to march in the same direction.

Every director we have worked with has been extremely interested in any ideas we could contribute to making shots cheaper, better, easier to shoot, or cheaper. They want to get the best images on film, and any resources saved on one shot can make the next one better.

best way to get into the industry?
by josepha48
What is the best way to get into the computer generated special effects industry? Is it who you know or what you know? If it is what you know what should one know? (Programming, graphics tools, etc...).

Thad:
Well, my first sincere, if unhelpful answer is "Are you sure that you want to?" It isn't really an industry in the traditional sense -- there is little or no job security, there are long hours typically with no overtime paid, the stress can be extreme and the rewards are not great. There are almost no rational reasons to choose CG visual effects as a career. So think about it before making that choice. If it really is the most important thing in the world for you, then read on.

Every person is different, and every position to be filled is different, so any advice given will either be too specific to be generally useful, or so broad as to be a platitude, but I'll do my best. Over the last few years it is my impression that there have been far more applicants trying to get into the field than available jobs; that might just be a cyclical problem or it might be persistent.

A solid undergraduate education is always a good thing. Some basic art experience is helpful, to learn color theory, layout -- basically learn what makes a good image good. Knowledge of mathematics and elementary physics is useful, to know how the world works. General computer experience is helpful, for example the ability to write and understand shell scripts. To get a job at a large facility a familiarity with the most commonly used tools is helpful.

Clearly you would like to have some animation experience. Computer animation is useful, but 2D hand-drawn animation is also an exceptionally good way to learn how to bring images to life.

When preparing a reel of your work, a few great shots is better than a large volume of mediocre work. You want something to make your reel stand out from the rest of them. Play to your strengths; concentrate on what you do best. If the work on your reel includes shots done by a team of people, be certain to call out your particular contribution. A demonstrated ability to work on teams with other creative people is a definite plus.

The Siggraph show every year is a good place to meet recruiters from many companies in a few days. It takes place in late July or early August. This year's conference took place last week, and all of the big companies demonstrated vigorous recruiting efforts. A few companies have great pages to assist people in planning their careers. Here is the employment FAQ from Pixar and the one from PDI. While they are animation companies as opposed to visual effects companies, their advice is still appropriate, by and large.

What movies have impressed you?
by Anonvmous Coward
When somebody has intimate knowledge about how a movie is made, it gets really hard to make their eyes jump out of their head.

For example, there's a scene in the Director's Cut of Robocop where Alex Murphy is just about to be shot in the head by the lead bad dude. The camera is pointing right at Alex's face, then swings around behind him. As soon as the camera is behind him the bad guy fires a gun, the back of Alex's head explodes and you can see a hole clean through it. This whole scene was one smooth camera movement, no edits.

I was *stunned* to find out that Alex was a puppet. They were able to make a puppet that totally convinced me that Peter Weller was sitting in front of this guy about to get his head blown off. I could not believe that they were able to do one that convincing.

I'm curious, what movies have had that affect on you? "OMG! I had no idea that was an effect!"

Thad:
Your example is a classic of FX misdirection. Another one is in 'Spiderman'. We see Peter Parker with his shirt off pretty early in the movie, and he's the scrawny little twerp that he's supposed to be, and you accept it without a second thought. Later, after he's been bitten, he takes off his shirt and he's totally ripped. Not until that point do you say "hey, wait a minute! How did they do that effect!" When, of course, the effect happened in the first shot with a body replacement that you never expected. I was blown away, it was just so cool, and so easy. The best effects are those that you would never expect, and that by the time you realize that they must have been effects they are over.

These days almost every film has FX shots that nobody could possibly see. Our first film was 'Showgirls', and I defy anybody to find the dozen shots we did in that movie -- they are not in-your-face effects. Two of our more recent films, 'The Fast and The Furious', and 'For Love of The Game' were praised in the Los Angeles Times and Variety as films with a refreshing lack of special effects. It's not that they're missing obvious things; it's just that FX can be undetectable.

So, when you say if there's anything where I'd say "I had no idea that was an effect", well, it's certainly true -- but for most of those shots I still don't know that it was an effect.

Project you'd like to tackle?
by seldolivaw
Although recently a lot of the big names in science fiction and fantasy are finally making it onto the screen in a plausible way (e.g. Tolkein) there are still plenty of great books out there that haven't even been optioned. If you could turn any science-fiction/fantasy book or series into a movie, which would it be?

[My personal choice: the Foundation saga by Asimov. So huge! Such a great plot! So eminently filmable! Somebody make this movie, dammit! :-)]

Thad:
Surprisingly, and contrary to your question, classic SF books like The Foundation Trilogy and Ender's Game are always in play; we get scripts or proposals for these every couple of years. You're not the only one that wants to see these books filmed; it's very difficult to do, though. While the tremendous success of The Lord of The Rings is on everybody's mind, don't forget that people have been trying for years to make those books into films with limited success. A good book has such scope and detail that it's hard to distill it into a reasonable-length movie. While I'd love to see a movie made from Stephenson's 'Snowcrash', any reasonable length movie would have to leave out at least half of the stuff that makes the book great.

Short stories are a better bet. The astonishing success of movie versions of Philip K. Dick's short stories would have completely bewildered him, but they are great source material. I'd love to see a John Varley short story -- say, 'The Phantom of Kansas' -- although I admit that 'Millenium', based on the book-length version of his short story 'Air Raid', was perhaps the worst movie I've ever seen.

Reduction in man-hours for CG?
by ceswiedler
At one point, as a film student, I was interested in computer animation as a way for a single person or small group to produce a film, without the expense of locations, casting, cameras, etc. I thought that soon, as hardware and software improved, it would be possible for me to create a film on my own computer at home.

But my experience in animation in college taught me that increasing hardware capacity doesn't reduce the time it takes to produce a film or demo reel; it simply increases the quality of the final output. I imagine that the modelling, animation, and rendering of the scenes in Tron took as much human time as comparable scenes in Fellowship of the Ring. It's possible to render Tron-quality CG in realtime on a modern PC, but nobody wants to watch it.

My question is this: do you think it will ever be possible to produce a full-length CG film in about a man-year or less, with effects which are reasonbly "modern" for the time? Will the technology curve eventually flatten out, once we get to a certain point where the human eye can't really tell the difference? Or is it implausible to think that a single person or small group could provide all of the artistic input (scriptwriting, directing, modelling, animation, acting, etc) to produce a full film, even ignoring all technological constraints?

Thad:
There are movies created by small teams of people; and some of these will be CG generated films. They won't be "Toy Story", though, they'll be motion-capture or cg-puppet films with relatively simple lighting setups; I don't think that you can do high-quality animation quickly, except through some kind of performance capture. There's a sort of Moore's Law at work with state-of-the-art animation where the complexity of scenes doubles every couple of years. Animators always will wait a certain amount of time for their frames to render, on the order of 15 minutes to an hour -- and that time hasn't changed even though computers are 1000 times the speed they were 10 years ago.

Your question has been answered in the affirmative last year by 'Jimmy Neutron, Boy Genius'. That was a relatively small team of people working for just a few man years; and they created an incredibly successful film. Compared to 'Monsters Inc.' it wasn't state-of-the-art, but compared to say 'Rugrats in Paris', you'd have to say that it was.

I think your question about the technology curve flattening out means that you're asking whether at some point the most elaborately specified scene might render in real-time. That it's not inconceivable but it is unlikely. It's possible that computer speed will finally outstrip the ability of an animator to create complexity, so that frames will render that fast; but I think it's more likely that database-amplification techniques will allow the specification of arbitrarily complex scenes.

To some extent what makes movies interesting is that a single two-hour movie can contain the distilled essence of a thousand man-years of work, and if it was a well managed process you can see each hour of effort up on the screen. You can see these movies over and over, and always see something new. It's like a tidal wave of information flooding over you. A small team of people won't be able to do that; but they can make perfectly good smaller movies.

Killing the Classics
by Skyshadow
Several directors have recently released "special editions" of their classic movies which subtly change the films by using computers effects to either clean up the old effects or (far worse) alter the original film.

The problem that I have with this is twofold: First, these "special editions" seem to be the ones that show up on TV and on video rental shelves, so that they and not the original become the pervasive copy.

Second, I can foresee a day when older movies are edited in this fashion so they can be remarketed to audiences with more "modern" attitudes (think similar to Speilburg taking the guns out of the hands of the pursuing authorities in the ET rerelease).

Do you believe that, as a creative professional, you have any sort of ethical duty to resist these sorts of changes? Is there a line to be drawn between merely cleaning up the original effects and replacing them entirely (as in the Star Wars special edition), or between effects-patchup and all-out content alteration (aka, the wussification of Han Solo by having Greedo shoot first)? Do you feel that old films should be left alone, or do you consider them more as ongoing acts of creation?

Thad:
I do not like the changing of movies. A movie, to me, reflects the time that it was created and becomes a kind of historical document. On the other, dominant hand, it is completely the choice of the owner of a film to do with it what he pleases.

I can understand the feeling that a movie is somehow owned by society at some point, but my point of view is different. Making a film is tremendously hard, making a good one far harder still -- and with that effort comes the right to muck it up down the road if that's what one decides to do. So, I don't see an ethical dilemma at all. I think that the place to make your protest felt is as a critic and as a movie patron; vote with your wallet. A related problem is that movies have a relatively short lifetime. The first Star Wars film reportedly had deteriorated quite a bit before the Special Edition was created; as the dyes in the film don't have good long-term stability. There will be a fifty-year period of movies that will be lost unless extraordinary (and unlikely) efforts are made. In the near future, though, all movies will be digital at some point in the process, and they have at least a fighting chance of being around for a long time. There are several digital-post facilities being set up now, which scan the whole film to allow better color correction and editing -- the most striking use of this was on O Brother, Where Art Thou?, where the final movie was dramatically color-corrected throughout in a way not possible with optical means.

question for thad
by Jucius Maximus
Thad: When designing tools for making 3D scenes or characters, how much does real world physics play into what is generated? Do you use fluid mechanical models to generate the flow of water over a waterfall or the movement of a large tree affected by a mass of air? Do you use vibro aoustical and biomechanical models to determine they way a CG mechanised character will walk?

In essence, how much do you take real physics into account when designing something a CG item to emulate a 'real' item on screen? What is the balance between physical limits and creative freedoms?

Thad:
Our charter is to create the sequence that the director of the film wants for his movie; that usually means building things that look and move like things do in real life. Often we would use real-world physics to do this. Typically, though, we take extremely simplified views of the real world to make the computations more simple, and to make them run faster.

As an example of physics in action, Nick Foster at PDI created a simplified fluid dynamics model to be used for animation; this was used to create several shots in ANTZ and Shrek. One of the big problems with simulation, as opposed to animation, is that it is difficult to control. Typically one sets initial conditions and then lets the simulation run. Having a system that runs very quickly enables the artist using the tool to try many different initial conditions, to try to create the desired final result. A slow, but more physically accurate solution would have been worthless if the animator couldn't get to a reasonable result.

Often what is done is an absurdly simplified model of reality is chosen, then it is made more accurate (and slow) until it looks good enough for the film. On our recent movie 'Showtime', we had to do a waterfall bursting out of a building, and we simulated the motion of water with air-drag, then simulated the water dragging the air with it, to get the characteristic motion of a waterfall; but we didn't have to go any further than that and simulate viscosity of the water or evaporation.

ILM has done some wonderful work simulating the dynamics of creatures, creating models of bones, muscles, fat, and skin. These give a character like a dinosaur a 'weight' that just can't be animated by hand. These dynamics are a great cue to they audience for how big and heavy these creatures are.

One curious reality of the FX world is that often reality is not what is wanted. A classic example of this is starfields. In any real-world photograph, the stars are invisible, they are far far darker than anything else in the scene. Directors often want stars in the sky to go with their actors, though; so that is what they get.

Finally, there are times when straightforward animation is the best approach. For the movie Red Planet, we had to create zero-g fire. I spent a few weeks trying to simulate the flow of smoke and fire in a zero-g environment, when my colleague Jamie Dixon thought that he could just animate all the shots by hand in a couple of days -- which he proceeded to do. When CalTech's physics department reviewed the movie for the Los Angeles Times, they panned every bit of science in the movie, except for the zero-g fire; which they thought looked "pretty cool."

CGI alternatives
by Strange Ranger
Do you think CGI can too often be seen as a "suppressor" of other art forms? The specific example in my head right now is Old Puppet Yoda vs. New CGI Yoda, we haven't seen (AFAIK) any major puppeteering work in cinema in a long time. Other possibly "suppressed" art forms might be makeup art, the art of the stunt man, set construction, backdrop painting, cinematograghy, heck even acting could be listed here. Will CGI be escorting some or all of these art forms down the same path as Silent Films, blacksmithing, and totem-pole carving?

Do you ever want to say "Hey this would be a lot better if it were done with [not CGI] instead"?

Thad:
There are many times as many people working in the FX field today as there were ten years ago. Now, it's true that some of the techniques are not as much in demand as they were, but it's not as bad as you might think. A company that we do quite a bit of collaborative work with is Illusion Arts, in Van Nuys California. The two founders, Bill Taylor and Syd Dutton, started doing practical camera effects and matte paintings, and built a very successful company around these kinds of classical techniques. Today, they are now doing synthetic 3D camera moves and painting on Macs; but 90% of the talents and skills they used before are still applicable today; just the medium is different. What makes a good artist is foremost their eye; their ability to see what is right and to see how to fix what is wrong. Illusion Arts was the lead shop in The Fast and The Furious, and along with us and Digiscope made a very modern movie.

In your example of puppetry, I too was a little disappointed to see the CG Yoda; especially in the closeups it just wasn't exactly the same. Of course, there's no way that a puppet could have done the lightsaber battle. Also, a growing area of FX is performance capture; recording data in real-time and applying it to CG characters. In Episode One and Two of the Star Wars movies, there is a tremendous amount of motion capture, used to animate robots and creatures. Performance capture is just puppetry with one's whole body, really.

Back in 1989, Graham Walters and I build the CG puppet Waldo C. Graphic for The Jim Henson Hour. The puppet was animated by putting one's hand into a 'waldo' (a mechanical tracking device reminiscent of a Luxo Lamp), and moving it around; and watching the results on a TV screen. This was so similar to the way that the Henson puppeteers usually work that it took no time at all to get comfortable with the puppet; I don't think it took Henson himself more than about 5 seconds to get totally up to speed on it.

Speaking about stunt work, one of the very first things that people realized with digital techniques is that 'wire-removal' is fairly straightforward. One can identify a moving wire in a scene and use several techniques to get rid of it. This meant that whereas stunt people used to use the thinnest possible safety wires, or none at all; now they could use systems with significant margins of safety. Also, face-replacement techniques coming to the fore means that stunt people can play far closer to the camera than they used to, opening up new opportunities for stunts.

When it comes to acting, though, I don't think that digital graphics will ever replace traditional techniques. There's no good reason to attempt it, and it's unbelievably hard. The subtlety and complexity of motion of skilled human actors is astonishing, and a ridiculous portion of the human brain is dedicated to interpreting those expressions and motions.

So, I would say that for every job lost, many are created -- and the people whose jobs are lost can often put those same skills to use in this new digital world.

Little studios vs Big Studios
by Milinar
I've followed your company's work over the past few years with great interest. It seems to me that the effects you do are pretty much on par with big studios like digital domain, etc. Have you purposefully stayed a small studio, with a few dedicated individuals? And what advantages has that given you?

Thad:
When we started Hammerhead, we made a deliberate commitment to stay small. We didn't see significant economies of scale in the field, and it seemed like we'd have much more fun in a smaller company. There is a strong culture in American business that you have to "grow or die", but it was our experience that growth was extremely difficult to manage and that companies that grew quickly found themselves dying quickly, too. Once you get past a couple of dozen people, there seems to be a phase change in company culture, and productivity declines.

We do find that our small ('boutique' is the term of art) studio can compete against companies one or two orders of magnitude larger than us on many jobs that don't require a huge volume of shots. While our staff is small, we are extremely experienced, having been doing digital visual effects since we helped create the field at the beginning of the 90's. We tend to hire very capable, experienced artists -- one way that we keep it interesting for them is that they are given a huge amount of creative control over their shots.

As a small company we can be very flexible, too. We can reconfigure ourselves for whatever project is at hand; and become the Deep Blue Sea company when that is what is going on. There is very little overhead not contributing to getting a particular job done. Paradoxically, in a small company you can do more different things. We've done FX for films, wrote and produced a big Hollywood film, made our own low-budget horror film, and wrote and sold software. We will very likely be making a couple of TV pilots next year of shows with substantial visual effects content. Bigger FX companies have to be more focused, they can't afford to be experimental and possibly make mistakes, because they would be much larger mistakes.

Our biggest weakness is that we cannot even begin to take on huge jobs. Movies like Pearl Harbor or Spiderman require hundreds of people; and we have to leave these jobs to the ILMs and Sonys of the world. Still, there are hundreds of movies a year with a few dozen to a hundred and fifty shots, with reasonable time schedules, where we can compete well. I think that we have found a 'sweet spot', where many features combine to make a pleasant, profitable, successful company -- and the small size is an important part of that.

Dropped crusade against Pixar patent?
by Anonymous Coward
I heard a rumor that you dropped your "crusade" against Pixar's software patent on deep-shadow technology? The rumor implied you were "bought-out"? Care to comment/share your thoughts on software patents in the VFX industry?

Thad:
While this was not moderated up, I do feel it needs an answer. The patent that is referred to is for the obvious enhancement of Lance Williams' 1978 z-buffer shadow scheme [pdf link] given that today's computers have more than 64 Kb of memory. In the Williams algorithm the scene is rendered from the point of view of the light, and the depth to the first surface is stored. Then, when rendering the image from the camera's point of view, you can easily tell whether a surface should be in shadow or not. The Deep Shadow Map idea was to store a function of depth vs. opacity at each pixel in the image rendered from the light POV, to allow partially transparent surfaces and subpixel shadow coverage.

Unfortunately, Pixar has decided to patent this. They presented the idea at Siggraph '00 but didn't mention in the paper the fact that they'd filed a patent; although word got out pretty soon. As the patent has not been granted yet, and they filed the patent before the Patent Office's policy change that now publishes patent applications, it's unknown what their claims are. What I am fairly sure of, though, is that Pixar didn't invent this technology, and that people at Pixar know this. So, it's not only really nasty to try to build on somebody else's technique and wrest it for yourself, but there may be legal problems as well.

I've discussed this with lawyers, and they say that the time to fight a patent is after it grants. While that seem weird and suboptimal, there's nothing about patent law that isn't weird and suboptimal. So, I'm going to wait and see what happens. There are other possibilities for fighting the patent that don't make sense to reveal at this time, for obvious reasons. Clearly this comment reveals that there is no agreement between Pixar and me to remain quiet on this issue.

It wouldn't surprise me if patents destroy the visual effects industry as we know it today. Pixar already has one notch in its belt, last week forcing the company ExLuna to withdraw its Entropy renderer that competed with Pixar's Renderman (and the shareware BMRT program that preceded Entropy, as well). A rational, cold-blooded analysis of the software patent situation would reveal that almost every complex program today could be attacked on patent grounds, as we've seen recently with the JPEG fiasco. Back when I worked at PDI, we were attacked a couple of times for patent violations, only escaping a devastating patent by NYIT on the thinnest of technicalities. In irony not lost on anybody, Ed Catmull of Pixar (with Disney's lawyers help) led the fight against NYIT's patent.

Interestingly, this has happened before in visual effects. Back in the bad old days, every single analog visual effects technology was patented and owned by the studios. Rear Projection, Front Projection, Blue Screens, Sodium Screens -- everything. The studios would in effect pool the patents between themselves; but if you wanted to make a visual effects film you had to do it completely within the studio system. It might happen again.

Mario Valente writes "Using my glass topped desk it should probably be possible to build your own digital playing desk, commonly referred in VR circles as a projection-table. I guess I'd need at least an LCD projector for rear-projection onto the desk and a mouse/finger tracker. Has anyone had experience building this type of stuff ? Are there any non-highend commercial systems available? -- Mario Valente"

Dare Obasanjo contributed this followup to an article entitled The Myth of Open Source Security Revisited that appeared on the website kuro5hin. He writes: "The original article tackled the common misconception amongst users of Open Source Software(OSS) that OSS is a panacea when it comes to creating secure software. The article presented anecdotal evidence taken from an article written by John Viega, the original author of GNU Mailman, to illustrate its point. This article follows up the anecdotal evidence presented in the original paper by providing an analysis of similar software applications, their development methodology and the frequency of the discovery of security vulnerabilities." Read on below for his detailed analysis, especially relevant with the currency of security initiatives in the worlds of both open- and closed-source software.

The Myth of Open Source Security Revisited v2.0 The purpose of this article is to expose the fallacy of the belief in the "inherent security" of Open Source software and instead point to a truer means of ensuring the quality of the security of a piece software is high.

Apples, Oranges, Penguins and Daemons

When performing experiments to confirm a hypothesis on the effect of a particular variable on an event or observable occurence, it is common practice to utilize control groups. In an attempt to establish cause and effect in such experiments, one tries to hold all variables that may affect the outcome constant except for the variable that the experiment is interested in. Comparisons of the security of software created by Open Source processes and software produced in a proprietary manner have typically involved several variables besides development methodology.

A number of articles have been written that compare the security of Open Source development to proprietary development by comparing security vulnerabilities in Microsoft products to those in Open Source products. Noted Open Source pundit, Eric Raymond wrote an article on NewsForge where he compares Microsoft Windows and IIS to Linux, BSD and Apache. In the article, Eric Raymond states that Open Source development implies that "security holes will be infrequent, the compromises they cause will be relatively minor, and fixes will be rapidly developed and deployed." However, upon investigation it is disputable that Linux distributions have less frequent or more minor security vulnerabilities when compared to recent versions of Windows. In fact the belief in the inherent security of Open Source software over proprietary software seems to be the product of a single comparison, Apache versus Microsoft IIS.

There are a number of variables involved when one compares the security of software such as Microsoft Windows operating systems to Open Source UNIX-like operating systems including the disparity in their market share, the requirements and dispensations of their user base, and the differences in system design. To better compare the impact of source code licensing on the security of the software, it is wise to reduce the number of variables that will skew the conclusion. To this effect it is best to compare software with similar system design and user base than comparing software applications that are significantly distinct. The following section analyzes the frequency of the discovery of security vulnerabilities in UNIX-like operating systems including HP-UX, FreeBSD, RedHat Linux, OpenBSD, Solaris, Mandrake Linux, AIX and Debian GNU/Linux.

Security Vulnerability Face-Off

Below is a listing of UNIX and UNIX-like operating systems with the number of security vulnerabilities that were discovered in them in 2001 according to the Security Focus Vulnerability Archive. AIX 10 vulnerabilities[6 remote, 3 local, 1 both] Debian GNU/Linux 13 vulnerabilities[1 remote, 12 local] + 1 Linux kernel vulnerability[1 local] FreeBSD 24 vulnerabilities[12 remote, 9 local, 3 both] HP-UX 25 vulnerabilities[12 remote, 12 local, 1 both] Mandrake Linux 17 vulnerabilities[5 remote, 12 local] + 12 Linux kernel vulnerabilities[5 remote, 7 local] OpenBSD 13 vulnerabilities[7 remote, 5 local, 1 both] Red Hat Linux 28 vulnerabilities[5 remote, 22 local, 1 unknown] + 12 Linux kernel vulnerabilities[6 remote, 6 local] Solaris 38 vulnerabilities[14 remote, 22 local, 2 both] From the above listing one can infer that source licensing is not a primary factor in determining how prone to security flaws a software application will be. Specifically proprietary and Open Source UNIX family operating systems are represented on both the high and low ends of the frequency distribution.

Factors that have been known to influence the security and quality of a software application are practices such as code auditing (peer review), security-minded architecture design, strict software development practices that restrict certain dangerous programming constructs (e.g. using the str* or scanf* family of functions in C) and validation & verification of the design and implementation of the software. Also reducing the focus on deadlines and only shipping when the system the system is in a satisfactory state is important.

Both the Debian and OpenBSD projects exhibit many of the aforementioned characteristics which help explain why they are the Open Source UNIX operating systems with the best security record. Debian's track record is particularly impressive when one realizes that the Debian Potato consists of over 55 million lines of code (compared to RedHat's 30,000,000 lines of code).

The Road To Secure Software

Exploitable security vulnerabilities in a software application are typically evidence of bugs in the design or implementation of the application. Thus the process of writing secure software is an extension of the process behind writing robust, high quality software. Over the years a number of methodolgies have been developed to tackle the problem of producing high quality software in a repeatable manner within time and budgetary constraints. The most successful methodologies have typically involved using the following software quality assurance, validation and verification techniques; formal methods, code audits, design reviews, extensive testing and codified best practices.

1. Formal Methods: One can use formal proofs based on mathematical methods and rigor to verify the correctness of software algorithms. Tools for specifying software using formal techniques exist such as VDM and Z. Z (pronounced 'zed') is a formal specification notation based on set theory and first order predicate logic. VDM stands for "The Vienna Development Method" which consists of a specification language called VDM-SL, rules for data and operation refinement which allow one to establish links between abstract requirements specifications and detailed design specifications down to the level of code, and a proof theory in which rigorous arguments can be conducted about the properties of specified systems and the correctness of design decisions.The previous descriptions were taken from the Z FAQ and the VDM FAQ respectively. A comparison of both specification languages is available in the paper, Understanding the differences between VDM and Z by I.J. Hayes et al.
   
   
2. Code Audits: Reviews of source code by developers other than the author of the code are good ways to catch errors that may have been overlooked by the original developer. Source code audits can vary from informal reviews with little structure to formal code inspections or walkthroughs. Informal reviews typically involve the developer sending the reviewers source code or descriptions of the software for feedback on any bugs or design issues. A walkthrough involves the detailed examination of the source code of the software in question by one or more reviewers. An inspection is a formal process where a detailed examination of the source code is directed by reviewers who act in certain roles. A code inspection is directed by a "moderator", the source code is read by a "reader" and issues are documented by a "scribe".
   
   
3. Testing: The purpose of testing is to find failures. Unfortunately, no known software testing method can discover all possible failures that may occur in a faulty application and metrics to establish such details have not been forthcoming. Thus a correlation between the quality of a software application and the amount of testing it has endured is practically non-existent.
   
   There are various categories of tests including unit, component, system, integration, regression, black-box, and white-box tests. There is some overlap in the aforementioned mentioned testing categories.
   
   Unit testing involves testing small pieces of functionality of the application such as methods, functions or subroutines. In unit testing it is usual for other components that the software unit interacts with to be replaced with stubs or dummy methods. Component tests are similar to unit tests with the exception that dummmy and stub methods are replaced with the actual working versions. Integration testing involves testing related components that communicate with each other while system tests involve testing the entire system after it has been built. System testing is necessary even if extensive unit or component testing has occured because it is possible for seperate subroutines to work individually but fail when invoked sequentialy due to side effects or some error in programmer logic. Regression testing involves the process of ensuring that modifications to a software module, component or system have not introduced errors into the software. A lack of sufficient regression testing is one of the reasons why certain software patches break components that worked prior to installation of the patch.
   
   Black-box testing also called functional testing or specification testing test the behavior of the component or system without requiring knowledge of the internal structure of the software. Black-box testing is typically used to test that software meets its functional requirements. White-box testing also called structural or clear-box testing involves tests that utilize knowledge of the internal structure of the software. White-box testing is useful in ensuring that certain statements in the program are excercised and errors discovered. The existence of code coverage tools aid in discovering what percentages of a system are being excercised by the tests.
   
   More information on testing can be found at the comp.software.testing FAQ .
   
   
4. Design Reviews: The architecture of a software application can be reviewed in a formal process called a design review. In design reviews the developers, domain experts and users examine that the design of the system meets the requirements and that it contains no significant flaws of omission or commission before implementation occurs.
   
   
5. Codified Best Practices: Some programming languages have libraries or language features that are prone to abuse and are thus prohibited in certain disciplined software projects. Functions like strcpy, gets, and scanf in C are examples of library functions that are poorly designed and allow malicious individuals to use buffer overflows or format string attacks to exploit the security vulnerabilities exposed by using these functions. A number of platforms explicitly disallow gets especially since alternatives exist. Programming guidelines for such as those written by Peter Galvin in a Unix Insider article on designing secure software are used by development teams to reduce the likelihood of security vulnerabilities in software applications.

Projects such as the OpenBSD project that utilize most of the aforementioned techniques in developing software typically have a low incidence of security vulnerabilities.

Issues Preventing Development of Secure Open Source Software

One of the assumptions that is typically made about Open Source software is that the availability of source code translates to "peer review" of the software application. However, the anecdotal experience of a number of Open Source developers including John Viega belies this assumption.

The term "peer review" implies an extensive review of the source code of an application by competent parties. Many Open Source projects do not get peer reviewed for a number of reasons including

• complexity of code in addition to a lack of documentation makes it difficult for casual users to understand the code enough to give a proper review
  
  
• developers making improvements to the application typically focus only on the parts of the application that will affect the feature to be added instead of the whole system.
  
  
• ignorance of developers to security concerns.
  
  
• complacency in the belief that since the source is available that it is being reviewed by others.
  
  

Also the lack of interest in unglamorous tasks like documentation and testing amongst Open Source contributors adversely affects quality of the software. However, all of these issues can and are solved in projects with a disciplined software development process, clearly defined roles for the contributers and a semi-structured leadership hierarchy.

Benefits of Open Source to Security-Conscious Users

Despite the fact that source licensing and source code availability are not indicators of the security of a software application, there is still a significant benefit of Open Source to some users concerned about security. Open Source allows experts to audit their software options before making a choice and also in some cases to make improvements without waiting for fixes from the vendor or source code maintainer.

One should note that there are constraints on the feasibility of users auditing the software based on the complexity and size of the code base. For instance, it is unlikely that a user who wants to make a choice of using Linux as a web server for a personal homepage will scrutinize the TCP/IP stack code.

References

1. Frankl, Phylis et al. Choosing a Testing Method to Deliver Reliability. Proceedings of the 19th International Conference on Software Engineering, pp. 68--78, ACM Press, May 1997. < http://citeseer.nj.nec.com/frankl97choosing.html >
   
   
2. Hamlet, Dick. Software Quality, Software Process, and Software Testing. 1994. < http://citeseer.nj.nec.com/hamlet94software.html >
   
   
3. Hayes, I.J., C.B. Jones and J.E. Nicholls. Understanding the differences between VDM and Z. Technical Report UMCS-93-8-1, University of Manchester, Computer Science Dept., 1993. < http://citeseer.nj.nec.com/hayes93understanding.ht ml >
   
   
4. Miller, Todd C. and Theo De Raadt. strlcpy and strlcat - consistent, safe, string copy and concatenation. Proceedings of the 1999 USENIX Annual Technical Conference, FREENIX Track, June 1999. < http://www.usenix.org/events/usenix99/full_papers/ millert/millert_html/ >
   
   
5. Viega, John. The Myth of Open Source Security. Earthweb.com. < http://www.earthweb.com/article/0,,10455_626641,00 .html >
   
   
6. Gonzalez-Barona, Jesus M. et al. Counting Potatoes: The Size of Debian 2.2. < http://people.debian.org/~jgb/debian-counting/coun ting-potatoes/ >
   
   
7. Wheeler, David A. More Than A Gigabuck: Estimating GNU/Linux's Size. < http://www.counterpane.com/crypto-gram-0003.html >
   
   

Acknowledgements

The following people helped in proofreading this article and/or offering suggestions about content: Jon Beckham, Graham Keith Coleman, Chris Bradfield, and David Dagon. © 2002 Dare Obasanjo

Dare Obasanjo contributed this followup to an article entitled The Myth of Open Source Security Revisited that appeared on the website kuro5hin. He writes: "The original article tackled the common misconception amongst users of Open Source Software(OSS) that OSS is a panacea when it comes to creating secure software. The article presented anecdotal evidence taken from an article written by John Viega, the original author of GNU Mailman, to illustrate its point. This article follows up the anecdotal evidence presented in the original paper by providing an analysis of similar software applications, their development methodology and the frequency of the discovery of security vulnerabilities." Read on below for his detailed analysis, especially relevant with the currency of security initiatives in the worlds of both open- and closed-source software.

The Myth of Open Source Security Revisited v2.0 The purpose of this article is to expose the fallacy of the belief in the "inherent security" of Open Source software and instead point to a truer means of ensuring the quality of the security of a piece software is high.

Apples, Oranges, Penguins and Daemons

When performing experiments to confirm a hypothesis on the effect of a particular variable on an event or observable occurence, it is common practice to utilize control groups. In an attempt to establish cause and effect in such experiments, one tries to hold all variables that may affect the outcome constant except for the variable that the experiment is interested in. Comparisons of the security of software created by Open Source processes and software produced in a proprietary manner have typically involved several variables besides development methodology.

A number of articles have been written that compare the security of Open Source development to proprietary development by comparing security vulnerabilities in Microsoft products to those in Open Source products. Noted Open Source pundit, Eric Raymond wrote an article on NewsForge where he compares Microsoft Windows and IIS to Linux, BSD and Apache. In the article, Eric Raymond states that Open Source development implies that "security holes will be infrequent, the compromises they cause will be relatively minor, and fixes will be rapidly developed and deployed." However, upon investigation it is disputable that Linux distributions have less frequent or more minor security vulnerabilities when compared to recent versions of Windows. In fact the belief in the inherent security of Open Source software over proprietary software seems to be the product of a single comparison, Apache versus Microsoft IIS.

There are a number of variables involved when one compares the security of software such as Microsoft Windows operating systems to Open Source UNIX-like operating systems including the disparity in their market share, the requirements and dispensations of their user base, and the differences in system design. To better compare the impact of source code licensing on the security of the software, it is wise to reduce the number of variables that will skew the conclusion. To this effect it is best to compare software with similar system design and user base than comparing software applications that are significantly distinct. The following section analyzes the frequency of the discovery of security vulnerabilities in UNIX-like operating systems including HP-UX, FreeBSD, RedHat Linux, OpenBSD, Solaris, Mandrake Linux, AIX and Debian GNU/Linux.

Security Vulnerability Face-Off

Below is a listing of UNIX and UNIX-like operating systems with the number of security vulnerabilities that were discovered in them in 2001 according to the Security Focus Vulnerability Archive. AIX 10 vulnerabilities[6 remote, 3 local, 1 both] Debian GNU/Linux 13 vulnerabilities[1 remote, 12 local] + 1 Linux kernel vulnerability[1 local] FreeBSD 24 vulnerabilities[12 remote, 9 local, 3 both] HP-UX 25 vulnerabilities[12 remote, 12 local, 1 both] Mandrake Linux 17 vulnerabilities[5 remote, 12 local] + 12 Linux kernel vulnerabilities[5 remote, 7 local] OpenBSD 13 vulnerabilities[7 remote, 5 local, 1 both] Red Hat Linux 28 vulnerabilities[5 remote, 22 local, 1 unknown] + 12 Linux kernel vulnerabilities[6 remote, 6 local] Solaris 38 vulnerabilities[14 remote, 22 local, 2 both] From the above listing one can infer that source licensing is not a primary factor in determining how prone to security flaws a software application will be. Specifically proprietary and Open Source UNIX family operating systems are represented on both the high and low ends of the frequency distribution.

Factors that have been known to influence the security and quality of a software application are practices such as code auditing (peer review), security-minded architecture design, strict software development practices that restrict certain dangerous programming constructs (e.g. using the str* or scanf* family of functions in C) and validation & verification of the design and implementation of the software. Also reducing the focus on deadlines and only shipping when the system the system is in a satisfactory state is important.

Both the Debian and OpenBSD projects exhibit many of the aforementioned characteristics which help explain why they are the Open Source UNIX operating systems with the best security record. Debian's track record is particularly impressive when one realizes that the Debian Potato consists of over 55 million lines of code (compared to RedHat's 30,000,000 lines of code).

The Road To Secure Software

Exploitable security vulnerabilities in a software application are typically evidence of bugs in the design or implementation of the application. Thus the process of writing secure software is an extension of the process behind writing robust, high quality software. Over the years a number of methodolgies have been developed to tackle the problem of producing high quality software in a repeatable manner within time and budgetary constraints. The most successful methodologies have typically involved using the following software quality assurance, validation and verification techniques; formal methods, code audits, design reviews, extensive testing and codified best practices.

1. Formal Methods: One can use formal proofs based on mathematical methods and rigor to verify the correctness of software algorithms. Tools for specifying software using formal techniques exist such as VDM and Z. Z (pronounced 'zed') is a formal specification notation based on set theory and first order predicate logic. VDM stands for "The Vienna Development Method" which consists of a specification language called VDM-SL, rules for data and operation refinement which allow one to establish links between abstract requirements specifications and detailed design specifications down to the level of code, and a proof theory in which rigorous arguments can be conducted about the properties of specified systems and the correctness of design decisions.The previous descriptions were taken from the Z FAQ and the VDM FAQ respectively. A comparison of both specification languages is available in the paper, Understanding the differences between VDM and Z by I.J. Hayes et al.
   
   
2. Code Audits: Reviews of source code by developers other than the author of the code are good ways to catch errors that may have been overlooked by the original developer. Source code audits can vary from informal reviews with little structure to formal code inspections or walkthroughs. Informal reviews typically involve the developer sending the reviewers source code or descriptions of the software for feedback on any bugs or design issues. A walkthrough involves the detailed examination of the source code of the software in question by one or more reviewers. An inspection is a formal process where a detailed examination of the source code is directed by reviewers who act in certain roles. A code inspection is directed by a "moderator", the source code is read by a "reader" and issues are documented by a "scribe".
   
   
3. Testing: The purpose of testing is to find failures. Unfortunately, no known software testing method can discover all possible failures that may occur in a faulty application and metrics to establish such details have not been forthcoming. Thus a correlation between the quality of a software application and the amount of testing it has endured is practically non-existent.
   
   There are various categories of tests including unit, component, system, integration, regression, black-box, and white-box tests. There is some overlap in the aforementioned mentioned testing categories.
   
   Unit testing involves testing small pieces of functionality of the application such as methods, functions or subroutines. In unit testing it is usual for other components that the software unit interacts with to be replaced with stubs or dummy methods. Component tests are similar to unit tests with the exception that dummmy and stub methods are replaced with the actual working versions. Integration testing involves testing related components that communicate with each other while system tests involve testing the entire system after it has been built. System testing is necessary even if extensive unit or component testing has occured because it is possible for seperate subroutines to work individually but fail when invoked sequentialy due to side effects or some error in programmer logic. Regression testing involves the process of ensuring that modifications to a software module, component or system have not introduced errors into the software. A lack of sufficient regression testing is one of the reasons why certain software patches break components that worked prior to installation of the patch.
   
   Black-box testing also called functional testing or specification testing test the behavior of the component or system without requiring knowledge of the internal structure of the software. Black-box testing is typically used to test that software meets its functional requirements. White-box testing also called structural or clear-box testing involves tests that utilize knowledge of the internal structure of the software. White-box testing is useful in ensuring that certain statements in the program are excercised and errors discovered. The existence of code coverage tools aid in discovering what percentages of a system are being excercised by the tests.
   
   More information on testing can be found at the comp.software.testing FAQ .
   
   
4. Design Reviews: The architecture of a software application can be reviewed in a formal process called a design review. In design reviews the developers, domain experts and users examine that the design of the system meets the requirements and that it contains no significant flaws of omission or commission before implementation occurs.
   
   
5. Codified Best Practices: Some programming languages have libraries or language features that are prone to abuse and are thus prohibited in certain disciplined software projects. Functions like strcpy, gets, and scanf in C are examples of library functions that are poorly designed and allow malicious individuals to use buffer overflows or format string attacks to exploit the security vulnerabilities exposed by using these functions. A number of platforms explicitly disallow gets especially since alternatives exist. Programming guidelines for such as those written by Peter Galvin in a Unix Insider article on designing secure software are used by development teams to reduce the likelihood of security vulnerabilities in software applications.

Projects such as the OpenBSD project that utilize most of the aforementioned techniques in developing software typically have a low incidence of security vulnerabilities.

Issues Preventing Development of Secure Open Source Software

One of the assumptions that is typically made about Open Source software is that the availability of source code translates to "peer review" of the software application. However, the anecdotal experience of a number of Open Source developers including John Viega belies this assumption.

The term "peer review" implies an extensive review of the source code of an application by competent parties. Many Open Source projects do not get peer reviewed for a number of reasons including

• complexity of code in addition to a lack of documentation makes it difficult for casual users to understand the code enough to give a proper review
  
  
• developers making improvements to the application typically focus only on the parts of the application that will affect the feature to be added instead of the whole system.
  
  
• ignorance of developers to security concerns.
  
  
• complacency in the belief that since the source is available that it is being reviewed by others.
  
  

Also the lack of interest in unglamorous tasks like documentation and testing amongst Open Source contributors adversely affects quality of the software. However, all of these issues can and are solved in projects with a disciplined software development process, clearly defined roles for the contributers and a semi-structured leadership hierarchy.

Benefits of Open Source to Security-Conscious Users

Despite the fact that source licensing and source code availability are not indicators of the security of a software application, there is still a significant benefit of Open Source to some users concerned about security. Open Source allows experts to audit their software options before making a choice and also in some cases to make improvements without waiting for fixes from the vendor or source code maintainer.

One should note that there are constraints on the feasibility of users auditing the software based on the complexity and size of the code base. For instance, it is unlikely that a user who wants to make a choice of using Linux as a web server for a personal homepage will scrutinize the TCP/IP stack code.

References

1. Frankl, Phylis et al. Choosing a Testing Method to Deliver Reliability. Proceedings of the 19th International Conference on Software Engineering, pp. 68--78, ACM Press, May 1997. < http://citeseer.nj.nec.com/frankl97choosing.html >
   
   
2. Hamlet, Dick. Software Quality, Software Process, and Software Testing. 1994. < http://citeseer.nj.nec.com/hamlet94software.html >
   
   
3. Hayes, I.J., C.B. Jones and J.E. Nicholls. Understanding the differences between VDM and Z. Technical Report UMCS-93-8-1, University of Manchester, Computer Science Dept., 1993. < http://citeseer.nj.nec.com/hayes93understanding.ht ml >
   
   
4. Miller, Todd C. and Theo De Raadt. strlcpy and strlcat - consistent, safe, string copy and concatenation. Proceedings of the 1999 USENIX Annual Technical Conference, FREENIX Track, June 1999. < http://www.usenix.org/events/usenix99/full_papers/ millert/millert_html/ >
   
   
5. Viega, John. The Myth of Open Source Security. Earthweb.com. < http://www.earthweb.com/article/0,,10455_626641,00 .html >
   
   
6. Gonzalez-Barona, Jesus M. et al. Counting Potatoes: The Size of Debian 2.2. < http://people.debian.org/~jgb/debian-counting/coun ting-potatoes/ >
   
   
7. Wheeler, David A. More Than A Gigabuck: Estimating GNU/Linux's Size. < http://www.counterpane.com/crypto-gram-0003.html >
   
   

Acknowledgements

The following people helped in proofreading this article and/or offering suggestions about content: Jon Beckham, Graham Keith Coleman, Chris Bradfield, and David Dagon. © 2002 Dare Obasanjo

Dare Obasanjo contributed this followup to an article entitled The Myth of Open Source Security Revisited that appeared on the website kuro5hin. He writes: "The original article tackled the common misconception amongst users of Open Source Software(OSS) that OSS is a panacea when it comes to creating secure software. The article presented anecdotal evidence taken from an article written by John Viega, the original author of GNU Mailman, to illustrate its point. This article follows up the anecdotal evidence presented in the original paper by providing an analysis of similar software applications, their development methodology and the frequency of the discovery of security vulnerabilities." Read on below for his detailed analysis, especially relevant with the currency of security initiatives in the worlds of both open- and closed-source software.

The Myth of Open Source Security Revisited v2.0 The purpose of this article is to expose the fallacy of the belief in the "inherent security" of Open Source software and instead point to a truer means of ensuring the quality of the security of a piece software is high.

Apples, Oranges, Penguins and Daemons

When performing experiments to confirm a hypothesis on the effect of a particular variable on an event or observable occurence, it is common practice to utilize control groups. In an attempt to establish cause and effect in such experiments, one tries to hold all variables that may affect the outcome constant except for the variable that the experiment is interested in. Comparisons of the security of software created by Open Source processes and software produced in a proprietary manner have typically involved several variables besides development methodology.

A number of articles have been written that compare the security of Open Source development to proprietary development by comparing security vulnerabilities in Microsoft products to those in Open Source products. Noted Open Source pundit, Eric Raymond wrote an article on NewsForge where he compares Microsoft Windows and IIS to Linux, BSD and Apache. In the article, Eric Raymond states that Open Source development implies that "security holes will be infrequent, the compromises they cause will be relatively minor, and fixes will be rapidly developed and deployed." However, upon investigation it is disputable that Linux distributions have less frequent or more minor security vulnerabilities when compared to recent versions of Windows. In fact the belief in the inherent security of Open Source software over proprietary software seems to be the product of a single comparison, Apache versus Microsoft IIS.

There are a number of variables involved when one compares the security of software such as Microsoft Windows operating systems to Open Source UNIX-like operating systems including the disparity in their market share, the requirements and dispensations of their user base, and the differences in system design. To better compare the impact of source code licensing on the security of the software, it is wise to reduce the number of variables that will skew the conclusion. To this effect it is best to compare software with similar system design and user base than comparing software applications that are significantly distinct. The following section analyzes the frequency of the discovery of security vulnerabilities in UNIX-like operating systems including HP-UX, FreeBSD, RedHat Linux, OpenBSD, Solaris, Mandrake Linux, AIX and Debian GNU/Linux.

Security Vulnerability Face-Off

Below is a listing of UNIX and UNIX-like operating systems with the number of security vulnerabilities that were discovered in them in 2001 according to the Security Focus Vulnerability Archive. AIX 10 vulnerabilities[6 remote, 3 local, 1 both] Debian GNU/Linux 13 vulnerabilities[1 remote, 12 local] + 1 Linux kernel vulnerability[1 local] FreeBSD 24 vulnerabilities[12 remote, 9 local, 3 both] HP-UX 25 vulnerabilities[12 remote, 12 local, 1 both] Mandrake Linux 17 vulnerabilities[5 remote, 12 local] + 12 Linux kernel vulnerabilities[5 remote, 7 local] OpenBSD 13 vulnerabilities[7 remote, 5 local, 1 both] Red Hat Linux 28 vulnerabilities[5 remote, 22 local, 1 unknown] + 12 Linux kernel vulnerabilities[6 remote, 6 local] Solaris 38 vulnerabilities[14 remote, 22 local, 2 both] From the above listing one can infer that source licensing is not a primary factor in determining how prone to security flaws a software application will be. Specifically proprietary and Open Source UNIX family operating systems are represented on both the high and low ends of the frequency distribution.

Factors that have been known to influence the security and quality of a software application are practices such as code auditing (peer review), security-minded architecture design, strict software development practices that restrict certain dangerous programming constructs (e.g. using the str* or scanf* family of functions in C) and validation & verification of the design and implementation of the software. Also reducing the focus on deadlines and only shipping when the system the system is in a satisfactory state is important.

Both the Debian and OpenBSD projects exhibit many of the aforementioned characteristics which help explain why they are the Open Source UNIX operating systems with the best security record. Debian's track record is particularly impressive when one realizes that the Debian Potato consists of over 55 million lines of code (compared to RedHat's 30,000,000 lines of code).

The Road To Secure Software

Exploitable security vulnerabilities in a software application are typically evidence of bugs in the design or implementation of the application. Thus the process of writing secure software is an extension of the process behind writing robust, high quality software. Over the years a number of methodolgies have been developed to tackle the problem of producing high quality software in a repeatable manner within time and budgetary constraints. The most successful methodologies have typically involved using the following software quality assurance, validation and verification techniques; formal methods, code audits, design reviews, extensive testing and codified best practices.

1. Formal Methods: One can use formal proofs based on mathematical methods and rigor to verify the correctness of software algorithms. Tools for specifying software using formal techniques exist such as VDM and Z. Z (pronounced 'zed') is a formal specification notation based on set theory and first order predicate logic. VDM stands for "The Vienna Development Method" which consists of a specification language called VDM-SL, rules for data and operation refinement which allow one to establish links between abstract requirements specifications and detailed design specifications down to the level of code, and a proof theory in which rigorous arguments can be conducted about the properties of specified systems and the correctness of design decisions.The previous descriptions were taken from the Z FAQ and the VDM FAQ respectively. A comparison of both specification languages is available in the paper, Understanding the differences between VDM and Z by I.J. Hayes et al.
   
   
2. Code Audits: Reviews of source code by developers other than the author of the code are good ways to catch errors that may have been overlooked by the original developer. Source code audits can vary from informal reviews with little structure to formal code inspections or walkthroughs. Informal reviews typically involve the developer sending the reviewers source code or descriptions of the software for feedback on any bugs or design issues. A walkthrough involves the detailed examination of the source code of the software in question by one or more reviewers. An inspection is a formal process where a detailed examination of the source code is directed by reviewers who act in certain roles. A code inspection is directed by a "moderator", the source code is read by a "reader" and issues are documented by a "scribe".
   
   
3. Testing: The purpose of testing is to find failures. Unfortunately, no known software testing method can discover all possible failures that may occur in a faulty application and metrics to establish such details have not been forthcoming. Thus a correlation between the quality of a software application and the amount of testing it has endured is practically non-existent.
   
   There are various categories of tests including unit, component, system, integration, regression, black-box, and white-box tests. There is some overlap in the aforementioned mentioned testing categories.
   
   Unit testing involves testing small pieces of functionality of the application such as methods, functions or subroutines. In unit testing it is usual for other components that the software unit interacts with to be replaced with stubs or dummy methods. Component tests are similar to unit tests with the exception that dummmy and stub methods are replaced with the actual working versions. Integration testing involves testing related components that communicate with each other while system tests involve testing the entire system after it has been built. System testing is necessary even if extensive unit or component testing has occured because it is possible for seperate subroutines to work individually but fail when invoked sequentialy due to side effects or some error in programmer logic. Regression testing involves the process of ensuring that modifications to a software module, component or system have not introduced errors into the software. A lack of sufficient regression testing is one of the reasons why certain software patches break components that worked prior to installation of the patch.
   
   Black-box testing also called functional testing or specification testing test the behavior of the component or system without requiring knowledge of the internal structure of the software. Black-box testing is typically used to test that software meets its functional requirements. White-box testing also called structural or clear-box testing involves tests that utilize knowledge of the internal structure of the software. White-box testing is useful in ensuring that certain statements in the program are excercised and errors discovered. The existence of code coverage tools aid in discovering what percentages of a system are being excercised by the tests.
   
   More information on testing can be found at the comp.software.testing FAQ .
   
   
4. Design Reviews: The architecture of a software application can be reviewed in a formal process called a design review. In design reviews the developers, domain experts and users examine that the design of the system meets the requirements and that it contains no significant flaws of omission or commission before implementation occurs.
   
   
5. Codified Best Practices: Some programming languages have libraries or language features that are prone to abuse and are thus prohibited in certain disciplined software projects. Functions like strcpy, gets, and scanf in C are examples of library functions that are poorly designed and allow malicious individuals to use buffer overflows or format string attacks to exploit the security vulnerabilities exposed by using these functions. A number of platforms explicitly disallow gets especially since alternatives exist. Programming guidelines for such as those written by Peter Galvin in a Unix Insider article on designing secure software are used by development teams to reduce the likelihood of security vulnerabilities in software applications.

Projects such as the OpenBSD project that utilize most of the aforementioned techniques in developing software typically have a low incidence of security vulnerabilities.

Issues Preventing Development of Secure Open Source Software

One of the assumptions that is typically made about Open Source software is that the availability of source code translates to "peer review" of the software application. However, the anecdotal experience of a number of Open Source developers including John Viega belies this assumption.

The term "peer review" implies an extensive review of the source code of an application by competent parties. Many Open Source projects do not get peer reviewed for a number of reasons including

• complexity of code in addition to a lack of documentation makes it difficult for casual users to understand the code enough to give a proper review
  
  
• developers making improvements to the application typically focus only on the parts of the application that will affect the feature to be added instead of the whole system.
  
  
• ignorance of developers to security concerns.
  
  
• complacency in the belief that since the source is available that it is being reviewed by others.
  
  

Also the lack of interest in unglamorous tasks like documentation and testing amongst Open Source contributors adversely affects quality of the software. However, all of these issues can and are solved in projects with a disciplined software development process, clearly defined roles for the contributers and a semi-structured leadership hierarchy.

Benefits of Open Source to Security-Conscious Users

Despite the fact that source licensing and source code availability are not indicators of the security of a software application, there is still a significant benefit of Open Source to some users concerned about security. Open Source allows experts to audit their software options before making a choice and also in some cases to make improvements without waiting for fixes from the vendor or source code maintainer.

One should note that there are constraints on the feasibility of users auditing the software based on the complexity and size of the code base. For instance, it is unlikely that a user who wants to make a choice of using Linux as a web server for a personal homepage will scrutinize the TCP/IP stack code.

References

1. Frankl, Phylis et al. Choosing a Testing Method to Deliver Reliability. Proceedings of the 19th International Conference on Software Engineering, pp. 68--78, ACM Press, May 1997. < http://citeseer.nj.nec.com/frankl97choosing.html >
   
   
2. Hamlet, Dick. Software Quality, Software Process, and Software Testing. 1994. < http://citeseer.nj.nec.com/hamlet94software.html >
   
   
3. Hayes, I.J., C.B. Jones and J.E. Nicholls. Understanding the differences between VDM and Z. Technical Report UMCS-93-8-1, University of Manchester, Computer Science Dept., 1993. < http://citeseer.nj.nec.com/hayes93understanding.ht ml >
   
   
4. Miller, Todd C. and Theo De Raadt. strlcpy and strlcat - consistent, safe, string copy and concatenation. Proceedings of the 1999 USENIX Annual Technical Conference, FREENIX Track, June 1999. < http://www.usenix.org/events/usenix99/full_papers/ millert/millert_html/ >
   
   
5. Viega, John. The Myth of Open Source Security. Earthweb.com. < http://www.earthweb.com/article/0,,10455_626641,00 .html >
   
   
6. Gonzalez-Barona, Jesus M. et al. Counting Potatoes: The Size of Debian 2.2. < http://people.debian.org/~jgb/debian-counting/coun ting-potatoes/ >
   
   
7. Wheeler, David A. More Than A Gigabuck: Estimating GNU/Linux's Size. < http://www.counterpane.com/crypto-gram-0003.html >
   
   

Acknowledgements

The following people helped in proofreading this article and/or offering suggestions about content: Jon Beckham, Graham Keith Coleman, Chris Bradfield, and David Dagon. © 2002 Dare Obasanjo

gary.flake asks: "My group at NECI is in need of a machine that can address 64GB of ram in a single process. This means we need 64-bit addressing. We'd prefer to go with a Linux solution because all of our development is under Linux. We've spec'ed out some reasonable machines (Dell can do 32GB, and Compaq can do 64GB) but they seem to be lacking in that they can only be loaded up with 4 x 800Mhz Itaniums. We would really like to have more processing power (2 Ghz x 4 would be a dream). Does anyone know of any monster Itanium machines that will meet our needs? (Please, no Alphas). Finding such a beast is harder than you'd think."

Tetard asks: "Long live the pipe! Since the `|' was invented by Doug McIlroy in 1973, has there ever been a more effective way of reusing tools and connecting data ? The mouse is a device of the Beatles era; Rather than try and provoke nostalgia in the older ones among us, I'm asking myself, as are others: when we don't try to reinvent the wheel, or at least improve it, why must we try and copy it every time ? Xerox PARC exposed us to WIMPs and we haven't done better: some innovation, some plastic surgery -- but no "paradigm shift" -- where's the creative destruction that will take us further ? Graphical component programming is turning us into click-happy bonobos^H^H^Hchimpanzees, as we fail to find new ways to manage and connect richer data streams. My web designer friends are damaged for life because of mice, and yet we persist... Where do we go from here ? If we ever invent the graphical pipe, let if have keyboard shortcuts." Yes, you've probably seen a similar question to this run by Ask Slashdot before, but this time I'm wondering if maybe we need new input devices before the WIMP paradigm is replaced with something better. Might any of you have ideas on what form these input devices might take?

For those interested, here are the previous stories that have handled this type of question:

• GUI Research - Is it Still Being Done?
• Pipes In GUI's?
• Pipes in GUI's? (Redux)

So what it will take to break us out of the WIMP box (or prison, depending on your bias), maybe new input devices would do it, but quite frankly, I wouldn't be surprised if a 3D interface might be another route (it would possibly spark interest in designing a new input device that would work better with 3D interfaces, or maybe data-gloves could serve this purpose?). Going on a limb, maybe this guy might just be the ticket.

A little while ago you asked Forth (and now colorForth) originator Chuck Moore about his languages, the multi-core chips he's been designing, and the future of computer languages -- now he's gotten back with answers well worth reading, from how to allocate computing resources on chips and in programs, to what sort of (color) vision it takes to program effectively. Thanks, Chuck!

FFP, Combinator Calculus and Parallel Forth
by Baldrson

In his 1977 Turing Lecture, John Backus challenged computists to break free of what he called "the von Neumann bottleneck". One of the offshoots of that challenge was work on massive parallelism based on combinator calculus a branch of mathematics that is far closer to Forth's formalism than parameter list systems (which are more or less lambda calculus derivatives).

The prolific Forth afficionado Philip Koopman did some work on combinator reduction related to Forth but seems not to have followed through with implementations that realize the potential for massive parallelism that were pursued in the early 1980s by adherents of Backus's Formal Functional Programming paradigm. Given recent advances in hierarchical grammar compression algorithms, such as SEQUITUR, that are one step away from producing combinator programs as their output, and your own statements that Forth programming consists largely of compressing idiomatic sequences, it seems Backus's original challenge to create massively parallel Formal Functional Programming machines in hardware are near realization with your new chips -- lacking only some mapping of the early work on combinator reduction machines.

It is almost certainly the case you are aware of the relationship between combinator reduction machines and Forth machines -- and of Backus's challenge. What have you been doing toward the end of unifying these two branches of endeavor so that the software engineering advantages sought by Backus are actualized by Forth machines of your recent designs?

Chuck Moore: What can I say? Backus did not mention Forth in his lecture. He probably didn't know of it then. Yet Forth addresses many of his criticisms of conventional languages.

He thinks a language needs or benefits from a formal specification. I grew up worshiping Principia Mathematica 'till I learned how Goedel refuted it. The result is that I distrust formal representations. For example, the ANSII Forth standard does not describe Forth, but a language with the same name.

Yes, I am struck by the duality between Lisp and Lambda Calculus vs. Forth and postfix. But I am not impressed by the productivity of functional languages. Even as research tools, they have failed to live up to their promise. By that I mean to do something with computers that I couldn't do more easily in Forth.

I designed the memory for the c18 to occupy the same area as the processor. This means small, fast and smart. c18 can respond to a bus request by fetching from its memory, accessing off-chip or performing a calculation. The 25x avoids the von Neumann bottleneck by making up to 27 memory accesses at the same time (2 off-chip). And its multiple buses do not substitute a network bottleneck for a memory one.

Standard code will be in the ROM of each computer. How this is customized in RAM and the computers assigned tasks is left to the ingenuity of the programmer, not a compiler. Automatically generated or factored code has never impressed me. Nor has automatic place and route for circuit boards or silicon. They are both an order-of-magnitude from human performance. Because humans understand the problem, judge the results and cheat as required.

Marginalizing of the blind
by Medievalist

When I built my first Internet node, the web did not yet exist, and one of the amazing things about the Internet was how friendly it was to the blind.

Now, with some computer experts estimating that over 50% of the Internet is incomprehensible to braille interfaces, and most computer operating systems devolving to caveman interfaces ("point at the pretty pictures and grunt") we seem to be ready to take the next step - disenfranchising the merely color-blind.

I realize that colorforth is not inherently discriminatory, in that there are a great many other languages that can be used to do the same work. The web is also not inherently discriminatory, because it does not force site designers to design pages as stupidly as, for example, Hewlett-Packard.

Would you care to comment on the situation, speaking as a tool designer? How would you feel if a talented programmer were unable to get a job due to a requirement for colored sight?

CM: I'm amazed at how effective blind programmers can be. I rely so strongly upon seeing the code that it's hard to imagine listening to it. Yet I know it can be done. Not being color-blind, it's hard to appreciate the degree of information loss. But it's less than being blind.

My goal is to develop tools that augment my abilities. If others can use them, fine. It would be foolish to lose an opportunity to explore or excel just to conform to some equalitarian philosophy. Too often our culture seeks the lowest common denominator.

20-20 vision is required for fighter pilots. I have no qualms about requiring color vision for programmers. Everyone does not need to be a programmer.

But in fact, color is merely a property of words that helps to distinguish them. As is intensity, size, font, volume and tone. I'm sure colorForth will be translated into these other representations. I, myself, will be exploring spoken colorForth. (As soon as I can decipher PC sound cards.)

Massively Parallel Computing
by PureFiction

The 25X system reminded me of IBM's Blue Gene computer, where a large number of inexpensive CPU cores are placed on a single chip.

The biggest problem in dealing with a large number of small cores lies in the programming. I.e. how do you design and code a program that can utilize a thousand cores efficiently for some kind of operation? This goes beyond multi-threading into an entirely different kind of program organization and execution.

Do you see Forth (or future extensions to Forth) as a solution to this kind of problem? Does 25X dream of scaling to the magnitude that IBM envisions for Blue Gene? Do you think massively parallel computing with inexpensive, expendable cores clustered on cheap dies will hit the desktop or power-user market, or forever be constrained to research?

CM: Forth is a massively pragmatic language: do whatever you can to solve a problem. Its strength is in the ease of violating whatever rules it has. The 25x is similarly pragmatic. I don't know how to program it yet, but I'm confident I can. It's just another level of factoring.

The parallelism provided by the 25x has a different slant from other parallel architectures. The computers are not identical. I expect many will have different ROM and different interface to the real world. This asymmetry is a powerful clue as to how applications will be factored.

A 10x10 array of 25x chips is an easy board to build. At 50 Watts, it needs as much power as a notebook. That's 2500 computers providing 6M Mips. I can't imagine programming them any other way than Forth.

The advantage of Forth in this kind of context is that it scales. Forth is the machine language, Forth is the high-level language, Forth is the task-control language, Forth is the supervisory language. Each of these has a different vocabulary, but they share syntax, compiler and programmer skills.

Back to the array of 25x chips. Each chip could be on a vertical and horizontal serial bus with 10 others. A half-duplex bus requires a computer to manage, so that accounts for 200 computers. Now whatever the application, data must be provided. Say 1GHz Ethernet. Data (and program) is received, distributed and crunched. The assignment and coding of computers follows the data flow. Results are routed back to Ethernet, or displayed or whatever. It's a nice programming problem, well within the ability of a human to organize.

Will this ever reach the mass market? I don't know.

The direction of 25x Microcomputer...
by Midnight Ryder

The 25x concept looks like it could really a damned interesting idea. But one of the questions in my mind is where you want to head with it? Is this something that is to be used for very specialized research and scientific applications, or is this something that you envision for a general 'desktop' computer for normal people eventually?

Secondly, if you are considering the 25x for a desktop machine that would be accessible by people that aren't full-time geeks, what about software? Forth is a lost development art for many people (It's probably been 10 years since I even looked at any Forth code) and porting current C and C++ application would be impossible - or would it? Is there a potential way to minimize the 'pain' of completely re-writing a C++ app to colorForth for the 25x machines, which could help to speed adoption of a platform?

CM: At this stage the 25x is a solution looking for a problem. It's an infinite supply of free Mips. There's no obligation to use them all, or even very many. But they can effectively be used to eliminate hardware. To bit-bang what would otherwise need a controller. So if you want video or audio or radio or ...

The first applications will doubtless be embedded. These offer greater volume, less software and less market resistance than a general-purpose computer. I see 25x reaching the desktop as dedicated appliances rather than universal golems.

I'm not interested in recoding C applications. My experience indicates that most applications are hardware-dependent. The 25x is as large a change in the hardware environment as I can imagine. This changes the program so much it might as well be rethought and recoded. The most efficient way to do that is Forth.

Forth is a simple, interactive language. Its learning curve is steep with a long tail. You can be productive in a day/week. This depends only on how long it takes to memorize pre-existing words. Good documentation and management helps mightily. I'd rather train programmers than fight code translators.

That said, there are those who look at the mountain of existing applications and want to mine it. C to Forth translators exist and with some pre/post editing could produce code for the c18 core. How to distribute the application among 25 tiny computers would be a good thesis.

Quick question
by jd

I have often conjectured that multi-threaded processors (ie: processors that can store multiple sets of internal states, and switch between them) could be useful, as the bottleneck moves from the processor core to communications and dragging stuff out of memory.

(If you could microcode the "instruction set", all the better. A parallel processor array can become an entire Object Oriented program, with each instance stored as a "thread" on a given processor. You could then run a program without ever touching main memory at all.)

I'm sure there are neater solutions, though, to the problems of how to make a parallel array useful, have it communicate efficiently, and yet not die from boredom with a hundred wait-states until RAM catches up.

What approach did you take, to solve these problems, and how do you see that approach changing as your parallel system & Forth language evolve?

CM: The 25x could implement a multi-thread application nicely indeed. Except that most applications expect more memory that a c18 core has. Whereupon memory remains the bottleneck.

It's important to choose problems and solutions that avoid using off-chip memory. Even so, with 25 computers to support, I expect that every memory cycle will be utilized. The computer controlling memory can be smart about priorities and about anticipating requirements. For example, it could guarantee enough access to support display computers.

And the nice thing about memory-mapped communication is that a computer need not be aware of its environment. It's an ordinary Forth program accessing data asynchronously. Delays are invisible, as is synchronization. Of course, due care is required to avoid lock-up loops.

These conjectures are fun. But in a year we'll have real applications to review. And a much better appreciation of the advantages and drawbacks of so many tiny computers.

Programming languages...
by Midnight Ryder

This one would probably require a bit more time to answer than you probably have available, but a quick rundown would be cool: Where do you see programming languages headed -vs- where do you think they SHOULD be headed?

Java, C#, and some of the other 'newer' languages seem to be a far cry from Fourth, but are languages headed (in your opinion) in the proper direction?

CM: I've been bemused with the preoccupation of new languages with text processing. I've been accused of not providing string operators in both Forth and colorForth. Indeed, I haven't because I don't use them. Editing a file to pass on to another program never struck me as productive. That's one reason I chose pre-parsed source, to break the dependence upon strings and text processors.

Languages are evolving, as evidenced by the new ones that arise. But as with natural evolution, the process is not directed. There is no goal to approach nor any reward for approaching it. But whatever progress you might perceive, I don't. New languages seem only to propose new syntax for tired semantics.

These languages are all infix. Which is extraordinarily clumsy for anything but arithmetic expressions. And even those are comfortable only because we learned them in Algebra 101. Do you remember the learning curve?

Does everyone really think that 50 years into the computer age we have hit upon the ultimate language? As more and more C code accumulates, will it ever be replaced? Are we doomed to stumble over increasingly bloated code forever? Are we expecting computers to program themselves and thus save civilization?

I'm locked in the Forth paradigm. I see it as the ideal programming language. If it had a flaw, I'd correct it. colorForth uses pre-parsed source to speed and simplify compilation. This solves a non-problem, but it's neat and worth exploring. At least it proves I haven't gone to sleep.

What about memory protection?
by jcr

From the web pages, I don't see any mention of access control.

Can this processor be used in a multi-user, general-purpose mode?

CM: If you had a chip, you'd physically control access to it. It doesn't make sense for another person to share your chip. He can get his own. Certainly an individual c18 has too little memory to multi-task. And I doubt 25 computers could run 25 tasks.

But the 25 computers can certainly perform more than one task. They have to share resources: communication buses, off-chip memory and interfaces. Access is negotiated by the computer in charge of the resource. There is no hardware protection. Memory protection can be provided by the access computer. But I prefer software that is correct by design.

Communication with other computers, via internal or external buses, is subject to the usual problems of scheduling, routing and authentication. Internally, at least, my goal is to minimize delay rather than attempt protection. I anticipate spectacular crashes while software is developed. (Have you ever crashed 2500 computers?)

Where is forth going?
by JanneM

I learned forth early on in my programming career; it was very memory and CPU efficient, something that was important on early microcomputers. It was also a great deal of fun (though far less fun to try and understand what you wrote a week earlier...). Today, even small, cheap microcontrollers are able to run fairly sophisticated programs, and it is far easier to cross-compile stuff on a 'big' machine and just drop the compiled code onto the development board.

Forth has (in my eyes) always been about small and efficient. Today, though, embedded apps are more likely to be written in C than in forth, and the "OS as part to the language" thing isn't as compelling today as it was in the eighties. Where is forth being used today, and where do you see it going in the future?

CM: Forth is being used today as it always has been. In resource-constrained applications. I think they will always exist. I'm creating some with the tiny c18 computers in the 25x. I imagine molecular computers will be limited when they first appear.

Personally, I don't mind losing a mature market that can afford abundant resources. Such applications aren't as much fun. But Forth isn't restricted to small applications. Even with huge memories and fast processors, small, reliable programs have an advantage.

The major project cost has become software, to the dismay of managers everywhere. On-time, bug-free software is the grail. Forth doesn't guarantee it, but sure makes it easier. Will this ever be convincingly demonstrated? Will management ever value results over procedures?

The currently popular language is selected by uninformed users. The only thing in favor of such democratic choice is that it's better than any other. But why would anyone want to debug 1M lines of code instead of 10K?

What's the next Big Computational Hurdle?
by DG

Now that sub-$1k computers are running in the GHz range, it seems that all the computational tasks on a common desktop system are not processor-bound.

3D, rendered-on-the-fly games get well over 30 frames per second at insanely high resolutions and levels of detail. The most bloated and poorly-written office software scrolls though huge documents and recalculates massive spreadsheets in a snap. Compiling the Linux kernel can be done in less than 5 minutes. And so on.

It seems that the limiting speed of modern computers is off the processor, in IO. What then, do you forsee coming down the pike that requires more processor power than we have today? What's the underlying goal you intend to solve with your work?

CM: Memory is cheap. I don't mind wasting memory as long as it's not full of code that has to be debugged.

Likewise, Mips are cheap. The trick is to find productive ways to waste them. A Pentium waiting for a keystroke isn't very clever.

So here's a huge pool of Mips. What can you do with them? Voice recognition comes instantly to mind. Image recognition close behind. The brain deploys substantial resources to these tasks, so I suspect a computer must.

IO is indeed a bottleneck, but not in principle. If you can't get data from the camera to the computer, combine them. Put the image recognition algorithms in the camera. Analyse, reduce, compress data at the source. Meanwhile, it helps to have multiple paths off-chip.

revolutionary
by rnd

What is the most revolutionary (i.e., it is scoffed at by those in control/power) idea in the software industry today? Explain how this idea will eventually win out and revolutionize software as we know it.

CM: Forth! But then I haven't been out looking for revolutionary ideas. I like the phrase Baldrson used above: compressing ideomatic sequences. If you do this recursively, you obtain a optimal representation. I see no way to get a more compact, clear, reliable statement of a problem/solution.

Forth clearly revolutionizes software as most know it. It could lead to efficient, reliable applications. But that won't happen. A mainstay of our economy is the employment of programmers. A winnowing by factor 100 is in no one's interest. Not the programmers, the companies, the government. To keep those programmers busy requires clumsy languages and bugs to chase.

I don't have to be glib or cynical. Those are facts of life. Society must cope with them. But I don't have to. Nor you. There are niches in which you can be creative, productive, inspired. Not everyone can be so lucky.

Forth as intermediate language
by Ed Avis

Many high-level languages compile into C code, which is then compiled with gcc or whatever. Do any use Forth instead? I understand Forth is a stack-based language: doesn't that present problems when compiling for CPUs that mostly work using registers?

CM: I remember my shock at learning that Fortran compiled into Assembler, that then had to be assembled. A language that can be translated into another is clearly unnecessary. Truely different languages cannot be translated: C into Lisp.

Forth would make a fine intermediate language. But why have an intermediate language? It introduces another layer of confusion and inefficiency between the programmer and her computer. Macros were invented to support compiling directly to machine code.

Stacks are a compiler-friendly construct. Every compiler has to use one to translate infix notation to postfix. If this stack solution has to be assigned to registers, it's an extra step. Forth uses stacks explicitly and avoids the whole subject.

Register-based CPUs have more problems than just the complexity of their compilers. Their instructions must contain register addresses, which makes them longer and programs bigger. And it is rare that every register can be used in every instruction.

Moreover registers need to be optimized. After assigning system registers, performance depends on how well the remaining registers are handled. Compilers can optimize infix expressions better than humans. But such expressions are no longer the preferred means of doing arithmetic. DSPs and super-computers integrate difference equations.

Design guidelines encourage code with many subroutine calls each with only a few arguments. This is the style Forth employs. But it plays havoc with optimization, since register usage must be resolved before each call. So apart from being unnecessary and difficult, optimization has no effect on good code.

Nate Truitt writes "Research published in the journal "Artificial Life" reveals that autonomous agents -- software entities capable of independent action in dynamic, unpredictable environments -- can communicate with one another as they work to achieve a common goal. They actually develop a language of sorts...." It doesn't look like the paper is available yet, but I know that Giles has been willing to send copies by email of his other papers.

Zarquon writes: "NEC has agreed to pay licensing fees and royalties to Rambus for production of SDRAM, according to this TechWeb article. They're the fourth company to give in to Rambus; Hitachi, Toshiba, and Oki have already been signed to similar agreements. If you're unfamiliar with the Rambus patent fiasco, LostCircuits has a good synopsis of the situation." Ah, yes -- beat them in the courtroom, not the marketplace.

MEconomy asks: "I'm the CTO of a commercial entity developing a technology that we plan to 100% open source. I'm looking for other options, opinions, and bulletproof arguments to take to Open Source-leery business types to convince them to release earlier (my preference) rather than later." While releasing the source early is a good thing, it can be released too early. I would never release the code to a project that wasn't in a runnable state, and would honestly consider holding off source releases until the project was at least in the "beta" stage. What do you think?

"I am very curious what peoples' thoughts are on the tradeoffs (business risks, community reputation, etc...) between:

1. immediate release of the source code at product launch,
2. waiting until the technology has acquired a large enough user base to insure a competitor won't just 'take the code and run',
3. commit to full source code release and release piecemeal (ala ZeroKnowledge),
4. a short-term (~1yr) closed-release to mutually trusted third parties (e.g. EFF),
5. placing the code in a provable timed-release escrow."

   [Update by nik]: Accepted practice seems to be to wait until about a year after you're bought by Sun. . .

Gary William Flake writes "A new study by Inktomi and NEC Research Institute show that there is at least one billion unique indexable Web pages on the internet. The details are pretty interesting; for example, Apache dominates the server market. "

Pierre Phaneuf hooked us up with all the juicy news from AMD. He writes " AMD announced that it's next generation processors K7 will be packaged in a cartridge (called Slot A) physically compatible with Intel's Slot 1.

Also, using Digital I/O bus technology, motherboards with system bus going at a cool 333MHz would be able to support both AMD and (Slot A-packaged) Alpha chips, with a simple BIOS change (which could be flash). This will also put the Alpha in the commodity market, using large-volume motherboards and chipsets made for the Intel-compatible K7."