Extensible Programming for the 21st Century

Anonymous Cowardly Lion writes "An interesting article written by a professor at the University of Toronto argues that next-generation programming systems will combine compilers, linkers, debuggers, and that other tools will be plugin frameworks [mirror], rather than monolithic applications. Programmers will be able to extend the syntax of programming languages, and programs will be stored as XML documents so that programmers can represent and process data and meta-data uniformly. It's a very insightful and thought-provoking read. Is this going to be the next generation of extensible programming?"

Comments

Rather Grand Conclusion ?

[beatleadam]

Instead of treating each new idea as a special case, they allow programmers to say what they want to, when they want to, as they want to.

Is this not the Ultimate goal of programming? The Holy Grail of programming perhaps?

Let's see here..

[k98sven]

If I recall correctly,
Fourth-Generation languages was going to be the future of programming back in the early 80's?
(Machine code, Fortran/Basic-type languages and Pascal/C-type languages being the supposed first, second and third generations, IIRC)

Then in the early 90's.. OOP was going to save the world. Not that it hasn't had impact, but it certainly hasn't fundamentally changed things.

And now it's XML that's going to save the programmers, while the old-timers whine that we should all really be using Lisp.

Not that I'm a computer-language conservative myself, but it's worth pointing out that historically, there has been quite a big discrepancy between which languages the Comp-Sci researchers feel everyone should be using, and the ones which actually are used.

The more things change...

[treerex]

that next-generation programming systems will combine compilers, linkers, debuggers,

...THINK Pascal (for the Mac) was doing this almost 20 years ago: the editor served as the front end to the compiler --- so the syntax highlighting in the THINK Pascal editor was driven by the lexer (really was the lexer): you knew about syntax errors immediately. The debugger was fully integrated into the environment. It was really sweet, and probably one of the best programming environments ever written.

and that other tools will be plugin frameworks

Like Unix pipes and Eclipse?

Tomorrow arrived yesterday and appears today.

Re:been done

[ron_ivi]

Agreed, and even mentioned in TFA:

3. You don't need XML to do this.

Scheme proves by example that everything described in this article could have been done twenty years ago, and could be done today without XML.

And IMHO lisp's syntax has always had a nicer structure than XML's repetitive redundancy.

_<whatever>
__<you> want to do in <xml>xml</xml>
__</you>
_</whatever>

is nothing but a set of s-expressions that read much nicer in a lisp-like syntax:

(whatever
_(you want to do in (xml xml)
_)
)

IMveryHO the big failure of the lisp guys of old was that they were so proud of how many ')' they could put next to each other that it made their code harder to read than necessary. I bet XML would have failed too if it were commonly written

<whatever>
_<you> want to do in <xml> xml </xml></you></whatever>

(and yes, the _ are just there for /.'s formatting)

I doubt it.

[srussell]

That would be a departure from what I see happening in softare development today. There seem to be three dominant camps:

• Low level developers. People programming in C; the ones writing Linux and KDE.
• Quasi-low level developers. People programming in Java; the one writing much of the business software right now.
• High level developers. People programming in scripting languages, like Ruby, Python, PHP, JSP, Javascript.

The second group is the most visible, because business loves them. The first group is the second most visible, because -- while it isn't as "hot" a technology in Monster -- most of the software we use is written at this level. I suspect that the third group is the one that will goose the business community in the future, and will probably eclipse the second group. I'd guess that this is a submarine technology; you don't see many job postings for Ruby programmers, but a heck of a lot of software is being written in it. Even more is being written in PHP, JSP, and Python.

I imagine something like Python or Ruby, or some other high-level language that's easy to write software with, coupled with a decent compiler will be the real winner in the near future. Get some type inferrence for one of these languages, and the ability to compile it (as with Parrot), and group two will mostly go away. Java claims to be a more productive language than C because of higher level features; modern scripting languages are even better at increasing productivity, and their only real limitation is their speed, or lack of it. Just as Java eventually overcame the speed issue, so, too, do I expect some future version of a scripting language.

But, maybe Java will hang in there. If you look at Java 1.5, you see a lot of increased syntactic sugar that has usually been only available in languages like Ruby -- I've heard that this was motivated by similar constructions in C#. Perhaps Java or C# will evolve enough syntactic sugar that hacking out code will be as easy as doing so in Ruby. IMO, it'll take a more radical language change than that provided by 1.5; my biggest complaint about Java these days is that it gets in your way; a large chunk of the code you write for any application is infrastructure, and you write it over, and over, and over (anybody else sick of ActionListeners yet?). I'd like to see the typing system changed to type inferrence... but it is possible.

I doubt, however, that software development is going to evolve into choosing black boxes from a set of tools and plugging them into each other, mostly because to do cover all possible jobs, the framework would have to have access to a huge amount of fine-grained tools, and by that point, you might as well just write the code yourself. Look at the size of the Java APIs. How many packages are there? How many classes? How many methods? This is making our lives, as programmers, easier... how?

This is far from a new idea.

[voodoo1man]

Terry Winograd wrote a paper, called Breaking the complexity barrier (again), in 1973 (it is reprinted as the first paper in Barstow, Shrobe, and Sandewall's excellent compilation, Interactive Programming Environments, 1984). In it he described the integrated programming environment of the future, speculated on the role AI would play in it, described the importance of extendable syntax and the need for data-code representation, and noted that all this would need to be deeply and intimately interconnected, all the while taking a technology agnostic view.

Where Wilson goes wrong is in assuming that this kind of environment will be built based on plug-ins. The interrelationships needed between the components to get the required level of functionality are too great. What many people have already noted is that the current Unix environment is in fact based on plug-in development. Editors, debuggers and compilers are modularized as programs, with clean lines of communication between them in the forms of files and streams (which Unix again abstracts to one concept). The limitation of this system lies in the fact that the modules all use their own separate address spaces, and hence each one has to have a private representation of the program. This can't be mitigated by having the separate tools communicate to a central database (this is the most that Wilson's proposal of using XML as the underlying format can accomplish), because then the method of communication would be the limiting factor. Of course, you can use the neutral code-data representation to make the communications between the modules and the database be in terms of sending closures (from reading the paper, I don't think Wilson even considers this), but then you've just designed a single distributed address space, and in the process removed all the encapsulation and modularity advantages of the communication links (not to mention introducing a whole slew of concurrency issues)!

One such integrated system has been built in the past, called Interlisp. Barstow, Shrobe, and Sandewall's book (mentioned above) has a few papers that describe the system, but briefly a few lessons can be distilled from it. First of all, the system itself was an integrated development environment for a dialect of Lisp, where everything was done in one in-core address space: source code (including comments) was represented by data structures in memory, upon which the structure editor (residing in the same address space) operated directly. Code could either be interpreted from the data structure or compiled by the (yes, in-core) compiler. There were several extended packages (besides a Lisp macro-like facility), notably the structure editor and "Conversational LISP," a pseudo-natural language command-prompt parsing system. Although source code (and data) could be serialized to files (there was a sophisticated change-tracking facility that took care of this), the usual way of working was by saving the core image to disk and loading it next session, so the whole environment was persistent. There were hooks for everything from the parser to the compiler to error handling down to the most basic frame-handling code of the stack-based VM, and in order to implement the facilities mentioned above (and some other ones I left out, like the ever-present DWIM automatic error-correction facility) the code took full advantage of them. This caused some trouble when it came to portability of the components and the Interlisp itself (the heavy interdependence caused many problems in bootstrapping the system). Some of these incidents are documented in Barstow et al.'s book, but the Interlisp bootstrapping difficulty has been mentioned in all of the Interlisp porting papers I've read. Unfortunately, I don't think a system with those capabilities can be built with the rescrictions of modularization, since all of the things it did are applicable to programming in any language, and to do them required precisely the