Designing a Programming Language For Embeddability

CowboyRobot writes "The creators of the Lua language describe the process of designing a new language and the constraints that certain parameters, specifically embeddability, place on the process. 'Many languages (not necessarily scripting languages) support extending through an FFI (foreign function interface). An FFI is not enough to allow a function in the system language to do all that a function in the script can do. Nevertheless, in practice FFI covers most common needs for extending, such as access to external libraries and system calls. Embedding, on the other hand, is harder to support, because it usually demands closer integration between the host program and the script, and an FFI alone does not suffice.'"

Comments

Re:Oh good

[pem]

It's 18 years old. If you don't care to know anything about existing languages, why would having newer ones bother you?

If they are going to, might as well do it right.

[pavon]

Since it's a given that people are going to be creating new languages, I certainly hope they listen to what Roberto has to say. Lua is a joy to embed into applications compared to python and others.

Forth

[smoothnorman]

Forth http://en.wikipedia.org/wiki/Forth_(programming_language)/ was designed to run on (albeit early) embedded platforms. Extensible, easy to learn, easy to implement. I guess it's just the RPN that scares folks off? ("no you old fool! modern processors are optimized away from stack intensive use!")

Re:Forth

[VortexCortex]

Wrong kind of "embedded" -- The Lua devs meant "easily accessed from within the host language ( C / C++ )".

Re:Oh good

[rrohbeck]

Bah. Newfangled crap. I'll stick to COBOL, thank you very much.

Lisp

[betterunixthanunix]

Why jump through so many hoops to get an embeddable language, when you could use the same language to write the "host program" as you use to write scripts for it? Lisp is an example of such a concept: compiled Lisp programs still have the ability to interpret Lisp code, and hence the most sensible scripting language for a programming written in Lisp is Lisp itself. Lisp is not the only language with this property, but it is one of the most prominent examples of such a language.

Re:Lisp

[Darinbob]

Because the host language may be a conventional language, designed to compile efficiently, run in small amounts of memory, has speed critical sections, etc. The embedded language on the other hand is used for doing stuff that wasn't thought of in the design phase and where it's simpler to add a small script than to add a new feature to the program. Generally what happens is that you're tying together a high level language with a low level language.

For instance, take a boot loader. It is written to fit into a small area of flash or just a few blocks of mass storage. So it needs to be small and tight. In manufacturing you'd like to run a set of tests and the boot loader provides some simple commands to call that test various features. But maybe you want a particular test that the built in commands won't do, for example a stress test of memory. The default method is to have development write a new command, check it into source code control, have QA run through it, approve it with the release schedule, then release it to manufacturing. Alternately one could just write a quick script and use that instead if the boot loader supported a scripting language, and manufacturing could manage this themselves without begging for R&D resources. Every time a bug occurs in the process someone could write a new regression test using the scripting language to check for the bug and add the script to a growing collection.

Now if we could somehow get fast/efficient/tiny host programs written in Lisp (which is indeed possible) and we manage to find a collection of developers who understand it (which is harder), then it makes sense to have the scripting language the same as the host language. But in the mean time most host programs are written in things like C or C++. Even if you have a high level interpreted language like Java that you wrote your host program in, it is very difficult to parse/interpret textual Java programs.

Even if your host program is in Lisp it will be difficult to isolate any textual Lisp scripts from the host program itself; you don't want a buggy script to cause havoc with the host program. Most Lisp systems I've seen that allow extra scripting basically use "eval" which is not very safe. So you still need to put up a solid wall with a well defined set of methods to allow a script to run in its own sandbox while interacting occasionally with the host program.

Self-morphing languages

[oldhack]

What's the track record for self-morphing languages like Lua and Forth? I just skimmed a Scala book, and it scares me.

I can see niches like producing DSLs, but smells like bad news for general application/system programming.

Tcl

[jdwoods]

http://sourceforge.net/projects/tcl/ says: "Tool Command Language (Tcl) is an interpreted language and very portable interpreter for that language. Tcl is embeddable and extensible, and has been widely used since its creation in 1988 by John Ousterhout. See http://www.tcl.tk/ for more info." Another good source of information on Tcl is http://wiki.tcl.tk/

Tcl functions well as glue between applications. Some folks know Tcl but call it "Expect" and may not realize Expect is simply Tcl plus an extension. Another extension, Tk, provides GUI features and is so powerful and popular that it's commonly used from other languages. Bindings exist for several other languages, including Ada (called TASH), Perl, Python (called Tkinter), Ruby, and Common Lisp.

Tcl is used by many people and companies (large and small). Cisco network gear uses embedded Tcl for automating tasks. Oracle uses Tcl for automating testing. The Fortune100 company where I work (but I am not a spokesman, so I won't name them) pays me to write and maintain an application written in Tcl to process payments for many thousands of customers totaling millions of dollars every day for payment through banks and the Federal Reserve.

Tcl is FOSS, but a very popular build is ActiveTcl from ActiveState. http://www.activestate.com/activetcl/