Designing a Programming Language For Embeddability

CowboyRobot writes "The creators of the Lua language describe the process of designing a new language and the constraints that certain parameters, specifically embeddability, place on the process. 'Many languages (not necessarily scripting languages) support extending through an FFI (foreign function interface). An FFI is not enough to allow a function in the system language to do all that a function in the script can do. Nevertheless, in practice FFI covers most common needs for extending, such as access to external libraries and system calls. Embedding, on the other hand, is harder to support, because it usually demands closer integration between the host program and the script, and an FFI alone does not suffice.'"

Comments

Re:Oh good

[pem]

It's 18 years old. If you don't care to know anything about existing languages, why would having newer ones bother you?

If they are going to, might as well do it right.

[pavon]

Since it's a given that people are going to be creating new languages, I certainly hope they listen to what Roberto has to say. Lua is a joy to embed into applications compared to python and others.

Forth

[smoothnorman]

Forth http://en.wikipedia.org/wiki/Forth_(programming_language)/ was designed to run on (albeit early) embedded platforms. Extensible, easy to learn, easy to implement. I guess it's just the RPN that scares folks off? ("no you old fool! modern processors are optimized away from stack intensive use!")

Re:Forth

[nielsm]

Except this is not about languages running on embedded platforms, this is about hosting a language inside an application, for providing e.g. scriptability of a GUI application.

Re:Forth

[VortexCortex]

Wrong kind of "embedded" -- The Lua devs meant "easily accessed from within the host language ( C / C++ )".

Re:Forth

[hoggoth]

Loeliger's book 'Threaded Interpreted Languages' was one of my favorites non-fiction books ever. (Yes, I am that geeky).
Reading how he started with just a few tens of opcodes and chapter by chapter boot-strapped an entire operating environment was like reading Genesis.

Re:Forth

[smoothnorman]

yeah! it along with the later one by Brodie always comes to mind when i'm writing embedded applications (in either of the disparate senses of "embedded". should be a clear way to specify them .. machine-embedded or Gui-bedded ?) http://www.amazon.com/Threaded-Interpretive-Languages-Design-Implementation/dp/007038360X/ref=sr_1_1?ie=UTF8&qid=1305766293&sr=8-1/

Re:Forth

[Darinbob]

I always liked REXX myself, and AREXX on Amiga. Simple, easy to embed, the host program is in control as opposed to the embedded language's loop, integrates with other programs using the same embedded language, etc.

Re:Forth

[sgt scrub]

if your going to write small blazing fast code you need to go old school. http://www.amigacoding.com/index.php/680x0:AsmOne it comes with its own IDE!

Re:Oh good

[rrohbeck]

Bah. Newfangled crap. I'll stick to COBOL, thank you very much.

GRUB?

[rrohbeck]

So what happened to Lua support in GRUB? I saw some pretty nifty things done with that.

Re:Apparently Lua needs some press

[Musc]

Inferior to what?

Lisp

[betterunixthanunix]

Why jump through so many hoops to get an embeddable language, when you could use the same language to write the "host program" as you use to write scripts for it? Lisp is an example of such a concept: compiled Lisp programs still have the ability to interpret Lisp code, and hence the most sensible scripting language for a programming written in Lisp is Lisp itself. Lisp is not the only language with this property, but it is one of the most prominent examples of such a language.

Re:Lisp

[Darinbob]

Because the host language may be a conventional language, designed to compile efficiently, run in small amounts of memory, has speed critical sections, etc. The embedded language on the other hand is used for doing stuff that wasn't thought of in the design phase and where it's simpler to add a small script than to add a new feature to the program. Generally what happens is that you're tying together a high level language with a low level language.

For instance, take a boot loader. It is written to fit into a small area of flash or just a few blocks of mass storage. So it needs to be small and tight. In manufacturing you'd like to run a set of tests and the boot loader provides some simple commands to call that test various features. But maybe you want a particular test that the built in commands won't do, for example a stress test of memory. The default method is to have development write a new command, check it into source code control, have QA run through it, approve it with the release schedule, then release it to manufacturing. Alternately one could just write a quick script and use that instead if the boot loader supported a scripting language, and manufacturing could manage this themselves without begging for R&D resources. Every time a bug occurs in the process someone could write a new regression test using the scripting language to check for the bug and add the script to a growing collection.

Now if we could somehow get fast/efficient/tiny host programs written in Lisp (which is indeed possible) and we manage to find a collection of developers who understand it (which is harder), then it makes sense to have the scripting language the same as the host language. But in the mean time most host programs are written in things like C or C++. Even if you have a high level interpreted language like Java that you wrote your host program in, it is very difficult to parse/interpret textual Java programs.

Even if your host program is in Lisp it will be difficult to isolate any textual Lisp scripts from the host program itself; you don't want a buggy script to cause havoc with the host program. Most Lisp systems I've seen that allow extra scripting basically use "eval" which is not very safe. So you still need to put up a solid wall with a well defined set of methods to allow a script to run in its own sandbox while interacting occasionally with the host program.

Re:Lisp

[Anonymous Coward]

"The default method is to have development write a new command, check it into source code control, have QA run through it, approve it with the release schedule, then release it to manufacturing. Alternately one could just write a quick script and use that instead if the boot loader supported a scripting language, and manufacturing could manage this themselves without begging for R&D resources."

This means that your development/release process is broken with respect to your business needs, not that you need a nifty embedded scripting tool. That whole "ad hoc script added by the end user" is how we wind up with lame websites performing mission critical functions with 1000s of lines of totally unreviewed javascript.

Well, huh... You don't say.

[VortexCortex]

So, if you design your DSL based on the D, it will effect the S of your L?

Who'da thunkit?

Embeddibility?

[Mentally_Overclocked]

After reading the headline, am I the only one that thought embedded in the context of microprocessors instead?

Re:Lua's Intermediate Representation

[Anonymous Coward]

I think the main reason Lua doesn't have a stable IR form is because in practice there simply isn't a need for the kinds of optimizations it would enable. Lua firmly establishes itself as an embedded scripting language, so in nearly all practical cases whatever Lua code is included in your application represents a negligible amount of resources used (or at the very least isn't one of the big bottlenecks). This is on top of the fact that Lua code already is very fast for a scripting language. If Lua performance is really needed, a JIT compiler exists that'd improve performance to the point of making further optimizations more or less irrelevant.

V8 and WebMonkey, conversely, need such optimizations because the JavaScript code becomes the bottleneck in big web applications. In their case, the interpreter needs to be as fast as possible, which makes every optimization useful.

Self-morphing languages

[oldhack]

What's the track record for self-morphing languages like Lua and Forth? I just skimmed a Scala book, and it scares me.

I can see niches like producing DSLs, but smells like bad news for general application/system programming.

Tcl

[jdwoods]

http://sourceforge.net/projects/tcl/ says: "Tool Command Language (Tcl) is an interpreted language and very portable interpreter for that language. Tcl is embeddable and extensible, and has been widely used since its creation in 1988 by John Ousterhout. See http://www.tcl.tk/ for more info." Another good source of information on Tcl is http://wiki.tcl.tk/

Tcl functions well as glue between applications. Some folks know Tcl but call it "Expect" and may not realize Expect is simply Tcl plus an extension. Another extension, Tk, provides GUI features and is so powerful and popular that it's commonly used from other languages. Bindings exist for several other languages, including Ada (called TASH), Perl, Python (called Tkinter), Ruby, and Common Lisp.

Tcl is used by many people and companies (large and small). Cisco network gear uses embedded Tcl for automating tasks. Oracle uses Tcl for automating testing. The Fortune100 company where I work (but I am not a spokesman, so I won't name them) pays me to write and maintain an application written in Tcl to process payments for many thousands of customers totaling millions of dollars every day for payment through banks and the Federal Reserve.

Tcl is FOSS, but a very popular build is ActiveTcl from ActiveState. http://www.activestate.com/activetcl/

Re:tcl

[dublin]

Tcl is indeed an excellent choice for many embedded applications. It's not as trendy as many of the newer languages, but it is an awesome tool for getting a lot done with a very small amount of code (some joke that it stands for "Try Coding Less"), and with very minimal overhead.

Lua's certainly intriguing, and I might look at it for future projects, but the vast majority of what I've had my teams build in the past few years is Tcl (if small size is important), or Python (if it's less important).

Like Python, Tcl approaches executable pseudocode - it's almost always trivial to read a Tcl program and be able to follow it, unlike many other languages. The value of this for things that you (or others) have to maintain and/or extend over time is incalculable...

Tcl's biggest ding right now is that it doesn't have a "popular and modern" snazzy web app platform right now, although AOLserver has great integrated-from-the-ground-up Tcl support, and is definitely one of the apps that shows that Tcl can scale up as well as it scales down....

Re:Apparently Lua needs some press

[lahvak]

It would be helpful if you included some details.

fanwagon

[Aighearach]

There are lots of peeps pushing FFI, it's a bunch of crap IMO. They never seem to have any reason except portability, and they always try to hide or deny that it is slow and not very powerful.

There are already so many C libs, actually doing anything with it requires re-inventing all your wheels, and the wheels to make those wheels.

If C is too hard, you're not going to be doing embedded stuff long enough to get good with FFI anyways.

Re:Apparently Lua needs some press

[LordLimecat]

World of Warcraft uses LUA for mods, and noone seems to be having any problems with it. If anything, theres too many mods.

Really you need to design your application round

[Anonymous Coward]

Really you need to design your application round your embeded scripted language. If added as an afterthought it becomes a "lowest common denominator" interface between your application's type model and the embeded language's.

tcl

[drolli]

i know, that will be hard to swallow, and tcl is not one of the , favourite 5 programming languages of mine, but its stable, easy to embed, has clear interfaces, and is leightweight. What else do you need?

Re:Apparently Lua needs some press

[tibit]

Hilscher uses it in some of their communications gateways. From my personal experience in one particular application, the interpreter was too slow to move all the bits and bytes that had to be moved between two systems speaking different protocols, while keeping the communications (machine) cycle at 1ms. YMMV of course, so this is just a specific case, not a generalization.

I think it'd have been much easier for me as a customer if they simply provided an export library exposing relevant APIs in their firmware, provided remote gdb interface, and provided customers with the build of the open source C compiler that they most likely use themselves (their chips use an ARM core). You'd then upload the linked "executable" into the system, it'd get linked there, and executed just like Lua interpeter is being executed.

I don't see any direct benefits from using scripting languages where it's not hard at all to link your own code into the running system. In many embedded applications, you need realtime and deterministic performance. With any garbage-collected language it means that every machine/process cycle when you need to do garbage collection, to keep the overhead distributed across machine cycles. If you do GC too rarely, it may take too long and won't fit into a machine cycle any longer.

Scripting languages kinda lose their appeal if the API exposed to the scripting language has 1:1 mapping to the underlying C API. High-level languages like Python or Lua are supposed to provide benefits because you program at a higher level of abstraction than bare C. Yet if the interface to the system is very low level, you end up writing C-like code in the scripting language. That's especially true if your functionality is fairly simple -- there's no reason to introduce additional layers of abstraction in your own code, after all. Yet the underlying abstractions in the platform you work on may have nice mappings to abstractions of the scripting language (collections, iterators, closures, etc) -- but they are not exposed as such. I'd much prefer straight C to a bastardized Lua.

Re:Culture

[WNight]

We have a prefix-form for math, observe:

add 3, 5, 10
multiply 2, 2, 2, 2, 2
etc.

Math is really the only thing we use infix for and if you discuss it using prefix operation, as above, nobody has a problem understanding, and therefore getting how lisp works. (add-this-list 1 2 3)

LISP's lispisms really only bug other, often narrowly skilled, programers who can't imagine another way to do anything. To someone learning from scratch it's so simple its oddity is of little importance.