Slashdot Mirror

← Back to Stories (view on slashdot.org)

Designing a Programming Language For Embeddability

Posted by timothy on from the get-in-there-you dept.

CowboyRobot writes "The creators of the Lua language describe the process of designing a new language and the constraints that certain parameters, specifically embeddability, place on the process. 'Many languages (not necessarily scripting languages) support extending through an FFI (foreign function interface). An FFI is not enough to allow a function in the system language to do all that a function in the script can do. Nevertheless, in practice FFI covers most common needs for extending, such as access to external libraries and system calls. Embedding, on the other hand, is harder to support, because it usually demands closer integration between the host program and the script, and an FFI alone does not suffice.'"

52 comments

  1. <body><head uuid="0"></body> by Anonymous Coward · · Score: 0

    <head uuid="0"/>
    <body head="0">
    'nuff said
    </body>

  2. Oh good by MightyMartian · · Score: 0

    Oh good, another language.

    --
    The world's burning. Moped Jesus spotted on I50. Details at 11.
    1. Re:Oh good by pem · · Score: 4, Informative

      It's 18 years old. If you don't care to know anything about existing languages, why would having newer ones bother you?

    2. Re:Oh good by Anonymous Coward · · Score: 0

      Your ignorance is showing. Here, I'll give you a geek card to cover it up.
      http://en.wikipedia.org/wiki/Lua_%28programming_language%29#Applications

    3. Re:Oh good by rrohbeck · · Score: 2

      Bah. Newfangled crap. I'll stick to COBOL, thank you very much.

      --
      thegodmovie.com - watch it
  3. If they are going to, might as well do it right. by pavon · · Score: 3, Interesting

    Since it's a given that people are going to be creating new languages, I certainly hope they listen to what Roberto has to say. Lua is a joy to embed into applications compared to python and others.

  4. Don't overlook the latest in maths discoveries. by OpenQL · · Score: 0

    That reminds me i really should start the OpenQL programming language, that exists not for reasons of grammar but the perfection of maths our true evolutionary route. Anyone interested in assisting the project mail me openql ..a..t gmail. Peace.

    1. Re:Don't overlook the latest in maths discoveries. by Anonymous Coward · · Score: 0

      It already exists. And it's called Haskell. ;)

  5. Lua's Intermediate Representation by Anonymous Coward · · Score: 0

    Interestingly, Lua does not offer a stable/standardized IR form (i.e. they offer no guarantees that it'll not change from version to version). Expressing a program in IR form is valuable in that you can apply a lot of existing IR optimizations to it to gain performance (IR is a representation somewhere in between abstract syntax trees and low level machine code symbolic representation.), e.g. dead code elimination, constant propagation, copy propagation, strength reduction, common subexpression elimination, loop unrolling, loop fission/fusion, memory layout optimization, structure splitting, array reshaping, etc.

    Some of the optimizations I mentioned are key in V8 and WebMonkey's performance gains...however you do get the usual caveat. Such analysis takes time and if you're running a run-once and throw-away code snippet, you have to think if it's worth the trouble.

    I think Lua can afford to do this, since most hosts that use Lua runs Lua scripts for a long period of time (e.g. a Game that runs a Lua script for controlling an AI). Applying some of the optimizations mentioned earlier should boost some of these codes' performance by a lot.

    1. Re:Lua's Intermediate Representation by Anonymous Coward · · Score: 1

      I think the main reason Lua doesn't have a stable IR form is because in practice there simply isn't a need for the kinds of optimizations it would enable. Lua firmly establishes itself as an embedded scripting language, so in nearly all practical cases whatever Lua code is included in your application represents a negligible amount of resources used (or at the very least isn't one of the big bottlenecks). This is on top of the fact that Lua code already is very fast for a scripting language. If Lua performance is really needed, a JIT compiler exists that'd improve performance to the point of making further optimizations more or less irrelevant.

      V8 and WebMonkey, conversely, need such optimizations because the JavaScript code becomes the bottleneck in big web applications. In their case, the interpreter needs to be as fast as possible, which makes every optimization useful.

  6. Forth by smoothnorman · · Score: 3, Informative

    Forth http://en.wikipedia.org/wiki/Forth_(programming_language)/ was designed to run on (albeit early) embedded platforms. Extensible, easy to learn, easy to implement. I guess it's just the RPN that scares folks off? ("no you old fool! modern processors are optimized away from stack intensive use!")

    1. Re:Forth by nielsm · · Score: 1

      Except this is not about languages running on embedded platforms, this is about hosting a language inside an application, for providing e.g. scriptability of a GUI application.

    2. Re:Forth by VortexCortex · · Score: 2

      Wrong kind of "embedded" -- The Lua devs meant "easily accessed from within the host language ( C / C++ )".

    3. Re:Forth by hoggoth · · Score: 1

      Loeliger's book 'Threaded Interpreted Languages' was one of my favorites non-fiction books ever. (Yes, I am that geeky).
      Reading how he started with just a few tens of opcodes and chapter by chapter boot-strapped an entire operating environment was like reading Genesis.

      --
      - For the complete works of Shakespeare: cat /dev/random (may take some time)
    4. Re:Forth by smoothnorman · · Score: 1

      yeah! it along with the later one by Brodie always comes to mind when i'm writing embedded applications (in either of the disparate senses of "embedded". should be a clear way to specify them .. machine-embedded or Gui-bedded ?) http://www.amazon.com/Threaded-Interpretive-Languages-Design-Implementation/dp/007038360X/ref=sr_1_1?ie=UTF8&qid=1305766293&sr=8-1/

    5. Re:Forth by Darinbob · · Score: 1

      I always liked REXX myself, and AREXX on Amiga. Simple, easy to embed, the host program is in control as opposed to the embedded language's loop, integrates with other programs using the same embedded language, etc.

    6. Re:Forth by Anonymous Coward · · Score: 0

      "Atlast" is an example of a forth that is intended for that kind of embedding. It's kind of showing its age, but it seems like it would be practical.

      It's over at fourmilab, and is public domain.

    7. Re:Forth by sgt+scrub · · Score: 1

      if your going to write small blazing fast code you need to go old school. http://www.amigacoding.com/index.php/680x0:AsmOne it comes with its own IDE!

      --
      Having to work for a living is the root of all evil.
  7. Apparently Lua needs some press by Anonymous Coward · · Score: 0

    All of the people I know who have experienced Lua (5 or so) think it's inferior. One even works at a company that embedded Lua into their product as a scripting language only later to realise that was big mistake.

    1. Re:Apparently Lua needs some press by Musc · · Score: 1

      Inferior to what?

      --
      Hamsters are at least as feathery as penguins. HamLix
    2. Re:Apparently Lua needs some press by lahvak · · Score: 1

      It would be helpful if you included some details.

      --
      AccountKiller
    3. Re:Apparently Lua needs some press by LordLimecat · · Score: 1

      World of Warcraft uses LUA for mods, and noone seems to be having any problems with it. If anything, theres too many mods.

    4. Re:Apparently Lua needs some press by tibit · · Score: 1

      Hilscher uses it in some of their communications gateways. From my personal experience in one particular application, the interpreter was too slow to move all the bits and bytes that had to be moved between two systems speaking different protocols, while keeping the communications (machine) cycle at 1ms. YMMV of course, so this is just a specific case, not a generalization.

      I think it'd have been much easier for me as a customer if they simply provided an export library exposing relevant APIs in their firmware, provided remote gdb interface, and provided customers with the build of the open source C compiler that they most likely use themselves (their chips use an ARM core). You'd then upload the linked "executable" into the system, it'd get linked there, and executed just like Lua interpeter is being executed.

      I don't see any direct benefits from using scripting languages where it's not hard at all to link your own code into the running system. In many embedded applications, you need realtime and deterministic performance. With any garbage-collected language it means that every machine/process cycle when you need to do garbage collection, to keep the overhead distributed across machine cycles. If you do GC too rarely, it may take too long and won't fit into a machine cycle any longer.

      Scripting languages kinda lose their appeal if the API exposed to the scripting language has 1:1 mapping to the underlying C API. High-level languages like Python or Lua are supposed to provide benefits because you program at a higher level of abstraction than bare C. Yet if the interface to the system is very low level, you end up writing C-like code in the scripting language. That's especially true if your functionality is fairly simple -- there's no reason to introduce additional layers of abstraction in your own code, after all. Yet the underlying abstractions in the platform you work on may have nice mappings to abstractions of the scripting language (collections, iterators, closures, etc) -- but they are not exposed as such. I'd much prefer straight C to a bastardized Lua.

      --
      A successful API design takes a mixture of software design and pedagogy.
  8. GRUB? by rrohbeck · · Score: 1

    So what happened to Lua support in GRUB? I saw some pretty nifty things done with that.

    --
    thegodmovie.com - watch it
    1. Re:GRUB? by Anonymous Coward · · Score: 0

      What the fuck? Seriously?
      Scripting language in a bootloader?
      What next, a rewrite in VB6 or in VB.NET?

  9. Domain-specific Languages by Anonymous Coward · · Score: 0

    Nowadays you would design special languages for domains using languages to help you built such language like Xtext, Xpand/Xtend. You use templates to generate Java, C or assembler code. This is more a stacking approach than an embedding approach. If I am not mistaken C# has some embedding features. Also tools like MPS support that direction. Even though the embedding method results in less developed DSLs than the top-down approach.

  10. Yea its called.. by Anonymous Coward · · Score: 0

    C

  11. Lisp by betterunixthanunix · · Score: 2

    Why jump through so many hoops to get an embeddable language, when you could use the same language to write the "host program" as you use to write scripts for it? Lisp is an example of such a concept: compiled Lisp programs still have the ability to interpret Lisp code, and hence the most sensible scripting language for a programming written in Lisp is Lisp itself. Lisp is not the only language with this property, but it is one of the most prominent examples of such a language.

    --
    Palm trees and 8
    1. Re:Lisp by Anonymous Coward · · Score: 0

      In less than #!140 http://twitter.com/Dzonatas_Sol/status/71018131946082304

    2. Re:Lisp by Anonymous Coward · · Score: 0

      The purpose of Lua is to provide a lightweight scripting environment for applications written in C/C++. In that case your suggestion would be to embed C code in a C program, which isn't very useful as it wouldn't have any of the advantages of embedding with Lua such as faster compilations and not being C.

      Oh, and do you want to know of another language that can have programs that interpret code written in the same language? Lua. It's even mentioned in TFA.

    3. Re:Lisp by Darinbob · · Score: 4, Informative

      Because the host language may be a conventional language, designed to compile efficiently, run in small amounts of memory, has speed critical sections, etc. The embedded language on the other hand is used for doing stuff that wasn't thought of in the design phase and where it's simpler to add a small script than to add a new feature to the program. Generally what happens is that you're tying together a high level language with a low level language.

      For instance, take a boot loader. It is written to fit into a small area of flash or just a few blocks of mass storage. So it needs to be small and tight. In manufacturing you'd like to run a set of tests and the boot loader provides some simple commands to call that test various features. But maybe you want a particular test that the built in commands won't do, for example a stress test of memory. The default method is to have development write a new command, check it into source code control, have QA run through it, approve it with the release schedule, then release it to manufacturing. Alternately one could just write a quick script and use that instead if the boot loader supported a scripting language, and manufacturing could manage this themselves without begging for R&D resources. Every time a bug occurs in the process someone could write a new regression test using the scripting language to check for the bug and add the script to a growing collection.

      Now if we could somehow get fast/efficient/tiny host programs written in Lisp (which is indeed possible) and we manage to find a collection of developers who understand it (which is harder), then it makes sense to have the scripting language the same as the host language. But in the mean time most host programs are written in things like C or C++. Even if you have a high level interpreted language like Java that you wrote your host program in, it is very difficult to parse/interpret textual Java programs.

      Even if your host program is in Lisp it will be difficult to isolate any textual Lisp scripts from the host program itself; you don't want a buggy script to cause havoc with the host program. Most Lisp systems I've seen that allow extra scripting basically use "eval" which is not very safe. So you still need to put up a solid wall with a well defined set of methods to allow a script to run in its own sandbox while interacting occasionally with the host program.

    4. Re:Lisp by Anonymous Coward · · Score: 0

      like a metacircular evaluator?

    5. Re:Lisp by Anonymous Coward · · Score: 1

      "The default method is to have development write a new command, check it into source code control, have QA run through it, approve it with the release schedule, then release it to manufacturing. Alternately one could just write a quick script and use that instead if the boot loader supported a scripting language, and manufacturing could manage this themselves without begging for R&D resources."

      This means that your development/release process is broken with respect to your business needs, not that you need a nifty embedded scripting tool. That whole "ad hoc script added by the end user" is how we wind up with lame websites performing mission critical functions with 1000s of lines of totally unreviewed javascript.

  12. Well, huh... You don't say. by VortexCortex · · Score: 1

    So, if you design your DSL based on the D, it will effect the S of your L?

    Who'da thunkit?

  13. Embeddibility? by Mentally_Overclocked · · Score: 1

    After reading the headline, am I the only one that thought embedded in the context of microprocessors instead?

    --

    Mathematician, n.:
    Someone who believes imaginary things appear right before your i's.
    1. Re:Embeddibility? by Anonymous Coward · · Score: 0

      Yeah, I thought about natural languages and processors.

      Like... personally I thought that German was designed with embedded systems in mind.

    2. Re:Embeddibility? by Anonymous Coward · · Score: 0

      I honestly wasn't aware there were other uses of the term in the context of computing.

  14. Self-morphing languages by oldhack · · Score: 2

    What's the track record for self-morphing languages like Lua and Forth? I just skimmed a Scala book, and it scares me.

    I can see niches like producing DSLs, but smells like bad news for general application/system programming.

    --
    Fuck systemd. Fuck Redhat. Fuck Soylent, too. Wait, scratch the last one.
  15. Tcl by jdwoods · · Score: 2

    http://sourceforge.net/projects/tcl/ says: "Tool Command Language (Tcl) is an interpreted language and very portable interpreter for that language. Tcl is embeddable and extensible, and has been widely used since its creation in 1988 by John Ousterhout. See http://www.tcl.tk/ for more info." Another good source of information on Tcl is http://wiki.tcl.tk/

    Tcl functions well as glue between applications. Some folks know Tcl but call it "Expect" and may not realize Expect is simply Tcl plus an extension. Another extension, Tk, provides GUI features and is so powerful and popular that it's commonly used from other languages. Bindings exist for several other languages, including Ada (called TASH), Perl, Python (called Tkinter), Ruby, and Common Lisp.

    Tcl is used by many people and companies (large and small). Cisco network gear uses embedded Tcl for automating tasks. Oracle uses Tcl for automating testing. The Fortune100 company where I work (but I am not a spokesman, so I won't name them) pays me to write and maintain an application written in Tcl to process payments for many thousands of customers totaling millions of dollars every day for payment through banks and the Federal Reserve.

    Tcl is FOSS, but a very popular build is ActiveTcl from ActiveState. http://www.activestate.com/activetcl/

    --
    -- Jeff Woods
    1. Re:tcl by dublin · · Score: 1

      Tcl is indeed an excellent choice for many embedded applications. It's not as trendy as many of the newer languages, but it is an awesome tool for getting a lot done with a very small amount of code (some joke that it stands for "Try Coding Less"), and with very minimal overhead.

      Lua's certainly intriguing, and I might look at it for future projects, but the vast majority of what I've had my teams build in the past few years is Tcl (if small size is important), or Python (if it's less important).

      Like Python, Tcl approaches executable pseudocode - it's almost always trivial to read a Tcl program and be able to follow it, unlike many other languages. The value of this for things that you (or others) have to maintain and/or extend over time is incalculable...

      Tcl's biggest ding right now is that it doesn't have a "popular and modern" snazzy web app platform right now, although AOLserver has great integrated-from-the-ground-up Tcl support, and is definitely one of the apps that shows that Tcl can scale up as well as it scales down....

      --
      "The future's good and the present is nothing to sneeze at." - Roblimo's last ./ post
  16. fanwagon by Aighearach · · Score: 1

    There are lots of peeps pushing FFI, it's a bunch of crap IMO. They never seem to have any reason except portability, and they always try to hide or deny that it is slow and not very powerful.

    There are already so many C libs, actually doing anything with it requires re-inventing all your wheels, and the wheels to make those wheels.

    If C is too hard, you're not going to be doing embedded stuff long enough to get good with FFI anyways.

  17. Entire article in three sentences by Anonymous Coward · · Score: 0

    1) It must have an eval function or something like that to turn source into executable code.
    2) It must be a library, so the host can load it.
    3) Global variables must be accessible, but local variables must not be.
    The article gives no argumentation for these.
    Without going so far as to judge the claims, I think I can safely say that reading this comment instead of the article will save time.

  18. Really you need to design your application round by Anonymous Coward · · Score: 1

    Really you need to design your application round your embeded scripted language. If added as an afterthought it becomes a "lowest common denominator" interface between your application's type model and the embeded language's.

  19. tcl by drolli · · Score: 1

    i know, that will be hard to swallow, and tcl is not one of the , favourite 5 programming languages of mine, but its stable, easy to embed, has clear interfaces, and is leightweight. What else do you need?

  20. Worst by TheDarkMaster · · Score: 0

    Another interpreted, garbage-collected, slow as hell but easy to use than evil C language... When this madness will end?

    --
    Religion: The greatest weapon of mass destruction of all time
  21. Nu has it all by Anonymous Coward · · Score: 0

    The should check out Nu: http://programming.nu/index
    Full interoperable with Objective-C and thus C/C++.

  22. Culture by Anonymous Coward · · Score: 0

    One word: Culture

    To flesh that out. Most cultures prefer infix notation. If you're targeting highly technical people as in Autocad, then embedding Lisp might work. Might. Even if the customers are as technical as Autocad users, they might still resist.

    1. Re:Culture by WNight · · Score: 1

      We have a prefix-form for math, observe:

      add 3, 5, 10
      multiply 2, 2, 2, 2, 2
      etc.

      Math is really the only thing we use infix for and if you discuss it using prefix operation, as above, nobody has a problem understanding, and therefore getting how lisp works. (add-this-list 1 2 3)

      LISP's lispisms really only bug other, often narrowly skilled, programers who can't imagine another way to do anything. To someone learning from scratch it's so simple its oddity is of little importance.

  23. Re:If they are going to, might as well do it right by Anonymous Coward · · Score: 0

    Lua is a good embeddabilibuddy.