Extending and Embedding Perl

habit forming writes "Enjoy using Perl? Ever marvel at how Perl can "do the right thing" but still be written in C? Extending and Embedding Perl aims to take the black magic out of understanding our favorite language. In fact, the authors flat out admit they think it is unfair that only so few of us get to have one foot in Perl and one in C. Tim Jenness and Simon Cozens attempt to break down that barrier with lots of annotated code examples, direct analogies from the structures in Perl to those in C, a fine-grain look at XS and what it takes to robustly use a Perl interpreter in C." Extending and Embedding Perl author Tim Jenness and Simon Cozens pages 375 publisher Manning (http://www.manning.com/) rating 8.7 out of 10 reviewer habit forming ISBN 1930110820 summary Get in touch with the inner Perl. What's that up your sleeve?

It is my experience that many situations require us to "look under the hood" of (thoroughly examine) a solution to understand how to best use it effectively. Perl is no exception. The ability to bring such a force as Perl to a project at the proper time is a valuable skill to possess. However, wading chest-deep into XS and the Perl internals is not for the faint of heart. Jenness and Cozens ease this process by stepping in lightly at first.

What's in it?

The book begins with simple C examples that are then related back to the readers' knowledge of Perl. Then the text seems to throw us a curve by leaping off into building Perl modules. But there is method to the madness: building Perl modules correctly is inextricably linked to XS. Light introductions to XS are performed and the reader is well on his/her way to building .so extensions to any .pm.

After building a very specific foundation of simple C examples, module building, and some XS, the text returns to C to introduce pointers, arrays, file I/O and memory management. With these new skills, we begin to explore the structure and implementation of Perl variable types. Chapter 4 provides many useful diagrams of how Perl variables "look" and what C structures they translate into.

Still following a logical and constant order, we explore the Perl 5 API, learning how to post and retrieve information to the variable types explored in the previous chapter. As much as it might seem, this is not a rehash of the perlapi doc. It is consistent with the perlapi doc, but Jenness and Cozens provide extensively annotated C code examples.

Casting deeper still, we add the advanced C of pointers, arrays, file I/O and memory management to our knowledge of XS. At this point we have everything we need to effectively extend Perl, but the text continues deeper still by exploring how XSUB interfaces to Perl's internals. It is only the clearly documented, step-by-step explanations of this chapter that make it manageable for an average user like myself. Chapter 7 ends our stint with XS by discussing some alternative XS (or equivalent code) generation suites.

Switching gears entirely, we grab libperl.a and stuff into a C program. Chapter 8 begins the task of embedding Perl into a C program. Jenness and Cozens continue the embedded discussion through a Case Study in Chapter 9 and end with a look through the Perl internals in Chapter 10.

The final chapter (Chapter 11) details some of Perl's history, its development process, how we could become involved and what the future of Perl and Perl 6 may entail.

Final Thought

This book was indispensable in gaining a good foothold on using Perl in, from, and around C. I found it a good reference guide as well as an easy ,linear read. It is not a replacement for the perlguts, perlapi and perlxs documentation, but then again, it doesn't try to be. The annotated code examples with every line explained make following the book with development of your own solution a lot easier than in some other books. However, the in-depth explanations can be a bit frustrating for the impatient.

You can purchase Extending and Embedding Perl from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

Comments

XS Isn't the only way.

[gorilla]

For many applications, you might find it easier to use Inline::C or SWIG. Neither gives you the total power that XS does, but they're much easier to get into.

Combining C and Perl

[NoOneInParticular]

So now we can combine the readability of both C and Perl with the elegant memory protection of C?

interpreter for other applications

[Sludge]

I'm interested in using Perl as an interpreter for other applications. Or, more speficially, game scripting logic. I am confident the time has come for a scripting language to be used for NPC interactions, even in high-performance environments. However, I've never worked with Perl in such a place. I would like to. Has anyone got any performance comments about embedded Perl?

Re:interpreter for other applications

[Avalonia]

We use the PerlStream classes to integrate a Perl interpreter into our C++ applications - it was covered in the January 2002 C/C++ Users Journal - with great success.

Our ASCII file import parsers are written in Perl and the data read into Perl data-structures. The contents of these data structures can then be accessed directly from C++.

The code is on the web (it has some subtle C++ bugs needed fixes using the base-from-member inititialisation idiom) here

XS Mechanics

[swm]

For an overview of XS, see

XS Mechanics

• Introduction
• Architecture
• Tools
• Modules
• Align::NW

Followup book coming soon...

[sammy baby]

Look for a companion volume to be published by Microsoft Press entitled, "Embracing and Extending Perl."

Re:Perl 5 API???

[Reality Master 101]

Isn't Perl 6 coming out soon?

"Soon"? Considering that they haven't even finished deciding the features and changes of Perl 6, I think it's safe to say that a release version is at least a few years off, with 50% adoption being another three years plus after that.

Re:Perl is Perl, C is C

[NerdSlayer]

If you want to complete a project in Perl, then just do it in Perl. If you want to use C for a project, just use C. It's bad enough when people start using buggy bloated libraries, but it's worse when they give their compiler an identity crisis.

I've done this on serveral occasions... did you ever have a huge project to finish in a small amount of time that had some sort of processor-critical portion that's run alot? Well then, I would suggest writing it in Perl and then optimizing that piece in C. Works for me.

Often in the "real world" you can't have your cake and eat it, too. That's why people embed C in Perl.

Re:Perl 5 API???

[hamsterboy]

AFAIK, Perl 6 is a whole other beast. It's a complete rewrite, with changes to the core language.

You'll still be able to run your Perl 5.x scripts under 6, but not vice-versa. Thus, with all the existing Perl 5.x scripts existing in the wild, having a Perl 5 book around may still be handy.

If you like analogies: why would you buy a C book when C++ has been around for years?

-- Hamster

Re:Perl is Perl, C is C

[shoppa]

If you want to complete a project in Perl, then just do it in Perl. If you want to use C for a project, just use C.

But many projects need both low-level access to X/graphics/system libraries and really do benefit from the high-level approach of Perl.

In many cases, there are already libraries available that link the two together (e.g. Perl/Tk if you're writing in Perl, or associative array libraries for use from C) but it's never been a secret that you can call C code from Perl. You seem to advocate keeping it a secret.

Re:Perl: Fitting into the Big Picture

[sohp]

Java does everything that C++ does

Uh, no. Thanks for playing. There are things that C++ does that Java does not -- some of which I'm thankful do not exist in Java (preprocessor) and some of which I miss (generics). But despite its C-like syntax and superficial resemblances (finalizers seem like destructors but aren't) Java is more like Smalltalk than C++.

Take a quick gander the section For C, C++ Fans in Peter van der Linden's Java Programmers FAQ

But then, why am I arguing over the relative merits of Perl, Java, C++, and C# with a user having the handle "Microsoft Research" who posts pure FUD?

Interfacing Perl with C...

[Snake]

Interfacing Perl with C has its uses, depending on what your current project is.

In my case, I'm part of a large scale C++ project. I have the ownership of a module with clearly defined interfaces with the other modules written in this project.

Since my module relies heavily on XML and strings, I have always wanted to pair it with the power of Perl for testing purpose.

Among various possibles solutions (XS, SWIG, etc.), I settled on SWIG because it could handle 'shallow' classes. (allowing to expose my module as a perl object)

This has been the best decision I have made over the last year: when I get a bug case, I simply write a perl script to try to reproduce the problem, add some loops to get some combinatory, then check the result. This drastically cuts down on the time spent on debugging my module (or the modules used by it, for that matter :)

Pros:

• SWIG: Relatively easy generation of stub code (by using interface files)
• SWIG: It is possible to use the same interface files to generate stub code for Java, Python (though, I didn't test this feature)
• SWIG: Excellent doc.
• Perl: you can leverage the CPAN/PPM modules to do some truly magical hackings

Cons:

• While SWIG does a good job of hiding the gory parts of Perl Internals, you still need to brush up on it to better understand how it works, if only to avoid memory leaks
• Perl: I haven't been to handle properly the passing of strings (I managed to do it by using a horrible hack that seems to work). I'm probably not smart enough to understand the documentation or the samples.
• SWIG: the stub code is dependant on the version of perl used. It is therefore difficult to release it. This is mostly a dev tool.

Summary: If you are a C/C++ developper and your code can use XML/text files/strings, consider using SWIG or XS for testing purpose.

PS: if you want to Quantify/Purify your module/Perl script, using ActiveState Perl, you need to recompile Perl with the -DPURIFY option toggled on.

Re:I think Perl5/XS will be with us for long time.

[Elian]

You missed the mark in just a few places here.

1. no calling conventions yet for subroutines. Alas not true. They're in place and have been for quite some time.
2. no conversion opcodes for various builtin types Also incorrect--we've got them and have since the very beginning. What we don't have is low-level support for specific size integer and floats, since our target languages (perl, python, ruby) don't have them. Adding them adds no overhead, though, so they're going in since it'll make base .NET and JVM compatibility simpler.
3. non-perl languages expected to provide additional support in the form of C code libraries for their opcodes. Once again, incorrect. No external libraries are or will be required. We're turing complete and generally have a richer set of base semantics than .NET or the JVM does. This doesn't mean that someone can't choose to require an alternate set of opcodes if they want, but the engine doesn't require it.
4. no way to call out to C code This one's actually true. We've not gotten to that part yet, though it's on the list.
5. no equivalent of Java's jar file or CLR's assemblies for parrot library distribution Incorrect, bytecode files work just fine for this. That part of the design is somewhat incomplete, though.
6. way too many registers: their register based VM ... requires a sophisticated compiler to do proper register allocation and needlessly complicates their VM Wrong yet again, sorry. Doesn't requore a sophisticated compiler at all. At best it requires some sophisticated register allocation algorithms. Luckily for us, those algorithms are old, known territory, which is why we've got them implemented already. So what if it makes the VM slightly more complex? (And it is only slightly more complex because of it) We're not writing something for CS101 here.
7. no consideration of threads in their design. Once again, incorrect, if you'd bothered to read any of the design documents. Threading isn't, at the moment, implemented because other things have been more important, but it has been thought about in the design.

1.5 for 7. Not too good there. Oh, and this:

seems that Parrot is too hung up on making the VM efficient

What drugs are you on? If we piss away efficiency in the design, no amount of clever coding will ever get it back. Maybe you're willing to sacrifice a factor of two in speed for "clarity" but so are nine of your friends. (To misquote the author of make) I, on the other hand, am not.

Finally

Or perhaps they should simply concentrate on getting Perl6 to work first. They need more focus. The project tries to be all things to all people, but ends up satisfiying no one.

If you'll look, you'll notice that perl 6 isn't fully designed yet, but the bits that are have been implemented.

Just because you can't (or won't, or don't want to) see the focus doesn't mean it's not there. It is, thanks very much, and we're well on track to do what we need and do it well. The design's flexible enough to pick up things like JVM or .NET compatibility without a loss of focus or efficiency, so there's no reason not to.

Re:I think Perl5/XS will be with us for long time.

[Elian]

Nope, but being right makes it true.

1. The Parrot calling convention POD was pulled a while back. No, it wasn't. PDD 3 is in, and has been for a long time. There have been minor revisions, but that's it--minor revisions.
2. Where's Parrot's 64 bit integer type? Inside a PMC, at the recommendation of the dotGNU folks. We'll stick the 128 bit, 256 bit, and 512 bit integers in one too, if we need them.
3. You *do* need some form of distributable bytecode library format No, we don't, not for perl, python, and ruby support. (Who was it whining about lack of focus earlier?) Bytecode files do work just fine for it anyway, however. Nobody said they needed to be structureless.
4. Threading issues *must* be considered from the very start Yes, I know. If you'd have bothered to do any research, you'd have seen the direction we're going, and you'd have realized that what we've done so far has no ramifications for our chosen threading model.

The only part of history repeating itself is the naysaying from people who've not done their research. If you're going to criticize, at least get it right.

Re:Save some money

[marklark]

also available from the publisher as an e-book for $22.47 (half SRP)

http://www.manning.com/jenness/index.html

Re:I think Perl5/XS will be with us for long time.

[Elian]

You seem to be missing the point with some regularity. PMCs don't have to be written in C--they will be doable in parrot code if need be. And just because 64 bit integers will be done with PMCs doesn't mean that they won't be part of the core distribution, or a recommended library.

There's nothing particularly wrong with saying "You must have the X library/module/kit to do Y". Requiring the install of the .NET library to get full .NET functionality will undoubtedly be needed. (We're certainly not going to ship the full .NET core library with Parrot any more than we're going to ship the full Java core library) If we don't ship 64 bit ints as part of the core, they'll either be in the .NET library, or in an extended data type library.

What's next, will you start complaining next that we're going to require installing Postgres to access Postgres databases? (Or will the next complaint be about the bloated size of the distribution to provide the features that match your expectations?)