Super-Fast Python Implementation for .NET and Mono

Lansdowne writes "Jim Hugunin, the creator of Jython, has released an incredibly fast implementation of Python for Microsoft .NET and Mono called IronPython. Here's his PyCON 2004 presentation, including some benchmarks. He concludes: 'Python is an extremely dynamic language and this offers compelling evidence that other dynamic languages should be able to run well on this platform.'"

Comments

cool stuff

[hattmoward]

I'm glad to see more alternative languages for the .NET/Mono platform... If we're gonna get stuck with it, we may as well make the best of it! :) Seeing Python run nice and fast, being a dynamic language and on a VM, is great stuff too.

Re:cool stuff

[AJWM]

Cool indeed. But is there a C# compiler targetting Parrot yet? ;-) They look like they're almost both pretty groovy.

(There's a language I'd like to see on more platforms.)

Re:cool stuff

[Brandybuck]

I'm still keeping my fingers crossed hoping we don't get stuck with it. It will be a sad era when Microsoft dictates specifications to the Open Source and Free Software communities...

Re:cool stuff

[hattmoward]

hehe
http://www.samba.org 'nuff said.

Re:cool stuff

[AJWM]

Troll?

This is obviously some new meaning of the word with which I was previously unfamiliar.

Perhaps he should have waited...

[clintp]

Maybe he could have gotten in on the Great Parrot/Python Pie-a-Thon. Perhaps an opportunity to plaster both Guido and Dan with a cream pie...

Well, that's great and all

[Anonymous Coward]

But Python really hold a candle to PHP or Perl. A language without dollar signs is a day without sunshine.

Sorry to break it to you.

Re:Well, that's great and all

[The Fink]

Using that as a benchmark, then some dialects of BASIC suit you too then, eh? :-)

Re:Well, that's great and all

[theantix]

eh? WTF, you Canadian or something?

Next Question

[complete loony]

Is there anything the MONO team can do to improve support for dynamic languages?
Though this would probably break their binary compatibility with MS's implementation.

Re:Next Question

[Chester K]

Is there anything the MONO team can do to improve support for dynamic languages?

Mono, today, actually would support dynamic languages better than Microsoft's framework, since they implement the DynamicMethod class, which Microsoft fully documents in their Longhorn SDK documentation, but won't actually be released until .NET 2.0 comes out.

The DynamicMethod class allows you to load a method into memory for JITting and execution, and preserves the ability for you to unload that code from memory, which you can't do with full-fledged Assemblies, even those generated via Reflection.Emit -- in .NET, once an Assembly is loaded into memory, it stays there until the AppDomain quits.

Re:Next Question

[bearclaw]

You can, however, create new AppDomains. Then destroy them.

Re:Next Question

[TummyX]

Yeah, and method calls into an object in another AppDomain is a thousand times slower than a locla method call isn't it?

Re:Next Question

[Chester K]

You can, however, create new AppDomains. Then destroy them.

Which is incredibly slow and limited, since you can only pass data between AppDomains if the data is serializable, which means no passing filehandles, sockets, API handles, or the like -- and then you encounter the overhead of having to serialize/deserialize it, which is considerable.

not released

[the quick brown fox]

I usually take "released" to mean there is an implementation that is publicly available. Unless I am somehow just missing it, it doesn't seem to have been released yet...

Re:not released

[Phronesis]

Indeed, anyone reading the links to IronPython would find that "IronPython is currently an unreleased research prototype."

Compiled Python

[Radical Rad]

I haven't worked with Python yet though I plan to take a look at it soon. I know it is a scripting language but is there a way to compile it into native executable form? If not, what is needed to make a Python app run on a system without the interpreter installed or with an older or newer version? Would I need to ship something JRE-ish that would that make a HelloWorld package megabytes in size?

Re:Compiled Python

[FrenZon]

I haven't worked with Python yet though I plan to take a look at it soon. I know it is a scripting language but is there a way to compile it into native executable form?

"You don't need the ability to compile Python to C code if all you want is a stand-alone program that users can download and run without having to install the Python distribution first. There are a number of tools that determine the set of modules required by a program and bind these modules together with a Python binary to produce a single executable..." Read More

Re:Compiled Python

[arkanes]

Theres a couple tools for packing Python into a single binary. The Python runtime itself is fairly small, but most apps will need a signifigant supset of the standard libraries as well, so you'll be shipping some pretty large files. When using wxPython it walks all over Java, though.

Re:stuck?

[Markus Registrada]

Who says we're stuck with dotnet? Most Free Software machines have no JVM installed, and most will have no more reason to have a CLI either. Just about anything you can do with either (i.e. anything except talk to other, proprietary, components), you can do better natively, in a language not specifically crippled to advance crabbed corporate interests.

The real lesson of this "superfast Python" is not that dotnet is a fast platform; it's the more prosaic observation that regular Python still has a great deal of room for improvement. Those Python weenies ought to get on it, and save adding LISPy new features for later.

A Python front end for Gcc could be fun.

What I want to know is...

[WTFmonkey]

Is is "IronPython," the way we'd usually pronounce it, or do we put a little Rocky Balboa in it and call it the "I- ron Py- thon ?"

Lisp (was Re:stuck?)

[Tool Man]

Those Lispy features are a powerful addition, which make it easier for a programmer to get their job done. If you would rather trade those features for performance, consider Fortran instead. It would be more useful if, rather than focus on the leverage of a single-platform solution, the implementors can approach the performance of various Common Lisp implementations. It is true that dynamic languages don't have to be slow; neither do they have to rely on Microsoft's runtime.

Re:Lisp (was Re:stuck?)

[Too Much Noise]

Fortran??? Surely you must be joking. The typical mantra about Fortran being fast is such a broad statement that it's most of the times wrong. Sure, lapack/blas are written in Fortran - but that's hardly a statement of Fortran's speed superiority (optimized binary code couldn't care less what language it was compiled from - and the platform-dependent blas optimizations use asm anyway).

The problem is rather that C (the natural contender) allows a non-experienced user to screw up really badly. Anyway, in one typical Fortran use, scientific computing, poorly written code more often than not nullifies any benefits of a smart compiler/optimized library - and that's probably a truer reason for these people to stick with Fortran, as writing efficient C code has a steeper learning curve.

Re:Lisp (was Re:stuck?)

[Tool Man]

Heh. My point was not really that Fortran is the best choice for most people, but that slagging Lisp features in Python makes no sense. Other languages are working hard at adding those features... usually poorly. Python gets closer than most.

Re:stuck?

[jsac]

A Python front end for Gcc could be fun. And nearly impossible. I suspect that Python's way too dynamic to be handled by anything but a run-time interpreter.

Great News

[ReyTFox]

I like Python very much. I may not like .Net, but I'll take my favored languages where I can find 'em :)

I'd suspect, though, that a Parrot implementation, as Parrot continues to develop, will prove to be faster, since it was designed with more dynamic languages in mind.

Psyco

[fredrikj]

* Fast - IronPython-0.2 is 1.4x faster than Python-2.3 on the standard pystone benchmark.

I don't know about pystone in particular, but Psyco (a dynamic compiler module that essentially replaces the Python interpreter's inner loop at runtime) tends to make code run much faster than that and can speed up algorithmic code tenfold or more.

When running with Psyco, quicksort written in Python is actually faster than Python's built-in C mergesort!

Re:Psyco

[Anonymous Coward]

Python's quicksort has also been shown to be faster than C's bubblesort, Python's FFT has been shown to be faster than any C DFT for arrays over a length of 1024, and Python's name has six times the number of letters that C's does!

Re:Psyco

[fredrikj]

Python's quicksort has also been shown to be faster than C's bubblesort

You got that comparison wrong; mergesort is algorithmically faster than quicksort. Mergesort operates in O(n log(n)) time for any input whereas quicksort generally operates in O(n log(n)) but jumps to O(n^2) in the worst case. Quicksort is the most widely used sort algorithm because it uses less overhead which makes it faster in practice.

Re:Psyco

[Anonymous Coward]

You got that comparison wrong; mergesort is algorithmically faster than quicksort.

"Faster" is the wrong word. Mergesort has a lower order of complexity, which means it scales better in suboptimal situations.

Remember, O(n^2) will always be "faster" than O(m) where n sqrt(m); the complexity of an algorithm is about scalability, not speed.

Re:Psyco

[fredrikj]

Well, I've encountered the term "algorithmic speed" myself as referring to complexity, but I guess it might be inaccurate.

Re:Psyco

[saurik]

It's also insanely more cache coherent in the way it processes it's input. Merge sort tends to randomly scatter accesses around the data. On common computer hardware, QuickSort is pretty much guaranteed to be faster. We actually went over this very point in a class I took on Computer Architecture a few years back.

Re:Psyco

[Anonymous Coward]

Insanely? Are you talking about in-place mergesort? The inner loop of an ordinary out-of-place mergesort is just three linear streams where quicksort would have two linear streams. There's even a neat reversal trick to make the mergesort stream accesses linear over boundaries of the recursive function calls in the algorithm...

Re:stuck?

[DrSkwid]

good luck getting eval to work in gcc

Dynamic?

[Anonymous Coward]

Does this word really mean anything? I keep hearing people talk obout "dynamic languages" and then when pressed they seem to provide rather contradictory definitions of what it actually means. I am convinced that it is merely slang for hip, cool, happening, righteous, etc. The interesting thing is that most of the people who are using it to describe programming languages seem to think that it has some specific technical meaning.

Re:Dynamic?

[vrt3]

I can't speak for the term in general, but in Python it means variables can have different types on different times, while still being strongly typed.

a = 1
Now a is an integer (actually what happens is this: an integer object is created, with the value 1, and the name a is bound to it)

a == '1'
Returns False: objects with different types can not be equal.

a = '1'
A new object is created, containing the string '1'. a now binds to that new object.

a == '1'
True.

Somewhat more useful: file objects implement a number of methods. If you write a class that implements those methods (often a subset is enough), you can use it any place where you can use a file object. No need to derive from file class. Of course this doesn't work only for files, it can be used for anything.

Re:Dynamic?

[Meijer]

Does this word really mean anything?

Besides dynamic typing which has already been discussed above, "dynamic" languages often offer additional features that qualify them as dynamic, such as creating classes at runtime (e.g. prototype-based, as in javascript).

Still, you are right. The term is used in a nebulous manner. A possible "definition" is:

A dynamic language is a language in which most decisions are deferred until run time. A static language is a language in which most decisions are made at compile time.

(don't know who said that)

An interesting discussion: Lambda the Ultimate (google cache, site is currently down)

Re:Dynamic?

[Jerf]

That's not the best way of describing how it works.

In C, a "variable" is a box that contains things. The box is designed to only hold one kind of thing, so an "int" box can't hold a "char *".

In Python, a "variable" is just a "post-it note" that can be stuck onto a value. The post-it note "a" can be stuck on to anything:

a = 1
a = "One"
a = MyCustomObject()

Nevertheless, Python is a strongly-typed language; this will raise an error:

a = 1 + " some"

(If you're on a Unix system, you most likely have Python installed. Type "python" on the command line and try typing that in.)

Objects with different types are allowed to be equal, though there is some obvious danger with that. Here's a pathological case:

class AlwaysEqual:
def __eq__(self, other):
return 1

a = AlwaysEqual()
a == 1
a == '1'
a == None

This is bad, bad code in real life, but it proves the point.

Re:Dynamic?

[elflord]

Does this word really mean anything?

Yes. Dynamic means that a lot of things are deferred to runtime as opposed to compile-time (static).

For example, some languages like python use dynamic typing :

a = 'hello'
a = 1
def foo(a): # we don't know what type a is -- deferred till runtime
print a

Some languages like C++ have dynamic binding (e.g. virtual functions).

Some languages allow dynamic loading. That is, you can load libraries at runtime and query those libraries. While nearly every language has some support (built in or library support like libdl) for dynamic loading, the functionality available in the dynamic loading varies.

Some languages allow dynamic execution. For example, eval. In most interpreted languages, one can write a program that reads code from the user and executes that code. One can't do that in most compiled languages.

The term "dynamic language" is used to refer to a language that supports many of the above. Typically, interpreted languages are very dynamic. VM languages are somewhat dynamic. Pure compiled languages are more static.

What changed?

[ameoba]

I remember some previous attempt to do Python under .NET that was painfully slow & eventually written off as a failure and blamed on .NET's inability to handle the level of dynamism required to implement Python. What's changed since then?

Are we looking at some sort of fundamental breakthrough in working with the CLR here or was the problem simply tackled by a more experienced/insightful developer?

Re:What changed?

[Anonymous Coward]

RTFA.

Why is IronPython Fast?

High system performance is the end result of hundreds of small decisions rather than a single large one. Much of IronPython's performance comes from careful consideration of performance in each design choice. I've found that profilers are generally useless for improving the performance of serious applications and that the only good route involves trying a variety of different implementation strategies and comparing their performance in both micro and macro benchmarks. That said, there were a few rules that served me well.

1. Use native CLR constructs whenever possible. These constructs are all heavily optimized by the underlying runtime engine. Sometimes these constructs will have to be used creatively to properly match Python's dynamic semantics.
2. Use code generation to produce fast-paths for common cases. This can either be development-time code generation with Python scripts creating C# code or run-time code generation using System.Reflection.Emit to produce IL on the fly.
3. Include fall-backs to dynamic implementations for less common cases or for cases where Python's semantics requires a fully dynamic implementation.
4. Finally, test the performance of every CLR construct used and find alternatives when performance is unacceptable. Type.InvokeMember is a great example a a function that must never be used in code that cares about performance.

Incredibly fast?

[Fweeky]

A 50% speed boost isn't *that* much; The Python JIT runtime that was on SlashDot just a few weeks ago cited significantly higher increases in performance. Fast, yes, but I'm not amazed or anything here :)

Vaporware

[hummassa]

the problem is that this stuff does not exist (until proven otherwise)

Re:Vaporware

[gtrubetskoy]

the problem is that this stuff does not exist (until proven otherwise)

It does exist, it is just not Open Source (at least yet).

Re:Vaporware

[hummassa]

It's not even closed source. There is no source or object code anywhere. If I can't test it, it does not exist. Even closed source can be demonstrated to exist. This can't.

Platform

[Hard_Code]

CLR is starting to become a prominent candidate for "open source development platform". Things like IronPython just make this sweeter. One-off interpreters are just a waste of time these days. You can get immense value by creating a generic-enough VM (like Parrot or CLR or JVM) and running <your-favorite-language> on it. Then you get all the benefits of consolidating effort on making the platform better. Next up should be PHP and Ruby (if they haven't already).

Re:Platform

[Anonymous Coward]

Sorry, NO PHP for CLR. Any language that contains a self-intrepreter as part of its language set proves particularly nasty to compile.

Re:Platform

[eurleif]

Like Python, you mean? It has an eval() function and an exec statment. From the interactive command line:

>>> def foo(): print "Foo called."
...
>>> eval('foo')
<function foo at 0xf70538b4>
>>> eval('foo()')
Foo called.
>>>

Re:stuck?

[LWATCDR]

I would not bet that most free software machines do not have a JVM installed. VMs are a great chance for OpenSource to really shine. I used java for an internal program at my company. It is a mission critical app and has worked well for two years now. The best part is that it also runs on Linux. When all of the mission crtical apps are available on Linux then we can move a large section of our office to Linux workstations.
What most people do not understand is that for the vast majority of business apps the choke point is IO and not CPU speed.

Re:stuck?

[jovlinger]

Much of python's slowness comes from its meta programming ability: variables and class members basically HAVE to be implemented by hashtables.

If you're willing to give up the dynamic nature and instead settle for scheme-y features, it could run a whole lot faster.

Re:Am I paranoid?

[Anonymous Coward]

No, you're not paranoid. You're just plain retarded.

Re:stuck?

[Euphonious Coward]

good luck getting eval to work in gcc

Most Free Software platforms have Gcc on them already. Gcc could be part of the runtime, in place of some anonymous JITter.

jython and gcj?

[pb]

I don't think a Python front end for gcc would be impossible, but it'd need its own set of libs, and it'd probably have a fair amount of code added in for when interpretation is necesssary. That having been said...

Maybe gcj would be a better place to start. Or, for that matter, maybe you could try to get jython compiled with gcj; sounds like it's been done before.