Python-to-C++ Compiler

Mark Dufour writes "Shed Skin is an experimental Python-to-C++ compiler. It accepts pure, but implicitly statically typed, Python programs, and generates optimized C++ code. This means that, in combination with a C++ compiler, it allows for translation of pure Python programs into highly efficient machine language. For a set of 16 non-trivial test programs, measurements show a typical speedup of 2-40 over Psyco, about 12 on average, and 2-220 over CPython, about 45 on average. Shed Skin also outputs annotated source code."

Comments

Sounds good...

[bblazer]

This is an interesting development. Programmers can now use the simple syntax of python and create faster machine code. THis may make rapid prototyping even more rapid, and allow programmers with little or no C++ experience create code that will run faster and will not require someone to install Python to run something. I will have to explore it more, but it will be intriguing to see how they handle things like pointers and structs that are not in python.

Re:Sounds good...

[B'Trey]

I will have to explore it more, but it will be intriguing to see how they handle things like pointers and structs that are not in python.

Uh, why would they have to? This goes from Python to C++, not vice versa. If there are no pointers or structs in the Python code, why would they have to handle them? Certainly, it's quite possible that some Python variable types will be converted to pointers or structs in the output code, but that's orthagonal to the issue of Python not having them natively.

If you were trying to go from C++ to Python, then you'd have to convert C++ pointers and structs to some sort of Python data type, and your comment would make sense. As it is, I'm not sure what you were trying to say.

Re:Sounds good...

[masklinn]

it will be intriguing to see how they handle things like pointers and structs that are not in python.

Why would one ever need to do that? The goal is not to write C++ in Python, it's to compile Python to machine code via an intermediate Python -> C++ compilation.

Re:Sounds good...

[Frizzle Fry]

THis may make rapid prototyping even more rapid

If you are just making a prototype, why is squeezing out extra perf so important? Prototyping is the sort of situation where you should be fine with just using straight python.

Re:Sounds good...

[bblazer]

I do understand that Python does not have structs or pointers. But pointers are fundamental to C++. I was very unclear in my first post. I was wondering if they would be converting certain types to pointers and I was curious how that was going to happen.

Re:Sounds good...

[j35ter]

If there are no pointers or structs in the Python code, why would they have to handle them?

whenever you pass a object to a method, you actualy pass a pointer who references this object(struct). Python just helps you not to think (too much) about it :)

Re:Sounds good...

[An Onerous Coward]

I don't think this will help with "rapid prototyping," since it has to add an additional compilation step to your change, test, change cycle.

Nor do I think most projects will be able to depend entirely upon it. Looking at the limitations listed in the announcement, it seems like it takes away many of the things that make Python engaging as a language. What seems to remain? Imagine taking a compiling C program, translating it (as literally as possible) into python, and then let this program translate it into machine code for you.

I'm probably overstating the case. There will be certain subsets of code that might benefit from this (where people need the performance, and are willing to give up architecture independence to get it). But even if this thing matures greatly (and I'd like to see that happen), I don't think it will become ubiquitous.

Re:Sounds good...

[Anonymous Coward]

It's not orthAgonal, it's purpindekular :-)

not terribly useful quite yet

[Surt]

Until he addresses mixed types in n-tuples, this won't be useful for very many people.

Re:not terribly useful quite yet

[goombah99]

But he's on the right track. Python allows dynamic typing but nearly all of ones programs do not take advantage of it. Recognizing that is key to making it go fast I think. It would be nice to have a filter you could run over python that would find all the type ambiguous points and let you insert some sort of compiler hinting.

I could envision it working like this. Instead of statically declaring all your variable types in every function, you instead simply declare that whatever tpyes are being used, they are always the same every time this function gets called. All the compiler then has to do is to find one instance of calling that function or one instance of the use of the arguments within the subrotuine in which the type is unambiguous to reverse engineer the types without having to be told. It could then flag the few cases where it can't resolve the type and either handle those in a slower dynamically typed fashion, or allow you to hint the types it was confused by.

Re:not terribly useful quite yet

[Surt]

I could well even imagine that using RTTI in C++ he could actually just split out whatever logic expected different types. Certainly there's some work there, but it seems plausible to me.

Re:not terribly useful quite yet

[Ed Avis]

I suggest you try Haskell or another language with type deduction. You don't normally have to declare the type for any function you write, but you can add explicit type declarations to help catch errors or as a form of documentation (which is checked by the compiler, of course).

Re:not terribly useful quite yet

[jgrahn]

But he's on the right track. Python allows dynamic typing but nearly all of ones programs do not take advantage of it. Recognizing that is key to making it go fast I think.

I suspect that varies with the programmer. I'm pretty certain that much of my Python code contains things that a type deduction system (SML, Haskell) wouldn't be able to cope with. Certainly I use duck typing a lot.

And besides, only one of maybe a hundred Python program I've written ran unacceptably slow. And that was a quick hack for an IP packet analysis tool, which someone suddenly wanted to feed dozens of megabytes of data.

I'd like to have type deduction -- but for static checking purposes, not speed optimizations.

Re:not terribly useful quite yet

[salotia]

This could be an approach for implementing n-tuples :-)

http://www.boost.org/libs/tuple/doc/tuple_users_gu ide.html

Re:not terribly useful quite yet

[Haeleth]

I suspect that varies with the programmer. I'm pretty certain that much of my Python code contains things that a type deduction system (SML, Haskell) wouldn't be able to cope with. Certainly I use duck typing a lot.

How exactly does duck typing differ from the structural subtyping of e.g. OCaml, which allows you to write a function that can be passed any object, of any class or none, if it provides all the methods that function uses? The type inference system handles it just fine.

Of course, "duck typing" isn't a very well-defined term. Maybe your definition includes things like automatic coercions, introspection, and polymorphic recursion, which are indeed not permitted in OCaml's type system. In that case, you're probably correct in your suspicion that its benefits would not outweigh its disadvantages for you. Though you might find it useful to dabble in anyway; there's nothing so educational as frustration.

And besides, only one of maybe a hundred Python program I've written ran unacceptably slow.

<flamebait> I didn't think my old 386 was unacceptably slow, either... in the days when that was what I was used to. </flamebait>

To be fair, Python is indeed plenty fast enough, as long as all the computation-intensive stuff is happening in external libraries written in C. It's a great language for glue and GUIs (with wxPython). It does get rather slow rather quickly if you try to do anything significant in pure Python code, though.

Why not just use pure C++?

[Anonymous Coward]

Why not just use pure C++?

Re:Why not just use pure C++?

[bzerodi]

Why not pure assembler ?

Re:Why not just use pure C++?

[boggartlaura]

Well, personally, I like Python more. I'm not against C++, but I find Python works better and is easier to use for what I want to do.

Re:Why not just use pure C++?

[Anonymous Coward]

Why not pure 1's and 0's?

Re:Why not just use pure C++?

[tiocsti]

Portability. Besides, why not leverage the decent optimization that already exists in the gnu compiler collection? I guess for me, a better question might be, why not a python frontend to gcc.

Re:Why not just use pure C++?

[masklinn]

That's pretty much what he's doing, ShedSkin is a Python to C++ compiler, then you need to compile the C++ code ShedSkin yields to machine code, you can do that with gcc.

The goal (for the author) at the moment is to get a fairly complete Python to C++ compiler (ShedSkin is already very good if you're mostly doing simple operations such as crunching numbers, but if your program is really complex or uses libraries then you're out of luck)

Re:Why not just use pure C++?

[SigmoidCurve]

bzerodi's point, made with Zen-like simplicity, is that language choice should be made to minimize programmer time, not machine time. I am at least a factor of ten more productive with Python than with C or C++. I am also far more confident in the correctness of what I write per line of Python than with what I write per line of C/C++.

Yes, I have have wasted some time staring at the shell waiting and waiting for it to return from some complicated Python routine. I know that compiled C would faster, and hand-rolled assembler would be faster still. But I say to myself: hey, I wrote this code in a single afternoon, how many weeks of hair-pulling would it take to re-engineer this - and make it bug-free - in C? When I put it that way, I don't mind waiting the extra minutes for Python to do my dirty work.

As a previous poster mentioned, the ability to handle tuples of mixed-types is critical. I look forward to seeing great things from Shed Skin in the future.

Re:Why not just use pure C++?

[tiocsti]

I meant frontend in a specific gcc sense (as in, something that parses the language and outputs rtl which can be consumed by gcc backends), not as a generic intermediate step that will eventually be compiled via gcc (or whatever), as that's generally true of most things that output c or c++ code these days, yeah?

Re:Why not just use pure C++?

[b17bmbr]

After four hours of tweaking, our expert C++ programmer was finally able to write something that beat our ten lines of Python code that took under five minutes to write. And it didn't beat it by much, whereas the first pass at a C++ version was an order of magnitude slower.

Which is why languages like python were written in the first place. They pretty much just make the underlying C calls anyways, but do so in a way that handles buffer overflows, pointers, etc., that pretty much make C/C++ so troublesome, hazardous, and hard to learn. I like java (alot really), but nothing beats a good scirpting language, like perl or python, to handle tasks like text manipulation. Python is especially good at using libraries, such as the imaging library, which are written in C anyways. How much faster can you get calling a C library from C than from python? I honestly don't know, but I can't imagine it's that much more. But when you add in speed of development, safety, and even portability, it's powerful.

Python's OOP is also a feature that makes it far more attractive than perl for me. Perl does OOP, but it's not as clean as python's, and I don't think it supports all the OOP features either. Doing GUI's is not the strength of any scripting language, but it depends on what you need to do. You can write a native frontend and embed python into a C or even a java application.

Re:Why not just use pure C++?

[dancpsu]

The programming language can encourage the programmer to write an application faster. People have pointed this out before. But also, a programming language can encourage better programming principles. C++ makes it difficult to use complex data structures, while scripting languages like Perl and Python make it a breeze. Python also has the advantage of making object oriented programming simple, whereas the convoluted structure of C++ struct-based objects discourages programmers from taking advantage of them.

Re:Why not just use pure C++?

[ZephyrXero]

"Doing GUI's is not the strength of any scripting language..."

This is why projects like pyGTK exist ;)

Re:Why not just use pure C++?

[joss]

Sorry, but without more details it would seem to me that
your "expert" C++ guy wasn't an expert. Can you describe the
problem a little better.. if what you say is true, I as
a long term C++ programmer would consider switching, but
I've looked at python, and I simply don't believe you.

I'll grant that C++ is a nightmare for beginners with more pitfalls
than an indiana jones movie, but once you know them, writing
poorly performing code is unlikely.

Re:Why not just use pure C++?

[19thNervousBreakdown]

Why would you even try to compete with file I/O? Disk access is slow enough that an interpreted language is going to have no problem keeping up with a compiled one. As for the 4 hours, yeah, there's some stuff that Python just does better, but I'm inclined to say they just didn't know what they were doing, mostly because they even attempted competing on the I/O front.

Re:Why not just use pure C++?

[mrchaotica]

Not to mention that you can speed up execution time by throwing more hardware at it. If you try that with programmer time you just end up with a bruised programmer, and nobody wants that!

Re:Why not just use pure C++?

[mypalmike]

C++ makes it difficult to use complex data structures, while scripting languages like Perl and Python make it a breeze.

Complex data structures in Perl? Such pain I wish to never endure.

Re:Why not just use pure C++?

[Bill Dog]

They could be experts in C++, and not be already familiar with the highly unusual case of manipulating insanely large text files efficiently. Like me.

Re:Why not just use pure C++?

[Bill Dog]

C++ makes it difficult to use complex data structures...

It does? I've always managed, somehow. If you're referring to things like structures of pointers to structures, smart pointer techniques have been around for ages that manage cleanup. Maybe not as trivial as a scripting language, but not difficult.

Python also has the advantage of making object oriented programming simple, whereas the convoluted structure of C++ struct-based objects discourages programmers from taking advantage of them.

Nice talking out your ass there. Ya, C++ programmers everywhere refuse to use struct-based objects (i.e. C++'s classes) and OOP because they're so discouraging.

Re:Why not just use pure C++?

[Anonymous Coward]

ShedSkin shows that it's possible to write fast programs with a really simple syntax.

Re:Why not just use pure C++?

[baadger]

Programmatically creating GUI's? Ewww, why not just take another leap and go with the python bindings for libglade?

Re:Why not just use pure C++?

[Schraegstrichpunkt]

I think the Python Glade bindings require the GTK bindings.

Re:Why not just use pure C++?

[radtea]

I like java (alot really), but nothing beats a good scirpting language, like perl or python, to handle tasks like text manipulation.

You say that like you think Java isn't a scripting language, but an analysis of language features, like anonymous inner classes (which encourage on-the-fly, non-designed extensions to applications) clearly shows that it is more appropriate to scripting than an applications development, particularly if you care about run-time performance (yeah, I know, with the right JVM and stuff Java can go fast--so prove to me that my customers will be running that JVM and that it will support all the language features I need for the specific application I'm building--"can go fast" != "does go fast".)

Re:Why not just use pure C++?

[Anonymous Coward]

C++ makes it difficult to use complex data structures

Now that compilers have actually caught up to the standard this is flat out false, the STL makes working with data structures at least as easy in C++ as it is in perl for standard cases, and easier for complicated cases. Many problems with perl "TMTOWTDI" become evident with complex data structures, you can turn on all the strict checking and warnings you want - but complex perl structures are very prone to doing the wrong thing silently.

Re:Why not just use pure C++?

[Geoffreyerffoeg]

Why not just use pure 0's?

After all, if you can express a program in binary, you can convert that binary string to a number. Just count off that many 0's (like they were tally marks).

Re:Why not just use pure C++?

[baadger]

Yes they do, hence a leap further and not a leap in another direction :P

Re:Why not just use pure C++?

[Anonymous Coward]

Why not just use pure 0's?

After all, if you can express a program in binary, you can convert that binary string to a number. Just count off that many 0's (like they were tally marks).

I won't even begin to think of ways that that is stupid.

Re:Why not just use pure C++?

[gnud]

I don't get why parent recommended pyGTK, but for wxPython or pyQT, you have gui builders.
WxPython has some commercial offerings, plus wxglade (it can output pure python code or xml) and boa constructor.
pyQt has Qt designer and pyuic (py user interface compiler, compiles the designers .ui file to python code).

Re:Why not just use pure C++?

[cervo]

One great example is text manipulation. A few lines of regular expression code in Perl/Python/Ruby can save hours of coding in C++ (well without a third party regular expression library). I'm not saying that this is the problem, but for complicated text parsing, the power of regular expressions is one big advantage of a scripting language over C++.

Re:Why not just use pure C++?

[Anonymous Brave Guy]

C++ makes it difficult to use complex data structures...

It does? I've always managed, somehow.

As have I, but I'd certainly rather manage in languages that support first order data structures, "for each" loops for iterations, proper disjunctive types, pattern matching, and so on. C++ is better than it used to be, but all the data structures and algorithms in the standard library barely hold a candle to the expressive power of many functional programming and "scripting" languages.

Re:Why not just use pure C++?

[Bill Dog]

Expressive power, or syntactic sugar? To me the former means all the things I can accomplish, not all the ways I can accomplish the same things.

Re:Why not just use pure C++?

[joss]

Before boost::regex was stable, ie 3 or 4 years ago, I would have agreed with you.

Re:Why not just use pure C++?

[Anonymous Brave Guy]

I don't think expressive power and syntactic sugar are independent concepts in practice. (C.f. any language is syntactic sugar, equivalence of Turing-complete languages, everything gets turned into machine code eventually, etc.)

But the original comment was that C++ made it more difficult to use complex data structures, and given that we can express several useful concepts much more concisely using language features such as those I mentioned than is possible with C++, I think that's a fair claim.

Re:Why not just use pure C++?

[juhaz]

They don't just require them, libglade bindings are part of pygtk, and as such, it's not a leap at all, and your original objection is pretty much pointless. The grand-grand-parent could've just as well meant using libglade as "programmatically creating guis".

Although if you do need to do the latter, python, with very little boilerplate required, is one of the best languages to do it in...

Re:Why not just use pure C++?

[juhaz]

Doing GUI's is not the strength of any scripting language, but it depends on what you need to do.

Why is that? GUI's are, if possible, even more about using existing libraries and "backend" applications than any other software, which, as you stated, is what "scripting" languages do so well.

And to boot, GUI applications spend 99.9% of their time idle, waiting for user input, so the relative slowness also matters less than usually.

2-40 what?

[Rorian]

2x to 40x speedup? 2% to 40%? 2 to 40 seconds?

Standardised units are your friend.

Re:2-40 what?

[chills42]

2 times as fast up to 40 times as fast is the Standard interpretation...

Re:2-40 what?

[Rorian]

I'm sure it is, but I hate it when people don't specify :(

Re:2-40 what?

[Schraegstrichpunkt]

"Times faster" is a unitless quantity.

Re:2-40 what?

[Anonymous Coward]

Uh, no. If I said it was 2-40 seconds faster would that be any better? What if the original time was 3 hours? And what if it was 2.01 seconds?

Times faster is entirely appropriate for this summary.

Re:2-40 what?

[Schraegstrichpunkt]

So, you're suggesting that it be reworded: "a typical speedup of 2-40 times faster"? Brilliant.

Ewwwww

[$RANDOMLUSER]

As a UNIX admin, I was saddled with one of these kinds of things years ago, a DEC-BASIC to C compiler for UNIX. The output code quality was incredibly bad: machine generated variable and function names, bizarro nested struct/union/struct data structures, 400-line functions peppered with calls to 1-line functions. Completely unreadable. Thank $DEITY that project died quickly.

Re:Ewwwww

[Anonymovs Coward]

Completely unreadable.

I think you're not supposed to read it. You're only supposed to feed it to your C++ compiler. f2c produced unreadable output too, but nobody read the output; at one time it was the only free fortran option on linux.

Re:Ewwwww

[Virak]

Which is why I suggest you use brainfuck for all your coding needs. The generated code will make just as much sense as the original, if not more.

Re:Ewwwww

[Anonymous Coward]

Ever try to read C-to-machien language compiler output? Good lord!

Re:Ewwwww

[IamTheRealMike]

If you actually tried ShedSkin you'd find the C++ it produces is very similar to what a human might produce, and is actually quite easily readable. But then - why would you want to anyway? It's an intermediate form useful to pass to an optimising C++ compiler, not as something to read.

Re:Ewwwww

[Tim Browse]

Yeah, whenever I look at the output of my optimising compiler, it's really hard to understand too. It's all in assembler, for a start.

Plus, the quality of C code generated by CFront was rubbish - unreadable.

Same with the Modula-3 compiler I tried. You couldn't work out what was going on in the resulting C code without a load of work.

Can you see where I'm going with this?

Re:Ewwwww

[akunak]

I too used f2c and had to put up with the unreadable code. For me it was an issue because I was using f2c to facilitate a conversion of a small amount old Fortran code to a Unix platform (i386 DG/UX) that had a free C compiler but an expensive or non-existent Fortran compiler. The issue was maintenance; the generated code became part of our production system.

In a previous generation we converted a major system from an odd dialect of Pascal to C using a heavily and brilliantly modified (as in convert a big program, compile, link, test with no errors) version of (I think Caltech's) p2c. In this case the win was no hand porting of several tens of thousands of lines of Pascal, but the loss was several years of dealing with at least some completely unmaintainable code. Imagine nice Pascal sets implemented with C bitfield operations (now that's total ewwwww.)

I think a code converter or generator can generate good code if the author actually thinks about it. Further, if the code generated is of high-quality, it makes debugging the code-generator or converter that much easier.

Re:Ewwwww

[alienmole]

Can you see where I'm going with this?

My guess is they probably can't. The idea of using a high-level language as an intermediate compiler target seems to require a mental leap for many people - they hear it's generating C, and they immediately assume that means it's meant to be read and worked on by humans. The parallels to assembler don't seem to register without an explanation.

Static Typing?

[Anonymous Coward]

If you're only allowed to use static typing, doesn't that defeat the purpose of coding in python in the first place?

Re:Static Typing?

[goofyheadedpunk]

No, not really. A large number of people, including myself, just use python as a nicer C. Futzing with pointers and other such things can be ingnored while making a prototype and, after finishing the prototype, the bits that need to be faster can then be rewritten.

I recently wrote a largish simulation in python for a Biology course. The goal was to watch how a species spread over a planet given other competing species, natural disasters and the like. It took four in deep hack mode to write the whole thing, all of it implicitely typed due to the equations at the base of the simulation. Implicite static typing is used a lot in large applications. So much so that in fact, if I recall, python 2.5 is supposed to have optional type declaration.

Re:Static Typing?

[Surt]

Well, it's not quite as bad as it sounds. He's seemingly only really forbidding incompatible mixed types in the same variable, a usage that isn't exactly extremely common.

A more significant roadblock, IMO, is that he can't handle mixed types in 3+-tuples, which is very common.

Re:Static Typing?

[masklinn]

1. Not necessarily, some things (such as reusing a name to bind widly different objects) is frowned upon and bad style anyway (since it makes the code much less readable and maintainable)
2. The compiler is still in a very early phase of it's development

Re:Static Typing?

[pthisis]

So much so that in fact, if I recall, python 2.5 is supposed to have optional type declaration.

No, it doesn't. AFAIK all the Type-SIG and other groups looking at it decided against it and the issue is dead.

Re:Static Typing?

[MBCook]

I love Python, but I hate the dynamic typing. It can be handy at times, but 99% of the time you make a variable to hold one kind of thing. Having the static typing would both improve performance (because the interpreter knew what you were up to) but would also eliminate bugs (because it would complain when I tried to set a double to "And now press...").

I'd love to see Python get optional static typing.

Re:Static Typing?

[mad.frog]

I'd love to see Python get *required* static typing.

But then, I guess it wouldn't be Python....

Re:Static Typing?

[Anonymous Coward]

I'd love to see Python get optional static typing.

Well, to each his own. But if that is the only thing you miss about python help is available.

Re:Static Typing?

[try_anything]

Didn't you know? Common Lisp had optional static typing in 1857. Then the Civil War came and people COMPLETELY forgot about it, babbling on about their Gatling guns and submarines and stuff that Lisp had had for years.

(I'm not really sure if Lisp's static typing guarantees compile errors. I'm just a beginner Lisp Weenie. I have potential, though, right?)

Re:Static Typing?

[Anonymous Coward]

It would be pike with hidious syntax. Better to just use pike with its nice syntax.

Re:Static Typing?

[Arkaein]

Maybe you should look into Pyrex, with is essentially Python with static, C primitive data types. I've never used it myself, and it's mainly intended for writing Python modules, but I see no reason why you couldn't write essentially an entire app as a Python module and call that from a very short Python script.

Re:Static Typing?

[chthon]

It depends upon the compiler.

CLISP ignores such statements, because it only compiles to bytecode anyway.

CMUCL and SBCL do generate compiler warnings, and sometimes even errors. Your static typechecking will not go unnoticed.

I think that to be good, the Python compiler should do the same, but that may mean introducing new statements in Python, like in Lisp :

(declaim (optimize (compilation-speed 0) (speed 3)))

Re:Static Typing?

[Anonymous Coward]

There are plenty of ways to simulate this: templates auto-generating as many different statically compiled versions as needed, a message passing system, functions inside hash tables....

Re:Static Typing?

[Anonymous Coward]

Or you could just use ML...

Re:Static Typing?

[jma05]

The time when Lisp dialects started using optional static typing, performance was still a big deal. Today it isn't. Plus Python has a large number of ways available to access native code with minimal fuss (mostly zero). There is simply no motivation to include that information and clutter the language anymore. In my 5 years of Python use, I never said - "Darn. This is too slow. I should have used a faster language". And I processed multi-gigabyte datasets.

Yeah, but that's not what we need.

[stonecypher]

See, it's all well and good to compile python to speed it up. The problem is, people are now saying that they can write efficient code in python just because it magically translates to C++, and because this translator is faster than other python compilers.

This won't be meaningful until a converted python script is compared to efficient code written natively in C++ in the first place.

Re:Yeah, but that's not what we need.

[Anonymovs Coward]

I don't see your point. Some of us use python. It takes me a fraction the time to do something in python than to do it in any other language. I'm not interested in writing native C++ code because it's hypothetically faster (it's not faster if I count coding time). But I am interested in a good python-to-C++ translator. Why wouldn't any python user be?

Re:Yeah, but that's not what we need.

[masklinn]

The problem is, people are now saying that they can write efficient code in python just because it magically translates to C++

No they aren't, if only because ShedSkin doesn't handle modules yet and Python without modules is not really useful.

and because this translator is faster than other python compilers.

Last time I checked, it was the only Python compiler... (CPython is an interpreter, PyPy is also an interpreter, I'm pretty sure Vyper was also an interpreter [written in O'Caml before it got trashed though], JPython and IronPython can hardly be called compilers)

Re:Yeah, but that's not what we need.

[advocate_one]

But I am interested in a good python-to-C++ translator. Why wouldn't any python user be?

no, I'd be far more interested in a good compiler to compile that python straight to machine code...

Re:Yeah, but that's not what we need.

[try_anything]

Yep, every new tool is just an opportunity for idiots to screw up in new ways.

What a gloomy way of looking at the world.

Re:Yeah, but that's not what we need.

[Abcd1234]

Why? If you can convert Python to reasonably optimized C++, then you can leverage the C++ compiler to do all the machine-level optimizations, rather than reinventing yet another wheel.

Re:Yeah, but that's not what we need.

[pthisis]

Last time I checked, it was the only Python compiler... (CPython is an interpreter, PyPy is also an interpreter

Neither CPython nor PyPy is a strict interpreter, both of them compile source to byte-code and then act as a virtual machine to run that byte-code. PyPy also does some work on compiling to native code on the fly, depending on which version you're using (Armin Rigo's is the most sophisticated on the JIT/native code front, but it's far from stable).

Re:Yeah, but that's not what we need.

[masklinn]

They're still not compilers. And CPython + Psyco is not a compiler either.

Re:Yeah, but that's not what we need.

[advocate_one]

it gives you an extra area for weird bugs to creep in... get the Python right and go straight to machine code with a trusted compiler.

Re:Yeah, but that's not what we need.

[mrchaotica]

...and that's why it shouldn't be a Python to C++ translator; it should be a GCC frontend instead (i.e., translating to GCC's internal representation).

Re:Yeah, but that's not what we need.

[styrotech]

it gives you an extra area for weird bugs to creep in... get the Python right and go straight to machine code with a trusted compiler.

Is that the same way the method of using layers of multiple simple tools that all do one thing really well is more buggy that just using one larger general purpose monolithic app?

A cross platform Python to machine code compiler would presumably need to reinvent a whole lot of difficult platform specific stuff that has already been solved by C++ compilers.

Re:Yeah, but that's not what we need.

[menace3society]

I'd prefer a python-to-Common-Lisp compiler, but only because I hate running out of stack space for recursive algorithms.

Re:Yeah, but that's not what we need.

[Schraegstrichpunkt]

Neither CPython nor PyPy is a strict interpreter, both of them compile source to byte-code and then act as a virtual machine to run that byte-code.

I think that falls under the standard definition of "interpreter".

Even ancient versions of MS BASIC were bytecode-oriented. They had to be to fit any decent-sized program into the limited RAM available.

Re:Yeah, but that's not what we need.

[Abcd1234]

Which has already been covered elsewhere. The GCC IL isn't suitable for a dynamic language like Python.

Re:Yeah, but that's not what we need.

[rbarreira]

Not quite true. Analogy:

Would you also like to translate a text from Arabic to English by passing through 3 or 4 languages in between?

In this analogy the problem would probably be accuracy, in the case you presented it would be performance being lost due to layers of conversion. Some high level optimizations are inevitably lost (unless the C++ compiler has some sort of strong AI).

Re:Yeah, but that's not what we need.

[Anonymous Coward]

The GCC IL isn't suitable for a dynamic language like Python.

Assuming the Shed Skin project overcomes the dynamic typing (and other) limitations so that any Python code is translatable to semantically equivalent C++, then it should be possible to skip the translation to C++ and go straight to GCC IL. Of course, that may prove to be too much of an assumption...

- T

Re:Yeah, but that's not what we need.

[Anonymous Coward]

Doesn't gcc do tail-call elimination?

Re:Yeah, but that's not what we need.

[Anonymous Coward]

Is that the same way the method of using layers of multiple simple tools that all do one thing really well is more buggy that just using one larger general purpose monolithic app?

That works great if you already have multiple simple tools that all do one thing really well. It works really poorly if you don't bother to check that your tools satisfy all those conditions. GCC is definitely not a simple tool that does one thing really well, and neither are most other C++ compilers.

Re:Yeah, but that's not what we need.

[Anonymous Coward]

Even ancient versions of MS BASIC were bytecode-oriented.

Not quite. They were tokenized: small integers were subsituted for common syntactical elements ("PRINT").

Re:Yeah, but that's not what we need.

[Samurai Crow]

Which machine code? That's the big question. Certainly Intel would be happy to have you believe that their processors are the only ones available but what about the PowerPC? Python is used in game scripting as well as server work so why not for the game platforms?

Re:Yeah, but that's not what we need.

[pthisis]

Neither CPython nor PyPy is a strict interpreter, both of them compile source to byte-code and then act as a virtual machine to run that byte-code.

I think that falls under the standard definition of "interpreter".

I don't. It's a mixed system, with both a compiler and an interpreter. A strict interpreter runs directly on the source code (cf Bash).

Note that a compiler doesn't have to be an ahead-of-time process that generates machine code directly. Many C compilers generate assembly code that is then compiled by a seperate ASM->object code compiler. On many modern machines, even that object code isn't the low-level machine code, the CPU breaks it down into micro-ops on most Intel/AMD machines and things are even murkier on Transmeta machines.

And PyPy has multiple backends; it can generate machine code directly, llvm code, CLI code that runs on a .NET runtime, etc. It also has multiple frontends (for a number of toy languages, and for Lisp, Javascript and others). I don't know if the PyPy JVM back-end is in a workable state, but jython certainly compiles Python to java bytecode.

But even in the case of CPython, it's very similar to the Java model with a compilation phase and an interpreted phase. You wouldn't say that javac isn't compiling if it happened to be in the same binary as the JVM.

Re:Yeah, but that's not what we need.

[pthisis]

They're still not compilers. And CPython + Psyco is not a compiler either.

Yes, they are. They translate from one language to another. In the case of Psyco, that language happens to be machine code; in the case of CPython it's bytecode. They also have an interpreter (or runtime, virtual machine, whatever you want to call it) which is no different from, say, C++ having a runtime required in addition to your code (and like C++ you can statically link the runtime into a standalone executable if you want to).

You can also run the compiled bytecode directly (and delete the source code) with CPython if you want to.

FWIW, PyPy has a number of backends (it can generate llvm bytecode, native code, CLI code that can run on a .NET runtime, etc) as well as multiple frontends (Python, Javascript, Lisp, etc).

Re:Yeah, but that's not what we need.

[stonecypher]

I don't see your point.

Indeed. But you replied anyway.

I'm not interested in writing native C++ code because it's hypothetically faster (it's not faster if I count coding time). But I am interested in a good python-to-C++ translator. Why wouldn't any python user be?

I never said they wouldn't be. Please feel free to re-read what I said until you understand it. However, please don't reply again and attempt to argue with me over things I didn't say. It's obnoxious.

Re:Yeah, but that's not what we need.

[stonecypher]

(unless the C++ compiler has some sort of strong AI)

I can't imagine by what process of thought you came to think this was a useful thing to say. Strong AI is AI which is self-aware. It has nothing to do with problem solving capabilities. Furthermore, strong AI does not yet exist. Moreover, compiler optimizations are a set of rule-driven alterations based on mathematical proof that things aren't changed; theoretical AI wouldn't actually help in any way.

Read a Searle book before engaging in this sort of self congratulatory nonsense again, please. You're embarrassing people who know what you're trying to talk about.

Re:Yeah, but that's not what we need.

[stonecypher]

The problem is, people are now saying that they can write efficient code in python just because it magically translates to C++

No they aren't, if only because ShedSkin doesn't handle modules yet and Python without modules is not really useful.

RTFA. They're saying this verbatim. This means that, in combination with a C++ compiler, it allows for translation of pure Python programs into highly efficient machine language.

Last time I checked, it was the only Python compiler

Then you need to check again. Arguing from ignorance doesn't do much other than to make you feel smart; if looking smart is what you're after, you're out in the cold.

Re:Yeah, but that's not what we need.

[stonecypher]

How can I resist?

The phrase "grammar nazis" is properly spelled as a capitalized compound noun: Grammarnazis.

No.

1. 
      
2. A phrase is not collapsed into a single word for no apparent reason.
3. 
      
4. What you're referring to are called compound words, not compound nouns. Adjectives are also frequently compound words, though people have a tendency to mistakenly hyphenate such words almost as frequently.
5. 
      
6. Compound nouns are not capitalized. Capitalization is used for sentence leading, for specific nouns, for particular nouns, for names, and not for making yourself look stupid on Slashdot.
7. 
      
8. New compound nouns are essentially not being made; for the last fifty years, hyphenation has been a near-dominant standard for modern English concatenation.
9. 
      
10. The phrase is correctly written 'grammar Nazis;' Nazis are a particular political group and are as such proper nouns.
11. 
       
12. You seem to have 'proper' and 'correct' mistakenly backwards. To be proper is to follow appreciable and estimable rules of conduct in a setting of judgement. This by definition excludes things for which there are a correct and an incorrect. For example, two plus two is not properly written as four; it is, rather, correctly written as four. (To use the word proper indicates something different, in the situation where the sum rather than its expression was required, such as on a tax form; the contrasting example would be '2+2' rather than an incorrect result.)
13. 
       
14. In English, double quotes indicate a specific quotation. This is an inspecific quotation - something which is commonly or theoretically said, rather than something that was said by a specific person on a specific date.
15. 
       
16. In English, the colon is used to indicate a list of items. Given that English does not have datatypes, whether something is a list is governed by whether there are more than one; otherwise you are dealing with an instance. The correct way to phrase what you said would be to refer to the replacement as the replacement; it is no less a single and specific thing because it's hypothetical (nor because you don't know how to conjoin.) Try ... is spelled as the compound noun 'Grammarnazis.'
17. 
       
18. Conjoinder of words is a conjugation issue, not a spelling issue.

Perhaps wait until you understand English before condescending to tell grammar Nazis about their own job, kthx.

Re:Yeah, but that's not what we need.

[stonecypher]

Best slashdot bug ever.

Re:Yeah, but that's not what we need.

[rbarreira]

OK, the term Strong AI was not the most fortunate but the idea still stands and it's valid.

Moreover, compiler optimizations are a set of rule-driven alterations based on mathematical proof that things aren't changed; theoretical AI wouldn't actually help in any way.

There are many optimizations which currently, humans can do and computers can't. Do you know of any compiler which can use knowledge about Fermat's theorem to optimize a computer algorithm? That's the kind of optimizations I was talking about, high level transformations which require very advanced reasoning to perform. That's the kind of optimizations which are very hard to do when a program is transformed to a different language, since the high level concepts are there only implicitly and very hard to discern. Just because current compilers don't do those advanced optimizations it doesn't mean that they are impossible (unless you have a very narrow view).

Re:Yeah, but that's not what we need.

[Schraegstrichpunkt]

It's a joke? "Nazis" is German and capitalized compound nouns are the norm in German.

Re:Yeah, but that's not what we need.

[stonecypher]

That would be Nazigrammatik. Thanks for playing; you still fail. By the way, Nazi isn't a word in German. It's an acronym. German doesn't allow the adoption of acronyms into compound nouns. Does it embarrass you to be corrected in your native tongue by an American?

Re:Yeah, but that's not what we need.

[Schraegstrichpunkt]

That would be Nazigrammatik.

Yes, it would be -- if I were writing in German.

Thanks for playing; you still fail. By the way, Nazi isn't a word in German. It's an acronym. German doesn't allow the adoption of acronyms into compound nouns.

You are taking this far to seriously. I know that it's ridiculously inaccurate. It's a joke. Get over it.

Does it embarrass you to be corrected in your native tongue by an American?

No, because German isn't my native tongue. In fact, I barely speak it. I am somewhat proud of myself, though, since I apparently writing an effective troll without even trying. :-P

Have a nice day!

Re:Yeah, but that's not what we need.

[Schraegstrichpunkt]

Oh, and now for my amusement, I don't think it's a Slashdot bug; It's probably a PEBKAC bug.

This:

• Test
• Test 2
• Test 3

is produced by this code:

<ul>
<li>
Test
</li>

<li>
Test 2
</li>

<li>Test 3</li>
</ul>

Thanks for playing!

Re:Yeah, but that's not what we need.

[stonecypher]

Man, you're just desperate to be right.

Re:Yeah, but that's not what we need.

[stonecypher]

Hahaha, let me get this straight. First you're going to make a "joke" about how to correctly write something. Then, you're going to try to tell me that I'm wrong to point out several flaws because it's a word following German rules. Then when I point out that that's just as wrong, you're just going to start saying "omg it's a joke don't take it so seriously!" ?

Here's the germane part: it makes you look like a dumbass. Fix it or replace it.

Re:Yeah, but that's not what we need.

[stonecypher]

This was my point exactly. The article says "this thing does a better job of converting Python to C++ in terms of efficiency than did the older one." People are hearing "This thing generates efficient C++." Nobody's tested that yet, though.

You are making a gigantic assumption that because this converter's better than the last one, that it's usable in efficiency arenas. By comparison, you might be looking at the difference between a shoe and a shoe with a spring (that's what air pumps do, don't laugh) when dealing with runners - you can grab another 10-20% speed out of the runner with the new shoe, and it's easier on the knees to boot.

They're still not racing a car.

I'm not saying this thing does a bad job; I honestly have no idea. The point, though, is that neither do you. If you're working a project which has a good reason to be Python, such as in an arena where programmer time is more important than execution time (a lot of programmers unfortunately believe that this is all programming, because they've never done performance-conscious things like operating systems, databases, video games, embedded or realtime software, and so on,) then this is great. Why? Because a new tool gives you an essentially free linear speed multiplier, which means you can crank X% more work out of the same machine farm, or whatever.

But, you've read this differently. You're starting to think of Python as an appropriate tool to write efficiency-conscious software, and there are not yet appropriate tests to display that this is an accomodating such tool. To be frank, I highly doubt that this is actually going to be a real performance-appropriate concern, and whereas a bunch of flametards will jump all over me demanding that I explain a lifetime's worth of instincts and experience to them in two paragraphs or I'm so obviously wrong, I still want to point out that technologies like this have come and gone for scripting languages for decades, that none of them have turned the python of their day to the C++ of their day, and that I don't see any compelling reason to believe that this will be any different.

What you're reading isn't what those tests say. Python is not a performance-appropriate language, and I believe that this tool will not make it so. No tests which determine whether it actually is a performance-appropriate environment have yet been run.

Besides, it's worth pointing out to all the flametards that C++ isn't actually the performance language of choice either. Depending on the nature of your problem, that crown usually goes to forth, erlang, k, mozart-oz or formulaONE. (And, for math-heavy stuff, it still goes to Fortran surprisingly often.)

Re:Yeah, but that's not what we need.

[Schraegstrichpunkt]

Oh no, I might look like a dumbass on Slashdot! Heaven forbid!

Re:Yeah, but that's not what we need.

[scott_karana]

Yeah, because everyone uses GCC, and the fact that C++ is standardized doesn't mean anything.

Re:Yeah, but that's not what we need.

[Breakfast Pants]

So, back when the G++ guys were deciding to make the C++ front-end to GCC, they should have instead just compiled the C++ to C++, sinc you know everyone uses GCC, and the fact that C++ is standardized doesn't mean anything.

Re:Yeah, but that's not what we need.

[Breakfast Pants]

The translation to bytecode isn't much more than what an assembler does... maybe it's an assembler.

Re:Yeah, but that's not what we need.

[pthisis]

I suppose you could argue that the CPython bytecode compiler is basically an assembler for the CPython VM.

Still doesn't change the point, though; an assembler is a particular kind of compiler.

Very nice, but...

[MikTheUser]

Since Python is about the only language I know very well, I find this fascinating. But it also reminds me of the .NET development suite, where the way you write your code in any language doesn't matter, since it all become one machine code. So if you think you can do something more memory efficient in C# than in VB.NET - well, no.
So the bottom line is, the quality will depend on the quality of the converter, and that's not so cool. Adding layers between code and machine code is not the best way IMHO.

Re:Very nice, but...

[jallen02]

Except that in .NET it all becomes MSIL, not Machine Code.

Jeremy

Re:Very nice, but...

[cnettel]

Well, C# has unsafe arrays, while VB.NET only exposes them quite indirectly through the marshalling API. Some other language implementations also uses some dose of reflection/late binding to implement certain features. You can sometimes avoid use features, but this will sometimes result in code that is "non-idiomatic" in that language. I like the .NET framework, but it's no panacea for a language-agnostic future.

Re:Very nice, but...

[rjshields]

MSIL is machine code for a virtual machine rather than a physical one. This distinction makes no difference to the point the GP was making.

Re:Very nice, but...

[2short]

"So if you think you can do something more memory efficient in C# than in VB.NET - well, no."

Well, yes. All code in any language whatsoever becomes machine code before it runs on the processor, and I can certainly do things more memory efficiently in some languages than others, so it's at least theoretically possible. C# and VB.NET code both become the same variety of intermediate binary code before being translated to machine code, but that doesn't mean anything you can do in one can be done in the other.

As for this converter, and what it means for Python, I've no real idea. But it certainly doesn't mean you can now do anything in Python you could do in C++.

Re:Very nice, but...

[fotoflojoe]

Well, yes.
Well, no. With .net, the language used becomes a mere choice of syntax. All .net languages are boiled down to MSIL and are on a level playing field. Whatever .net language you write in, it's all just food for the CLR. And the whole thing is just swiped from Sun - conceptually that is. C#, VB.net = Java, CLR = JVM

Re:Very nice, but...

[2short]

Well, Yes. Consider that by your logic, all languages everywhere should be equal:

With any particular processor, the language used becomes a mere choice of syntax. All languages are boiled down to machine code and are on a level playing field. Whatever language you write in, it's all just food for the processors instruction set.

C++ and BASIC both get reduced to x86 machine code before running on my machine. This does not mean you can do anything in BASIC that you can do in C++. There are things you can do in C++ that produce x86 code that can not be produced by BASIC. Similarly, there are things you can do in C# that produce CLR code that can not be produced by VB.net, and vice-versa.

FWIW, compiling code to a lower level machine independent language was not invented by Sun. BCPL was being compiled to O-code in the late 60s. There may be earlier examples, there are certainly several dozen between there and Java.

Re:Very nice, but...

[fotoflojoe]

You don't get it.


There are things you can do in C++ that produce x86 code that can not be produced by BASIC.

True.

Similarly, there are things you can do in C# that produce CLR code that can not be produced by VB.net, and vice-versa.

Not true.

Under the .net framework, the CLR is the ultimate arbitor of what can and cannot be done. All languages written for it share the same level of abstraction, no exceptions. Anything you can do in C#, you can do in VB.net, and vice versa.

Re:Very nice, but...

[2short]

Under the x86 achitecture, the x86 instruction set is the ultimate arbitrator of what can and cannot be done. Clearly, anything you can do in C++ you can do in x86, and anything you can do in BASIC you can do in x86. But that doesn't mean anything you can do in C++ can be done in x86. The logic doesn't get any better because you substitute C#, VB.net and CLR. In either case, two higher-level languages both compile into code the same lower-level language. In both cases, the two higher level languages have different subsets of the lower level languages capabilities; the higher level languages have different cpabilities.

I could write a new .net language tommorow called IRock.net It would compile to CLR, but it would only have one instruction (ROCK!) that on compilation becomes the CLR code to print "2Short Rocks!" to standard out. Clearly, IRock.net would not have the same capabilities as C# or VB.net, even though all three compile to CLR. Because C# and VB.net are capable of producing CLR code that IRock.net is not. Just as C# is capable of producing CLR code that VB.net is not and VB.net is presumably capable of producing CLR code that C# is not.

Re:Very nice, but...

[rjshields]

Obviously it's dependent on languages having the same features, which is the case for VB.Net and C#. There may be one or two minor differences but for the most part I'd imagine they compile to identical MSIL.

Re:Very nice, but...

[2short]

Indeed, VB.net and C# have very similar features and capabilities, and if there are big performance differences between them, it's because the authors of one of the compilers screwed up.

But the other posters were arguing that their performance and capabilities should be identical because they both compile to MSIL, and in fact that any language that does so would have equal performance and capabilities. Which is just silly; hence my silly IRock.net example. For a less silly example, Managed C++ certainly has different capabilities than VB.net or C#.

VB.net and C# produce very similar performance, because they are very similar to begin with. Not because their existing compilers target the same virtual machine.

Native code

[Roy van Rijn]

This is a good step to make Python run a bit faster, but I don't think it'll really make a huge difference.

The best way to get some speed and still keep the nice Python functions and layout is just to export the most heavily used functions to native code (C/C++).
I don't know if its possible to take the C++ output and optimize it seperatly, that way you will have a good start to make native code though.

In short: Better, fast and easy, but not the best (if you can write native code)

Re:Native code

[Surt]

I think this sort of tool serves a different purpose. If you have an evolving program, one that needs some speed, but also needs rapid development, then hopefully what this allows is that you do not need to write the heavily used functions in c++, but instead can translate and compile each version of your python program as part of your tool chain. Your strategy makes more sense as your project reaches maturity and stability, whereas this sort of technique is more effective for situations where performance is important, but there is also still rapid development going on.

Re:Native code

[Anonymous Coward]

The best way to get some speed and still keep the nice Python functions and layout is just to export the most heavily used functions to native code (C/C++).

Well, no, the best way is to use a better algorithm.

Very interesting...

[FuzzyDaddy]

This is a very interesting development, both from the practical promise and just 'cause it's cool. However, as a python programmer myself, it's not yet in a usable form. Much of the efficiency of programming in python is the standard libraries (in particular Tkinter for user interfaces), and the non-standard libraries (for example, the serial port library). This project does not yet support these.

Among python programmers, I'm curious - how many use psyco (another python performance enhancement tool) for their projects? I fiddled with it a while ago (it didn't work because of a C module that it didn't like), but never had a compelling reason to go back to it. Performance optimization has never been important enough for my applications to merit the effort.

Re:Very interesting...

[tcopeland]

> However, as a python programmer myself, it's not yet in a usable form

Yup. Along the same lines, Ruby has a related project by Ryan Davis, Ruby2C. It's useful for small localized speedups, but you wouldn't want to try to write your entire app in it.

Re:Very interesting...

[masklinn]

A friend used it once because he was generating Fractals in python and Psycho significantly sped up his script (77% gain with his first algorithm [run time dropped from 197s to 46.52s], 98% with his second algorithm [runtime dropped from 154s to 13.4s])

Re:Very interesting...

[zhiwenchong]

It's all a matter of magnitude.

I use Psyco in my work. My app is a code generator that processes multiple models and transforms them into optimization code. Psyco reduced the time it took for process 1 model from 20 seconds to 2 seconds. It doesn't sound like much, but when you have to do it for lots of models, the speedup suddenly becomes quite substantial.

Re:Very interesting...

[Just Some Guy]

I'm curious - how many use psyco (another python performance enhancement tool) for their projects?

<aol>me too!</aol> Much of my code spends the majority of its time waiting for database queries to finish, ImageMagick to finish doing its stuff, files to copy, etc. Psyco doesn't do a thing for that stuff. On the other hand, a small amount of my code is pretty CPU intensive - not enough to break out of pure Python into Pyrex or anything else, but enough to want a performance upgrade. For those modules, psyco is the clear choice since I can point it at pre-existing code and get a nice boost for free.

Lots of cross-language compiling...

[tcopeland]

...kind of reminds me of the Google Web Toolkit which is more or less a Java to Javascript/HTML compiler. It's not an optimization thing like ShedSkin, instead it lets folks use the Java skills they already have to write better web apps. I wonder what they use to parse the Java code? I don't see any mention of JavaCC on their site, or ANTLR either for that matter...

Underlying technology

[Anonymous Coward]

PyPy

Re:Underlying technology

[ZephyrXero]

"Underlying technology: PyPy"

I could see this Python > C++ > Machine code project being something PyPy was built on top of, but not the other way around? Please explain...

I'm confused...

[advocate_one]

surely the best way to speed it up is to compile it straight to object code... c++ has to be compiled and just adds an intermediate step which will make things harder to debug...

Re:I'm confused...

[Dasher42]

I think that the best example of what you're saying would be the Java compiler in the gcc suite. That separate front-end, back-end approach of gcc is terribly helpful.

And yet, if you're going to compile Python, I'd want the translation into source code. If it's worth rewriting in C++, it's worth tuning, especially if you can improve the usage of type-safe code.

If they can do this...

[Ant P.]

...why not make it into a GCC frontend so Python can be compiled directly?

Re:If they can do this...

[sploxx]

...why not make it into a GCC frontend so Python can be compiled directly?
Probably because it is fairly hard to translate a dynamically typed language into the RTL GCC uses for its backends.

I think they had a lot of problems even 'only' with C++ and all its powerful constructs like templates etc.

Re:If they can do this...

[Overzeetop]

We must not be programmers; I though the same thing. The only advantage is that you could go in and tweak the C++. But if you can read a machine compiled code well enough to make "corrections", wouldn't you just code it that way to begin with?

On second thought, it could be to allow compilation on different platforms. Write once and precompile from Python, then compile for each system you prefer, using the system-specific compiler to optimize for the processor architecture. Of course, I'm just guessing. Hell, the closed I've gotten to Python is a /. recommended intro in a three ring binder on my nightstand. I use it when I have trouble sleeping - it's cheaper than prescription meds.

Re:If they can do this...

[rpwoodbu]

It is worth mentioning that one of the the original implementations of C++ (if not the very first) was "cfront", a C++-to-C converter. I see this as a much easier way to get a new language implemented quickly, as you can take advantage of the common functionalities already implemented in the target language of the converter. Although Python is not a new language, using it as a compiled language is new, and thus I believe it is comparable to being a new language for this argument. C++ and Python have a lot in common, which makes C++ a very suitable target language for a Python-to-[compiled_language] converter.

If this converter proves to be successful, I believe that a GCC frontend will be written eventually. There are probably potential optimizations that would be difficult or impossible to implement any other way.

Some may think that the dynamic nature of Python may preclude its inclusion in GCC. Technically, all that would need to be done is to have a runtime to handle dynamic things, similar to how Objective-C (for which there is GCC support) has a runtime to handle message passing and late binding. However, a large portion of the potential efficiency of a compiled version of the language would be lost to these dynamic capabilities; luckily, a compiler can detect when things are implicitly static (in fact, this converter is limited to implicitly static constructs), and optimise them to be truly static at compile-time.

Re:If they can do this...

[CableModemSniper]

Python -> C++ is easier?

File as NBNC (Nice But No Cigar)

[suitepotato]

Why? Read the linked page? Says it all. Violates most any Python code of any complexity out there. So if it doesn't convert Python code from the real world, what is it for? Making Python coders learn enough about C++ to remember the limitations and write/rewrite Python code to use it?

What the Python C/C++ interested people REALLY need is a book written by a group of Python AND C/C++ masters which teaches the two simultaneously showing complimentary methods of doing any given thing working from beginner to advanced and I DON'T mean "How to turn your n00b Python code into C/C++ hotness" sort of viewpoint. I mean both taught simultaneously in synch showing how they can interchange and compliment.

Software tricks for converting? Ultimately worse than not having them because it leads to horrible obfuscation because we don't know exactly what is going on when 13,412 lines of Python is turned into C++ because WE DIDN'T WRITE IT AND WE NEVER LEARNED C/C++. "Say Mike, that's great but you're the company code cowboy and you don't do C++ natively and I sure as hell don't read it being management so exactly what happens if this needs to be fixed? We've gone from importing open source code you couldn't read to writing our own open source code you can't read."

Re:File as NBNC (Nice But No Cigar)

[hubritc]

The way this will be used by pythonistas is not to convert 13,412 lines of code blindly in to C++. Rather, it provides a pythonic way of getting some speed benefit for those parts of the program that need it and that code will also be accessable to C++ programs as an added benefit.

Re:File as NBNC (Nice But No Cigar)

[Schraegstrichpunkt]

Why? Read the linked page? Says it all. Violates most any Python code of any complexity out there. So if it doesn't convert Python code from the real world, what is it for? Making Python coders learn enough about C++ to remember the limitations and write/rewrite Python code to use it?

And when Linux was still at 1.x, it should have been dismissed by business because it didn't support SMP.

The software is barely written. Have patience.

Re:File as NBNC (Nice But No Cigar)

[try_anything]

Software tricks for converting? Ultimately worse than not having them because it leads to horrible obfuscation because we don't know exactly what is going on when 13,412 lines of Python is turned into C++ because WE DIDN'T WRITE IT AND WE NEVER LEARNED C/C++. "Say Mike, that's great but you're the company code cowboy and you don't do C++ natively and I sure as hell don't read it being management so exactly what happens if this needs to be fixed?"

That isn't how a compiler is used. When you compile a C++ program, you don't throw away your C++ source and check the executable into source control. "Oh, no! We used gcc and now we have a bunch of gobbledygook we don't understand!"

The C++ is an intermediate stage in the make process, akin to the output of various phases of gcc.

Re:File as NBNC (Nice But No Cigar)

[chthon]

Yes, lightyears ahead of code which is talked about, but still not publicly available : Arc.

Re:File as NBNC (Nice But No Cigar)

[Anonymous Coward]

Exactly. Shedskin does not have some very basic stuff like dynamic attributes, exec, etc. You can't even put different values in a list. And thus its benchmarks are flawed: they are all programs that *could be easilly done in C* and their implementation is very static-typed. So saying that it is 60 times faster than python is dishonest. You could easilly do the same thing in pyrex or pyinline and have it 1000 times faster because these benchmarks *do not use any pythonic/dynamic features*.

This ain't Jim Beam^H^H^H^H^HPython.
-- Hercules

C++front?

[ClosedSource]

Somebody had to say it.

Human language analogy

[Anonymous Coward]

Writing a program in C++ and compiling it with a C++ compiler is like writing poetry in German and using babelfish to convert it to English.

Writing a program in Python, then converting it to C++, then compiling with a C++ compiler is like writing poetry in French, then using babelfish to convert it to German, then using babelfish again to convert it to English.

If it's not obvious, then the fewer levels of translation you have, the better your output will be: "The vodka is good, but the meat is terrible."

Re:Human language analogy

[Anonymous Coward]

Reading your post is like trying to translate to English something written in stupid.

Stupid comparison

[ardor]

As another poster already said, file I/O is a bottleneck regardless of ANY language. So, try something different. Real-time h264 decoding for example.

Re:Stupid comparison

[Anonymous Coward]

Agreed. The only possible order of magnitude performance increase I can even think of by writing in C/C++ would possibly be to use mmap rather than stdio/sfio/iostreams or read/write, and I wouldn't be surprised if a few minutes searching didn't turn up something to do that in python.

IO intensive tasks are probably the absolute worst language shoot out benchmark you could choose

Re:Stupid comparison

[Tim Browse]

Or, to put it another way:

"More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity."

C++ Proto-typing

[oblivion95]

Python is by far the best language for proto-typing code intended to be written in C++ later. ShedSkin facilitates this process.

Not enough people proto-type their code, which is why hardly anybody talks about how to do it.

Speaking from experience...

[Anonymous Coward]

Writing a interpreted language to compiled language converter is a fairly simple step that can yield an impressive speedup with very little effort. However, doing so makes you dependent on a separate compiler, and it can also force you to make unwise decisions when you attempt to coerce dynamic language features into a static language. Clearly the "Shed Skin" compiler is in its early stages, and its author is currently favoring the removal of Python features in order to accomplish his goal. That's a shame, because it need not be so. Hopefully his later attempts will support the full range of Python syntax and semantics.

A couple years I wrote a specification for a dynamically typed language with first order functions. Over a period of 3-4 months, I wrote an the interpreter in C++, and then as an experiment I made a bytecode-to-C++ converter. That first prototype was about 3x faster than the already blazing fast interpreter, but I didn't want to have to depend on gcc, so my next step was to write a bytecode-to-x86 converter. That allowed the intepreter to compile down to raw machine code in the same pass as the bytecode verification. With the JIT in place, its performance compared favorably to an unoptimized C compiler, even for tight loops and recursive function call benchmarks like the Ackermann function. Keep in mind that we're talking about dynamically resolved types as loop counters and first class function objects that carry writable closure state from their parent functions, so this is quite a bit more impressive than it might seem at first glance.

At the time, my little unoptimized JIT compiler would have easily made the top 10 on the language shootout page. With the benchmarks normalized to show C, C++ and Ocaml as 1.0, my simple JIT compiler was about a 4-5, considerably faster than my interpreter which ran at about 30-40. Meanwhile, all the other interpreters including perl, python, ruby, lua came in at about 60-200 (*note: my first attempt at the interpreter ran in the 150 range -- about the same as perl).

My experience convinced me that it should be fairly easy (read: only a couple months of work) to speed up most interpeted languages by at least a factor of 10. For some of the dogs out there, you could probably make it closer to a 40x speedup. Granted, it would take a few days to a week to write the machine code driver for each new platform; however, the benefit of not having to run a separate (slow) compilation phase would outweigh that negative, IMHO.

Unfortunately though, some languages constructs are just inherently slow. For example, perl and python both treat their objects as string-based lookup tables, so any code that uses that functionality will be limited by the access speed of the lookup table -- potentially close to O(1) in the average case but no better than O(lgN + length of string) in the worst case. As another example, perl uses a list construction for its function arguments instead of a vector. List allocations require significantly more memory than arrays, and their indexed lookup time is O(N) instead of O(1).

Finally, as the "Shed Skin" compiler author realized (and dutifilly noted that his implementation ignored), there are serious performance penalties that come with allowing container objects to hold dynamic types. These penalties really can't be worked around unless you add optional static typing or simply ignore the feature. I prefer optional static typing, because dynamic typing is way too valuable of a tool to remove from a language.

Re:Speaking from experience...

[oblivion95]

"boo", a .NET language, allows dynamic typing by specifying 'duck' type. It achieves near-c# speed because all other data are statically typed.

It's a great language -- combining the benefits of Python, Ruby, and C# -- and it's wonderful for proto-typing in the .NET world.

Re:Speaking from experience...

[Anonymous Coward]

...at which point you may just as well be targeting any JIT capable VM that has been designed specifically to host dynamic languages.

Re:Speaking from experience...

[jma05]

How many such good VMs do you know? Java and .NET are the dominant VMs now and currently only .NET CLR has builtin support for dynamic typing.

Python as prototyping language

[radtea]

Python is a terrific prototyping language (and lots of other things besides.) As a C++ coder I've been using it for prototyping stuff that will eventually be integrated into a larger application and therefore MUST be translated to C++. So what I'd like to see is a tool (written in Perl, just for the fun of having a linguistic threesome) that just does a light gloss on Python syntax to get me most of the way to human-readable C++. That would be far more useful (to me) than thsi thing, which sounds more like f2c, whose output could case brain damage in humans and cancer in rats, or possibly the other way around.

Re:Python as prototyping language

[oblivion95]

I do what you suggest all the time. The trick is to use Python's "introspection". Since Python knows about itself, it can convert itself to C++ headers easily. (Converting the fucntion bodies is harder.)

% pydoc inspect

Using Perl for this would be nuts!

Buisness Proposition

[JimXugle]

1) Download Bittorrent
2) Download Shed Skin
3) ???
4) Profit!

Ever heard of cfront?

[Ritchie70]

I'm amused that the same mechanism that was originally used to implement C++ (a precompiler that, in that case, generated C code) is now being used with C++ as the "low-level" language with a readily-available compiler.

Surely some of you remember cfront? Generated some truly bloated, completely unreadable stuff, but humans weren't supposed to read its output - cc was.

here's one way to look at it

[kpharmer]

Assume that it takes:
    - 4 hours to write a given program in python, 32 hours to write same program in C++
    - 10 seconds to run the python program, but just 2 seconds to run the faster C++ program
    - the program is run 20 times a day
    - assume the developer time costs as much as the the time of the person that runs it

Ok, so it'll take 630 days of running this program for the faster C++ program to make up for the extra time to develop it. So, if you can wait two years for a payback then C++ is the way to go, otherwise code it in python.

There that was easy. Ok, any other simple problems out there? Which editor you should use? What's just the right amount of comments per program? Which is better - cvs or subversion?

Re:here's one way to look at it

[insignificant1]

I'd mark you funny, if I could, for YASA (yet another simplistic analysis).

GCC -- not suitable for dynamic language?

[DragonWriter]

Which has already been covered elsewhere. The GCC IL isn't suitable for a dynamic language like Python.

There's an Objective-C GCC front-end? Isn't Objective-C dynamic? What's the problem with Python that doesn't exist for Objective-C?

Re:GCC -- not suitable for dynamic language?

[Anonymous Coward]

There's an Objective-C GCC front-end? Isn't Objective-C dynamic? What's the problem with Python that doesn't exist for Objective-C?

According to another post (last paragraph), GCC supports parts of Objective-C with a runtime. Perhaps a similar approach would work for a Python front end.

- T

Interfacing with native code?

[tepples]

If there are no pointers or structs in the Python code, why would they have to handle them?

Interfacing between functions written in Python and functions written in C++, perhaps?

It's almost the opposite

[alienmole]

surely the best way to speed it up is to compile it straight to object code...

Absolutely not. It takes an enormous amount of effort to compete with the native code generation of good C or C++ compilers - and much of that effort has to be repeated for every platform you target.

Many language implementations for less mainstream languages compile through C, treating it as a "portable assembler", and leveraging all the work that's been done to optimize C compilation. This is even done for some high-end languages used e.g. in aerospace - Airbus jets run on a lot of generated C code, for example. C++ is less commonly used for this purpose, but if you know what you're doing, it's no slower than C.

c++ has to be compiled and just adds an intermediate step which will make things harder to debug...

Quite the opposite, having the intermediate step be in a higher level language tends to be very useful for debugging. You don't need to debug your C++ compiler these days, so if there's something that requires you to look at the generated output, you'd usually have to look no further than the C++.

The main disadvantage of using C or C++ as compile targets is that your compiler then depends on some other compiler, and can't work all by itself. Also, for some less standard languages, such as functional languages like Haskell and Scheme, C and C++ aren't such a good fit - but people still work around that and compile them through C in many cases anyway, for the reasons I've mentioned.

That's great.

[warrax_666]

Except I have to convice my boss that Boost (sub-)libraries are (generally) solid. Normally you'd think that would be easy seeing as many Boost (sub-)libraries are getting into TR1 and are being regression tested on many platforms, and are written by some of the most brilliant C++ programmers on the planet. Unfortunaely my boss is a dumbass oldskooler who wouldn't recognize modern C++ if it raped his youngest daughter and who suffers from chronic NIH syndrome, meaning that all we get is shitty homegrown C++ libraries or glibc. With an all-batteries-included language we'd at least be spared the crappy home-grown reimplementations of the most basic stuff like regexes, shared_ptr, etc.

wrong comparison

[m874t232]

This won't be meaningful until a converted python script is compared to efficient code written natively in C++ in the first place.

That's the wrong comparison to make, because it assumes that the C++ programmer has unlimited time to make his C++ code efficient and correct. In real life, programmers have time constraints, and under given time constraints, the Python program will often be faster than the C++ program.

In fact, even without time constraints, C++ code often ends up far less efficient than the optimum possible, simply because using the optimal algorithm or memory management strategy is so hard in C++ that programmers can't do it.

Re:wrong comparison

[stonecypher]

That's the wrong comparison to make, because it assumes that the C++ programmer has unlimited time to make his C++ code efficient and correct.

Well, yes and no. I actually got into this else-thread; there are a hell of a lot of programming jobs where the programmer actually does have time to make their code correct. Not everything behaves like the web; when you're writing a video game, an operating system, a database, embedded or realtime control software, or in fact many many other things, performance just isn't sacrificable.

(And, actually, in most situations, a programmer has all the time in the world to make their software correct; the number of software houses which will willingly release software containing flaws is vanishingly small. Most released flaws are the result of bad development practice and insufficient testing methods, not short schedules.)

In real life, programmers have time constraints, and under given time constraints, the Python program will often be faster than the C++ program.

Yeah. If you'd take a look at the numbers, though, the vast bulk of software is actually embedded software. Embedded software can't tolerate execution delays. Behind embedded software, the next largest group is in-house software; that kind of software generally can't tolerate production delays. Those two groups are a wonderful example of the extreme divergence in needs in development - python is exactly the right thing to do for the second group, but exactly the wrong thing to do for the first.

As far as the Python program often being faster than the C++ program, that sounds an awful lot like an expectation, rather than experience.

In fact, even without time constraints, C++ code often ends up far less efficient than the optimum possible, simply because using the optimal algorithm or memory management strategy is so hard in C++ that programmers can't do it.

Yeah. Now it's time for me to call bullshit. This is such a weirdly interesting myth, that algorithms and memory management in C++ are harder than they are in other languages, that I don't really know what to say.

See, implementing algorithms and memory management in C was really, really hard - you had to, well, keep track of a pointer and an offset. (Because most people confuse the overflows that come from bad engineering practices with something that's just magically too hard to do, presumably because they've never seen anything harder, and they can't imagine a world wherein garbage collection and pointers aren't the alpha and omega of memory management. Wait'll you try COBOL, FORTRAN or machine assembly, all of which are still common languages - in fact, all moreso, if you count all assembly languages as one, than python.)

That said, in C++ it's nowhere near that difficult. If you want garbage collection and can't be bothered to write new in the constructor and delete in the destructor, bust out a smart pointer. Get a container; it'll self allocate just fine, and ridiculously efficiently. Need something more complex, like pooling? No problem - pooling is two lines of code in operator new, or you could just use the policy class in Boost. Algorithms are in fact so fundamental to the design of C++ that there's a specific section on them in the standard, defining how they are to be implemented such that they magically and correctly attach to any correct container. They are trivially easy to implement; a naive bubble sort which is correct can be implemented in a single line of code, in a way that will work for any user defined type implementing operatoreveryone knows it's hard" or fall silent. It actually isn't, and you'll find that a talented C++ programmer can often spank the pants off of a Python programmer, except when dealing with string parsing or GUI stuff (at which point it's time to remember that Delphi, REXX, perl and PHP are a bigger threat than Python.)

I'm not saying Python doesn't have its place. What I am saying is

Re:wrong comparison

[stonecypher]

Oh, hell.

That'll teach me to hit submit without checking the preview. I lost a big and important chunk of the reply after operator< because I forgot to write out the entity for <. Here's a repaste; yay form buffers, boo no edit button for the first five minutes of a post.

-----------------

That's the wrong comparison to make, because it assumes that the C++ programmer has unlimited time to make his C++ code efficient and correct.

Well, yes and no. I actually got into this else-thread; there are a hell of a lot of programming jobs where the programmer actually does have time to make their code correct. Not everything behaves like the web; when you're writing a video game, an operating system, a database, embedded or realtime control software, or in fact many many other things, performance just isn't sacrificable.

(And, actually, in most situations, a programmer has all the time in the world to make their software correct; the number of software houses which will willingly release software containing flaws is vanishingly small. Most released flaws are the result of bad development practice and insufficient testing methods, not short schedules.)

In real life, programmers have time constraints, and under given time constraints, the Python program will often be faster than the C++ program.

Yeah. If you'd take a look at the numbers, though, the vast bulk of software is actually embedded software. Embedded software can't tolerate execution delays. Behind embedded software, the next largest group is in-house software; that kind of software generally can't tolerate production delays. Those two groups are a wonderful example of the extreme divergence in needs in development - python is exactly the right thing to do for the second group, but exactly the wrong thing to do for the first.

As far as the Python program often being faster than the C++ program, that sounds an awful lot like an expectation, rather than experience.

In fact, even without time constraints, C++ code often ends up far less efficient than the optimum possible, simply because using the optimal algorithm or memory management strategy is so hard in C++ that programmers can't do it.

Yeah. Now it's time for me to call bullshit. This is such a weirdly interesting myth, that algorithms and memory management in C++ are harder than they are in other languages, that I don't really know what to say.

See, implementing algorithms and memory management in C was really, really hard - you had to, well, keep track of a pointer and an offset. (Because most people confuse the overflows that come from bad engineering practices with something that's just magically too hard to do, presumably because they've never seen anything harder, and they can't imagine a world wherein garbage collection and pointers aren't the alpha and omega of memory management. Wait'll you try COBOL, FORTRAN or machine assembly, all of which are still common languages - in fact, all moreso, if you count all assembly languages as one, than python.)

That said, in C++ it's nowhere near that difficult. If you want garbage collection and can't be bothered to write new in the constructor and delete in the destructor, bust out a smart pointer. Get a container; it'll self allocate just fine, and ridiculously efficiently. Need something more complex, like pooling? No problem - pooling is two lines of code in operator new, or you could just use the policy class in Boost. Algorithms are in fact so fundamental to the design of C++ that there's a specific section on them in the standard, defining how they are to be implemented such that they magically and correctly attach to any correct container. They are trivially easy to implement; a naive bubble sort which is correct can be implemented in a single line of code, in a way that will work for any user defined type implementing operator< and on any ordered container.

Exactly what memory management

GCC signatures

[CustomDesigned]

There was an experimental feature in GCC called 'signatures' that was like "duck typing". You declared a function to take a signature, which was like a Java interface. You could then pass any type to that function provided the type provided methods of the same name and C prototype. The compiler would construct a simple wrapper class that translated calls to the signature methods to calls to the actuall object. If a type was an "almost" fit, you could provide manual translations of the signature methods as C++ code.

hmmmm...

[sh3l1]

i can't figure out how to get python to do anything but figure pi... :( oh! but that cool calculator feature is so cool!!! thanks to python i know that 2+2/2*6+8-4+23/432 infact equals 12 :)