Van Rossum: Python Not Too Slow

snydeq writes "Python creator Guido van Rossum discusses the prospects and criticisms of Python, noting that critics of Python performance should supplement with C/C++ rather than re-engineering Python apps into a faster language. 'At some point, you end up with one little piece of your system, as a whole, where you end up spending all your time. If you write that just as a sort of simple-minded Python loop, at some point you will see that that is the bottleneck in your system. It is usually much more effective to take that one piece and replace that one function or module with a little bit of code you wrote in C or C++ rather than rewriting your entire system in a faster language, because for most of what you're doing, the speed of the language is irrelevant.'"

Comments

Re:It's still too slow, despite what he says.

[steveha]

Says the guy whose whole life is tied up in the language

That's fair.

and whose project, at Google, to speed it up, crashed and burned.

That's completely wrong. Unladen Swallow was not GvR's project, and it didn't "crash and burn". Unladen Swallow found that their approach was not speeding up Python as much as they had hoped, and the two Google employees were moved on to other projects. The code lives on, and I think people are still doing things with it, although it's clear that PyPy is a better approach.

To me at least, "crash and burn" implies a horrible failure with catastrophic consequences, and Unladen Swallow was considerably less dramatic than that. If you just meant to say "they didn't accomplish their goals", that would be a fair statement.

Python is slow because von Rossum refuses to cut loose the boat-anchor of "anything can change anything at any time".

That was a basic decision way back when, and it would be a profound change to make (the language wouldn't really be Python anymore). I personally don't make much use of this, but supposedly there are some programs that do.

PyPy, the newer Python implementation, uses two interpreters and a JIT compiler to try to handle the dynamism with less overhead. They're making progress, but they need a very complex implementation to do it, and they're many years behind schedule.

There is a very small group of people working on PyPy, and they are doing ambitious things. I'm not overly worried about their schedule.

You failed to mention that PyPy has already achieved great speed when compared to CPython. The latest PyPy is, on average, over 5x as fast as CPython.

http://speed.pypy.org/

if a module or class could only be modified from outside itself if it contained explicit self-modification code (like a relevant "setattr" call) most modules and classes could be nailed down as fixed, "slotted" objects at compile time.

This is an interesting idea. But I am not aware of anyone working on something like this.

Claiming that the "slow parts" should be rewritten in C is a cop-out.

I disagree. Way back at the dawn of time, GvR designed Python to make it easy to interface C code, just exactly as this sort of "escape hatch" to help projects that work well but are too slow, and also for libraries.

I think that the ability to link in C code was an important feature that helped Python win hearts and minds. It's slower than C, but at least you knew there is an escape hatch if you wind up having trouble with it.

Except for number-crunching, or glue code for existing libraries, it's seldom done.

Yeah, but that's kind of like saying that except for stop signs and traffic lights, you usually don't use the brake pedal in a car.

There are all sorts of useful libraries that have been glued into Python, and a major part of Python's popularity is that you can use powerful libraries from a convenient and friendly language.

SciPy is a great example: they took ferociously powerful libraries (written mostly in Fortran) and made them usable from Python. I know I for one can get more work done in Python than in Fortran.

As another example, a few years ago I worked on a project to make a DVD player with some fancy features, and we were doing our development in Python. It was fast enough, even on a cheap embedded processor, because all the heavy lifting was being done by C library code. And we got a lot of work done quickly.

(I have a Python program running right now which will run for over a week, parsing the street address of every business in the US into a standard format. The parser is complex enough that rewriting it in C would be a big job. There's no "inner loop".)

This sounds interesting.

Can you run this in PyPy?

Can you fan it out to multiple processes using the multiprocessing

Want a good example?

[Sycraft-fu]

Civilization 4. Firaxis wanted to make it nice n' moddable. So they scripted a bunch of it out in Python. Makes it real easy for users to edit in the field. However the AI they couldn't do that with. They wanted the users to be able to edit that too, but it was too slow in Python. So they did that in C++ and made it a DLL, and then distributed the sources. Harder to work on than a Python script, but fast enough that the game didn't bog down. The core of the game engine was C++ too and the Python was integrated with Boost.Python.

Works really, really well. Again, the right tool for the job. If they'd tried to do the whole game in Python, it would have been a disaster performance wise (and I'm not even sure if it would be possible, if Python can call on DirectX in the way C++ can). However a mixture worked brilliantly.