Slashdot Mirror
Google Boosts Python By Turning It Into Go (infoworld.com)
An anonymous reader quotes InfoWorld:
Grumpy, an experimental project from Google, transpiles Python code into Go, allowing Python programs to be compiled and run as static binaries using the Go toolchain... In a blog post announcing the open source release, Google stated the project stemmed from its efforts to speed up the Python-powered front end for YouTube. But Google hit an obstacle that's familiar to folks who've deployed Python in production: It's hard to get CPython -- the default Python interpreter written in C -- to scale efficiently. "We think Grumpy has the potential to scale more gracefully than CPython for many real world workloads," writes Google...
Because it doesn't support C extensions, Grumpy doesn't have CPython's Global Interpreter Lock, which is commonly cited as a roadblock to running Python concurrent workloads smoothly. Grumpy also uses Go's garbage collection mechanisms to manage memory under the hood, instead of CPython's. Grumpy creates close interoperation between Python and Go by allowing Go packages to be imported and used with the same syntax as Go modules.
Because it doesn't support C extensions, Grumpy doesn't have CPython's Global Interpreter Lock, which is commonly cited as a roadblock to running Python concurrent workloads smoothly. Grumpy also uses Go's garbage collection mechanisms to manage memory under the hood, instead of CPython's. Grumpy creates close interoperation between Python and Go by allowing Go packages to be imported and used with the same syntax as Go modules.
A Python eating its tail and forming a wheel, making it Go easier.
1. parse python into AST.
2. reformat with curly braces.
3. take over the world; there would be no stopping a curly-braces version of python.
Python's GIL. Pythonistas keeping pushing for Python everywhere but don't realize that Python does have its limits and is not the language of choice if you need performance on multiple cores. In my experience, when you try to emulate multithreading in Python using some message passing scheme you end up with something that is more complicated, harder to debug and tune, harder to maintain, than the equivalent written in good C++.
Google didn't invent Python. That was developed by Guido R. at another organization. Years later, Guiodo went to work for Google, well after Python's ascendant popularity.
Android was also developed by another company that Google later bought. Same thing with Youtube. Do you see a pattern yet?
It should use the Java JIT compiler to run much faster than any byte code interpreter.
That said, I do not know why anybody would even think of using a programming language without static typing. Not for performance, but rather for sanity when writing and (more importantly) maintaining code. With type inference it costs virtually no extra typing.
Go offers nothing that an established language can't already do. It'll die just because Google supports it...because Google has no issues with throwing away stuff as soon as they lose interest, no matter how many people are invested in it.
Pretty much every established language can do the stuff the new languages do. It doesn't mean they do them well, or that they can't be improved upon or that we should simply live with the flaws in those languages even if they cause untold numbers of errors in production code. A simple example would be C and C++'s never ending supply of bugs related to dangling pointers, memory leaks etc., many of which are caused by inherent default choices in the language.
While I don't program Go myself it clearly demonstrates a number of advantages for writing multi-threaded, high availability, high performance applications than some other languages. It compiles to a native standalone executable, it's portable, it has garbage collection, it doesn't carry the baggage of a runtime around with it. Performance wise it sits somewhere between C and Java. I've found tools written in Go such as Prometheus to be remarkably stable and reliable.
Unfortunately, the GIL does not only inhibit performance scaling with threads, it causes performance to scale *negatively* on multicore systems. Don't believe me? Try to run a threaded code on a single core system, and then on multicore systems (`cpusets` come in handy for this test). That holds for I/O bound threads just as well as for CPU bound ones or mixtures thereof. Singlecore systems are hard to find nowadays, and cpusets are not always available (in terms of permission and kernel support), and it impacts process based scaling obviously. Negative scaling is hard to accept (for my use cases at least).
Further, the GIL causes some strangeness (to put it mildly) with respect to signals and interrupt handling -- simply put, signals and interrupts can get lost in multithreaded python codes, which makes mix of threading and processing *very* cumbersome and unstable.
*Only* using multiprocessing is hard, too, for two reasons: not having threads makes async programming difficult (think GUIs). It makes I/O or event driven applications much harder. And, multiprocessing has its own quirks: ever tried to spawn a process in a daemon (hits an `assert`)? Ever tried to pass around a process handle to a central process manager (`wait()` will cease to work)? And some other funny things. Python does not make any serious attempt to provide POSIX level process handling, in my opinion, and that shows for non-trivial use cases.
In summary: people like me whining about the GIL have more reasons than performance. But, performance is one, too. Threading in Python is in a sorry state, and that is a shame. There is no solution in sight.
Python is a neat language. I would not ever (again) recommend it for production code.
PS.: I should add that, in this context, the GO runtime is interesting, as it puts away several of those limitations. So I do understand the rationale behind this I guess. As others above, I don't really see the advantages of simply using GO directly, or any other mature language. My guess is that there is too much Python code around to simply abandon or re-implement it.
Real men code in Interpreted COBOL running on top of a Javascript framework.
Even though a bunch of new languages coming out. Thought I want to make a new programming language. Next thought should be “No.” This is how really bad programming languages get out there in the world. Javascript is a good example. Javascript was designed and implemented by this guy who went to the University of Indiana. Made a scheme interpreter. Working at Netscape. Need some language to script webpages in web browser. I know I'll just do it with Scheme. Curly braces lot faster than parentheses. Notation that C programmers will like and call it Java-something for marketing purposes. In a week, came up with first Javascript implementation. No going back. On every computer and phone. Can't make any changes, so many JS programs in the world. Arguably the most widely used programming language.
Well, and that's what threads are: something like multiple UNIX processes sharing all their memory via shared memory and a few other niceties. Unfortunately, Python doesn't support that kind of fast, consistent IPC. What it supports is either a limited form of shared memory, or slow IPC that involves copying.
So, you can look at the Python runtime either as lacking thread support or at lacking good IPC support, but it is lacking something that many other runtimes have.
There have already been several excellent implementations of Python that compile to native code, either via JIT or directly, including IronPython. But Python's performance problems aren't in Python code, since anything compute intensive is done in native libraries. The real problem with all of those implementations is that, like Grumpy, they don't have native, thread-safe libraries, foremost Numpy and Scipy.
Grumpy sounds like an attempt to migrate Python users to Go. That's probably the right idea, because Python seems to be going nowhere fast. Go, however, is probably not the right language to migrate to, despite its moderate adoption in some systems applications, mostly because Go just gives the finger to existing programming practice for no other reason than FYTW.
it has garbage collection, it doesn't carry the baggage of a runtime around with it
Sigh. Of course it has a runtime. Where else would the garbage collection that you just mentioned be implemented? Or GoRoutines. Or reflection.
I think you're confusing not having a runtime with having the Go compiler statically link the runtime into each executable. That has some benefits that you were alluding to (e.g. "no baggage") but it also has drawbacks such as increased executable size, increased memory usage (with a dynamic runtime, different instances all share the same library in memory), decreased cache usage (since if you have two Go executables, they are constantly evicting each others runtimes from cache, even though they are identical and could be shared) and the maintenance issues having to recompile to take advantage of security/bug/performance improvements in the runtime itself.
I have no issue if you claim that in some (your) use cases the advantages of a statically compiled runtime are worth the disadvantages. But that's not the same as claiming that either the runtime doesn't exist or that it's always advantageous.