Slashdot Mirror
Google Boosts Python By Turning It Into Go (infoworld.com)
An anonymous reader quotes InfoWorld:
Grumpy, an experimental project from Google, transpiles Python code into Go, allowing Python programs to be compiled and run as static binaries using the Go toolchain... In a blog post announcing the open source release, Google stated the project stemmed from its efforts to speed up the Python-powered front end for YouTube. But Google hit an obstacle that's familiar to folks who've deployed Python in production: It's hard to get CPython -- the default Python interpreter written in C -- to scale efficiently. "We think Grumpy has the potential to scale more gracefully than CPython for many real world workloads," writes Google...
Because it doesn't support C extensions, Grumpy doesn't have CPython's Global Interpreter Lock, which is commonly cited as a roadblock to running Python concurrent workloads smoothly. Grumpy also uses Go's garbage collection mechanisms to manage memory under the hood, instead of CPython's. Grumpy creates close interoperation between Python and Go by allowing Go packages to be imported and used with the same syntax as Go modules.
Because it doesn't support C extensions, Grumpy doesn't have CPython's Global Interpreter Lock, which is commonly cited as a roadblock to running Python concurrent workloads smoothly. Grumpy also uses Go's garbage collection mechanisms to manage memory under the hood, instead of CPython's. Grumpy creates close interoperation between Python and Go by allowing Go packages to be imported and used with the same syntax as Go modules.
Is Mozilla doing something similar, but using Rust instead of Go?
Just rewrite the code to Go. Or, even better, don't use Go because Go is a trash fad language that'll be dead in five years.
A Python eating its tail and forming a wheel, making it Go easier.
What are they talking about? Stop inventing new shit all the fucking time, and just fucking stick to what fucking works and is stable. You never let anything settle. You constantly crave change for the sake of change. Fuck off. Slow the fuck down. My Windows 10 hasn't even updated to Anniversary Edition yet, and there are already multiple major updates since. It's insanity. Stop changing everything. Go back to the UI of Windows 95 and start over from there. Stop making new programming languages. Stop doing all this stupid shit.
Four post and no mention of religion, trump, racism, or apers aping apes. Am I in the wrong place.
You be the judge.
1. parse python into AST.
2. reformat with curly braces.
3. take over the world; there would be no stopping a curly-braces version of python.
Python's GIL. Pythonistas keeping pushing for Python everywhere but don't realize that Python does have its limits and is not the language of choice if you need performance on multiple cores. In my experience, when you try to emulate multithreading in Python using some message passing scheme you end up with something that is more complicated, harder to debug and tune, harder to maintain, than the equivalent written in good C++.
Google didn't invent Python. That was developed by Guido R. at another organization. Years later, Guiodo went to work for Google, well after Python's ascendant popularity.
Android was also developed by another company that Google later bought. Same thing with Youtube. Do you see a pattern yet?
So it can only support pure Python code? That pretty much rules it out for many applications where speed is an issue, where C extensions are common (and not just for performance reasons). In which case, might as well use PyPy: all the advantages of Python, the JIT can give native performance, and you can use all the existing high performance libraries written in C (or Fortran!). Sure, I grant you that Grumpy does have the advantage of making a self-contained executable, but there are ways to do that with PyPy.
Is this going to go the way of Google's Unladen Swallow fork of CPython?
oH whoops, cant use a facebook framework/jsx.
hahaha
why cant google use its own technologies, such as Polymer/webcomponents
But youtube interface sucks , thumbnails have no tooltips, dont show time stamps. When a paused video in full screen goes to window mode, it starts play back again. Also they show ads that are longer than the videos sometimes, (39min ad for 4min video)
Where is the SBS mode 3D interface for 3dTVs ? I can play back SBS 3d videos, but the interface web page it self is not in 3d mode, or even viewable in 3d mode.
Liberty freedom are no1, not dicks in suits.
YouTube downloader services make you "please wait" for conversion because every YouTube downloader uses youtube-dl on the backend and youtube-dl is a giant bloated python script.
Well COBOL has both funky syntax rules and scalability. Nearly there guys!
I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
It should use the Java JIT compiler to run much faster than any byte code interpreter.
That said, I do not know why anybody would even think of using a programming language without static typing. Not for performance, but rather for sanity when writing and (more importantly) maintaining code. With type inference it costs virtually no extra typing.
Even if this was true (and there are some problems with using multiple processes), nothing fixes that processes are utter crap on Windows, anything involving them in slow as molasses on it, so unfortunately process-based parallelism is close to a no-go on Windows. That may well be the one huge reason why we got stuck with threads as basically the only "official" way of doing it.
Unfortunately, the GIL does not only inhibit performance scaling with threads, it causes performance to scale *negatively* on multicore systems. Don't believe me? Try to run a threaded code on a single core system, and then on multicore systems (`cpusets` come in handy for this test). That holds for I/O bound threads just as well as for CPU bound ones or mixtures thereof. Singlecore systems are hard to find nowadays, and cpusets are not always available (in terms of permission and kernel support), and it impacts process based scaling obviously. Negative scaling is hard to accept (for my use cases at least).
Further, the GIL causes some strangeness (to put it mildly) with respect to signals and interrupt handling -- simply put, signals and interrupts can get lost in multithreaded python codes, which makes mix of threading and processing *very* cumbersome and unstable.
*Only* using multiprocessing is hard, too, for two reasons: not having threads makes async programming difficult (think GUIs). It makes I/O or event driven applications much harder. And, multiprocessing has its own quirks: ever tried to spawn a process in a daemon (hits an `assert`)? Ever tried to pass around a process handle to a central process manager (`wait()` will cease to work)? And some other funny things. Python does not make any serious attempt to provide POSIX level process handling, in my opinion, and that shows for non-trivial use cases.
In summary: people like me whining about the GIL have more reasons than performance. But, performance is one, too. Threading in Python is in a sorry state, and that is a shame. There is no solution in sight.
Python is a neat language. I would not ever (again) recommend it for production code.
PS.: I should add that, in this context, the GO runtime is interesting, as it puts away several of those limitations. So I do understand the rationale behind this I guess. As others above, I don't really see the advantages of simply using GO directly, or any other mature language. My guess is that there is too much Python code around to simply abandon or re-implement it.
Well, and that's what threads are: something like multiple UNIX processes sharing all their memory via shared memory and a few other niceties. Unfortunately, Python doesn't support that kind of fast, consistent IPC. What it supports is either a limited form of shared memory, or slow IPC that involves copying.
So, you can look at the Python runtime either as lacking thread support or at lacking good IPC support, but it is lacking something that many other runtimes have.
There have already been several excellent implementations of Python that compile to native code, either via JIT or directly, including IronPython. But Python's performance problems aren't in Python code, since anything compute intensive is done in native libraries. The real problem with all of those implementations is that, like Grumpy, they don't have native, thread-safe libraries, foremost Numpy and Scipy.
Grumpy sounds like an attempt to migrate Python users to Go. That's probably the right idea, because Python seems to be going nowhere fast. Go, however, is probably not the right language to migrate to, despite its moderate adoption in some systems applications, mostly because Go just gives the finger to existing programming practice for no other reason than FYTW.
Can we fucking STOP using this abomination? A compiler ALREADY translates between two languages!!!!!! what the fuck is new about "transpiling" STOP IT RETARDS
Nitpicking, buy it was JPython in 1997
It has been Jython since 1999
https://wiki.python.org/jython...
> I do not know why anybody would even think of using a programming language without static typing.
- Dynamically typed programming languages are more productive when writing smaller quantities of rapidly evolving code.
- It is mainly a library and an ecosystem issue. Python tends to have all the modules I need, while Haskell, OCaml and Scala often don't... and they often seem to be much easier to pick up and use.
For example, Pandas equivalents are much less mature in other ecosystems. On language merits alone, I should be using Scala more than Python, but in practice, Python modules win me over.
I wanted to memoise a function. To look up a module and put in the couple of lines (an import and a decorator) needed to achieve that probably took a couple of minutes in Python, and I was back to the real meat of my code. I would have spent much longer in Java.
I like that these languages all bring new ideas and are a great learning opportunity; erlang and go have interesting programming models and CSP/actor functionality. But I feel I should be able to enjoy these things in other languages after they have a while to get absorbed, that's not generally the case though, for example I've been reading about using akka in java for a while now and the syntax looks clumsy to my eye compared with erlang and I've totally failed to get my head around scala.
same thing with improvements in the build/deploy chain. If transpiling to golang is significantly better then I'll take that win, and simply use go as a stepping stone to deploy the python syntax that I like.
I'm fumbling to say that I think that once several years have passed and a new programming idea has become accepted and proven its worth, them most of the major language syntaxes should grow support for it. There should be a simple CSP syntax for Python, it should be possible to compile erlang for the JVM, to reasonably implement actors in java etc..
Then I won't have to endure so many eye rolls when I suggest erlang for that highly concurrent system to the diehard python or java devs or to the managers that want nothing to do with anything too shiney
Nullius in verba
Both 2.7 and 3.6 documentation clearly state that you can't have multiple statements in a lambda function definition:
https://docs.python.org/2/refe...
https://docs.python.org/3/refe...
In fact the function body part of the lambda is not a statement at all, but a single expression that becomes the function's return value:
https://docs.python.org/3/refe...
supporting shared memory is a facility which comes from the operating system. And it doesn't involve copying (although it does involve cache invalidation, but that's true for both processes and threads). No gains in performance come from reading/writing to memory shared between threads bound to different cores vs system shared memory accessed by processes running on different cores.
Unfortunately, the GIL does not only inhibit performance scaling with threads, it causes performance to scale *negatively* on multicore systems. Don't believe me? Try to run a threaded code on a single core system, and then on multicore systems (`cpusets` come in handy for this test).
A python process is always using one core at a time (not always the same core because it can block and then get scheduled to run on a different core). If you find a way to bind your Python process to a single core, all your Python threads will be IO bound threads and GIL will not hinder performance. People are just bitching about having to think through the manager/workers pipeline, but any design which wants to take advantage of multiple cores should do that anyway. Multiple cores are best thought of as multiple single-core machines joined by a really, really fast network. And then run multiple Python processes (each bound to a single core). Then each one of them will have its own GIL and when you do want to share data between cores, you'd use system's shared memory. This works both in Windows and in Linux. As long as your process does jump from core to core, there is no cache invalidation, so GIL will not hinder you (because it will work as if it was running on a single-core machine).
Any guest worker system is indistinguishable from indentured servitude.
*Only* using multiprocessing is hard, too, for two reasons: not having threads makes async programming difficult (think GUIs). It makes I/O or event driven applications much harder.
No one is talking about using only multiprocessing. Threads should always be for IO blocking while processes should be for CPU bound contexts. This is done to avoid the expensive context switch when scheduling waiting on IO while going off to do other work. And each process should be bound (as much as possible) to one core.
Any guest worker system is indistinguishable from indentured servitude.
Re Python:
I would settle for the ability to extend the built-in classes, str in particular. My "settle" went like this:
1) Inquired politely about same
2) Python nerds have orgasm telling me why this is terrible. I am, to put it mildly, dubious.
3) I write 100% compatible pre-processor that gives me the syntax I wanted.
4) PROFIT. Okay, well, not really, but EXTENDED STRING CLASS METHOD SYNTAX!
Like...
myString = 'foo'
otherString = myString.doHorribleThing('bar')
print 'good'.grief()
So...
You could do the same. What you want, perhaps, might be much easier than what I did. In fact, you could fork my project and add what you want to it. I'm already parsing the language reasonably well, which is arguably one of the difficult parts.
You don't always have to wait for a language's maintainers to get off their butts to address shortcomings or instantiate new goodies. Or eventually not do anything at all. There are other paths to nerdvana.
I've fallen off your lawn, and I can't get up.
As long as your process does jump from core to core, there is no cache invalidation, so GIL will not hinder you (because it will work as if it was running on a single-core machine).
should have been
" As long as your process does not jump from core to core, there is no cache invalidation, so GIL will not hinder you (because it will work as if it was running on a single-core machine)."
But, as always, you cannot edit a Slashdot comment after it's published.
Any guest worker system is indistinguishable from indentured servitude.
Yes, the OS supports shared memory; it's Python's support that is poor.
That's bullshit. Numerical and AI applications benefit greatly from sharing memory that way, compared to other IPC mechanisms.
Instead of translating to Go, why not directly translate to C, or even to language machine?
How is this different from what Cython has been doing for years? There's nothing new about the idea of translating Python to a statically compiled language to improve performance.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."