Slashdot Mirror
Google Boosts Python By Turning It Into Go (infoworld.com)
An anonymous reader quotes InfoWorld:
Grumpy, an experimental project from Google, transpiles Python code into Go, allowing Python programs to be compiled and run as static binaries using the Go toolchain... In a blog post announcing the open source release, Google stated the project stemmed from its efforts to speed up the Python-powered front end for YouTube. But Google hit an obstacle that's familiar to folks who've deployed Python in production: It's hard to get CPython -- the default Python interpreter written in C -- to scale efficiently. "We think Grumpy has the potential to scale more gracefully than CPython for many real world workloads," writes Google...
Because it doesn't support C extensions, Grumpy doesn't have CPython's Global Interpreter Lock, which is commonly cited as a roadblock to running Python concurrent workloads smoothly. Grumpy also uses Go's garbage collection mechanisms to manage memory under the hood, instead of CPython's. Grumpy creates close interoperation between Python and Go by allowing Go packages to be imported and used with the same syntax as Go modules.
Because it doesn't support C extensions, Grumpy doesn't have CPython's Global Interpreter Lock, which is commonly cited as a roadblock to running Python concurrent workloads smoothly. Grumpy also uses Go's garbage collection mechanisms to manage memory under the hood, instead of CPython's. Grumpy creates close interoperation between Python and Go by allowing Go packages to be imported and used with the same syntax as Go modules.
A Python eating its tail and forming a wheel, making it Go easier.
What are they talking about? Stop inventing new shit all the fucking time, and just fucking stick to what fucking works and is stable. You never let anything settle. You constantly crave change for the sake of change. Fuck off. Slow the fuck down. My Windows 10 hasn't even updated to Anniversary Edition yet, and there are already multiple major updates since. It's insanity. Stop changing everything. Go back to the UI of Windows 95 and start over from there. Stop making new programming languages. Stop doing all this stupid shit.
1. parse python into AST.
2. reformat with curly braces.
3. take over the world; there would be no stopping a curly-braces version of python.
When Google abandons 8.8.8.8 we can switch to 80.80.80.80.
Python's GIL. Pythonistas keeping pushing for Python everywhere but don't realize that Python does have its limits and is not the language of choice if you need performance on multiple cores. In my experience, when you try to emulate multithreading in Python using some message passing scheme you end up with something that is more complicated, harder to debug and tune, harder to maintain, than the equivalent written in good C++.
Google didn't invent Python. That was developed by Guido R. at another organization. Years later, Guiodo went to work for Google, well after Python's ascendant popularity.
Android was also developed by another company that Google later bought. Same thing with Youtube. Do you see a pattern yet?
oH whoops, cant use a facebook framework/jsx.
hahaha
why cant google use its own technologies, such as Polymer/webcomponents
But youtube interface sucks , thumbnails have no tooltips, dont show time stamps. When a paused video in full screen goes to window mode, it starts play back again. Also they show ads that are longer than the videos sometimes, (39min ad for 4min video)
Where is the SBS mode 3D interface for 3dTVs ? I can play back SBS 3d videos, but the interface web page it self is not in 3d mode, or even viewable in 3d mode.
Liberty freedom are no1, not dicks in suits.
Well COBOL has both funky syntax rules and scalability. Nearly there guys!
I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
It should use the Java JIT compiler to run much faster than any byte code interpreter.
That said, I do not know why anybody would even think of using a programming language without static typing. Not for performance, but rather for sanity when writing and (more importantly) maintaining code. With type inference it costs virtually no extra typing.
It's really just to transition a few large in house Google services to Go because Python is slow as fuck and that's never going to change. Most new Google projects just use C++, Java, Go or Dart from the start now. Python is mostly being phased out and this is a tool to help with that.
Of the popular languages used with real purpose perl python go java c# ruby php which ones used language research crap you are talking about.
Language research is for academic post doc useless stuff. Real languages that are developed by smart ppl solving real problem embraced by ppl whom buy the proposition
I don't count c and c++ cause they are pioneers
Or is job security with a shitty dead language your plan?
hell no, my current job is programing cgi in perl on apache, using cvs for versioning so i'm all set, thanks.
Did you write the above post in some kind of programming language? Because it certainly does not seem to be English.
Go offers nothing that an established language can't already do. It'll die just because Google supports it...because Google has no issues with throwing away stuff as soon as they lose interest, no matter how many people are invested in it.
Pretty much every established language can do the stuff the new languages do. It doesn't mean they do them well, or that they can't be improved upon or that we should simply live with the flaws in those languages even if they cause untold numbers of errors in production code. A simple example would be C and C++'s never ending supply of bugs related to dangling pointers, memory leaks etc., many of which are caused by inherent default choices in the language.
While I don't program Go myself it clearly demonstrates a number of advantages for writing multi-threaded, high availability, high performance applications than some other languages. It compiles to a native standalone executable, it's portable, it has garbage collection, it doesn't carry the baggage of a runtime around with it. Performance wise it sits somewhere between C and Java. I've found tools written in Go such as Prometheus to be remarkably stable and reliable.
this isn't the special reading group, when you get 3 stars in the special group, the teacher will let you post here.
Unfortunately, the GIL does not only inhibit performance scaling with threads, it causes performance to scale *negatively* on multicore systems. Don't believe me? Try to run a threaded code on a single core system, and then on multicore systems (`cpusets` come in handy for this test). That holds for I/O bound threads just as well as for CPU bound ones or mixtures thereof. Singlecore systems are hard to find nowadays, and cpusets are not always available (in terms of permission and kernel support), and it impacts process based scaling obviously. Negative scaling is hard to accept (for my use cases at least).
Further, the GIL causes some strangeness (to put it mildly) with respect to signals and interrupt handling -- simply put, signals and interrupts can get lost in multithreaded python codes, which makes mix of threading and processing *very* cumbersome and unstable.
*Only* using multiprocessing is hard, too, for two reasons: not having threads makes async programming difficult (think GUIs). It makes I/O or event driven applications much harder. And, multiprocessing has its own quirks: ever tried to spawn a process in a daemon (hits an `assert`)? Ever tried to pass around a process handle to a central process manager (`wait()` will cease to work)? And some other funny things. Python does not make any serious attempt to provide POSIX level process handling, in my opinion, and that shows for non-trivial use cases.
In summary: people like me whining about the GIL have more reasons than performance. But, performance is one, too. Threading in Python is in a sorry state, and that is a shame. There is no solution in sight.
Python is a neat language. I would not ever (again) recommend it for production code.
PS.: I should add that, in this context, the GO runtime is interesting, as it puts away several of those limitations. So I do understand the rationale behind this I guess. As others above, I don't really see the advantages of simply using GO directly, or any other mature language. My guess is that there is too much Python code around to simply abandon or re-implement it.
Even though a bunch of new languages coming out. Thought I want to make a new programming language. Next thought should be “No.” This is how really bad programming languages get out there in the world. Javascript is a good example. Javascript was designed and implemented by this guy who went to the University of Indiana. Made a scheme interpreter. Working at Netscape. Need some language to script webpages in web browser. I know I'll just do it with Scheme. Curly braces lot faster than parentheses. Notation that C programmers will like and call it Java-something for marketing purposes. In a week, came up with first Javascript implementation. No going back. On every computer and phone. Can't make any changes, so many JS programs in the world. Arguably the most widely used programming language.
Well, and that's what threads are: something like multiple UNIX processes sharing all their memory via shared memory and a few other niceties. Unfortunately, Python doesn't support that kind of fast, consistent IPC. What it supports is either a limited form of shared memory, or slow IPC that involves copying.
So, you can look at the Python runtime either as lacking thread support or at lacking good IPC support, but it is lacking something that many other runtimes have.
There have already been several excellent implementations of Python that compile to native code, either via JIT or directly, including IronPython. But Python's performance problems aren't in Python code, since anything compute intensive is done in native libraries. The real problem with all of those implementations is that, like Grumpy, they don't have native, thread-safe libraries, foremost Numpy and Scipy.
Grumpy sounds like an attempt to migrate Python users to Go. That's probably the right idea, because Python seems to be going nowhere fast. Go, however, is probably not the right language to migrate to, despite its moderate adoption in some systems applications, mostly because Go just gives the finger to existing programming practice for no other reason than FYTW.
Nitpicking, buy it was JPython in 1997
It has been Jython since 1999
https://wiki.python.org/jython...
> I do not know why anybody would even think of using a programming language without static typing.
- Dynamically typed programming languages are more productive when writing smaller quantities of rapidly evolving code.
- It is mainly a library and an ecosystem issue. Python tends to have all the modules I need, while Haskell, OCaml and Scala often don't... and they often seem to be much easier to pick up and use.
For example, Pandas equivalents are much less mature in other ecosystems. On language merits alone, I should be using Scala more than Python, but in practice, Python modules win me over.
I wanted to memoise a function. To look up a module and put in the couple of lines (an import and a decorator) needed to achieve that probably took a couple of minutes in Python, and I was back to the real meat of my code. I would have spent much longer in Java.
it has garbage collection, it doesn't carry the baggage of a runtime around with it
Sigh. Of course it has a runtime. Where else would the garbage collection that you just mentioned be implemented? Or GoRoutines. Or reflection.
I think you're confusing not having a runtime with having the Go compiler statically link the runtime into each executable. That has some benefits that you were alluding to (e.g. "no baggage") but it also has drawbacks such as increased executable size, increased memory usage (with a dynamic runtime, different instances all share the same library in memory), decreased cache usage (since if you have two Go executables, they are constantly evicting each others runtimes from cache, even though they are identical and could be shared) and the maintenance issues having to recompile to take advantage of security/bug/performance improvements in the runtime itself.
I have no issue if you claim that in some (your) use cases the advantages of a statically compiled runtime are worth the disadvantages. But that's not the same as claiming that either the runtime doesn't exist or that it's always advantageous.
Oblig: https://xkcd.com/224/
You have the right to remain sentient. If you give up the right to remain sentient, you will be elected to public office
I like that these languages all bring new ideas and are a great learning opportunity; erlang and go have interesting programming models and CSP/actor functionality. But I feel I should be able to enjoy these things in other languages after they have a while to get absorbed, that's not generally the case though, for example I've been reading about using akka in java for a while now and the syntax looks clumsy to my eye compared with erlang and I've totally failed to get my head around scala.
same thing with improvements in the build/deploy chain. If transpiling to golang is significantly better then I'll take that win, and simply use go as a stepping stone to deploy the python syntax that I like.
I'm fumbling to say that I think that once several years have passed and a new programming idea has become accepted and proven its worth, them most of the major language syntaxes should grow support for it. There should be a simple CSP syntax for Python, it should be possible to compile erlang for the JVM, to reasonably implement actors in java etc..
Then I won't have to endure so many eye rolls when I suggest erlang for that highly concurrent system to the diehard python or java devs or to the managers that want nothing to do with anything too shiney
Nullius in verba
Both 2.7 and 3.6 documentation clearly state that you can't have multiple statements in a lambda function definition:
https://docs.python.org/2/refe...
https://docs.python.org/3/refe...
In fact the function body part of the lambda is not a statement at all, but a single expression that becomes the function's return value:
https://docs.python.org/3/refe...
Every language with a standard library has a runtime library. Are you a moron? I'll give you one guess what the RT in MSVCRT means.
Unfortunately, the GIL does not only inhibit performance scaling with threads, it causes performance to scale *negatively* on multicore systems. Don't believe me? Try to run a threaded code on a single core system, and then on multicore systems (`cpusets` come in handy for this test).
A python process is always using one core at a time (not always the same core because it can block and then get scheduled to run on a different core). If you find a way to bind your Python process to a single core, all your Python threads will be IO bound threads and GIL will not hinder performance. People are just bitching about having to think through the manager/workers pipeline, but any design which wants to take advantage of multiple cores should do that anyway. Multiple cores are best thought of as multiple single-core machines joined by a really, really fast network. And then run multiple Python processes (each bound to a single core). Then each one of them will have its own GIL and when you do want to share data between cores, you'd use system's shared memory. This works both in Windows and in Linux. As long as your process does jump from core to core, there is no cache invalidation, so GIL will not hinder you (because it will work as if it was running on a single-core machine).
Any guest worker system is indistinguishable from indentured servitude.
*Only* using multiprocessing is hard, too, for two reasons: not having threads makes async programming difficult (think GUIs). It makes I/O or event driven applications much harder.
No one is talking about using only multiprocessing. Threads should always be for IO blocking while processes should be for CPU bound contexts. This is done to avoid the expensive context switch when scheduling waiting on IO while going off to do other work. And each process should be bound (as much as possible) to one core.
Any guest worker system is indistinguishable from indentured servitude.
Re Python:
I would settle for the ability to extend the built-in classes, str in particular. My "settle" went like this:
1) Inquired politely about same
2) Python nerds have orgasm telling me why this is terrible. I am, to put it mildly, dubious.
3) I write 100% compatible pre-processor that gives me the syntax I wanted.
4) PROFIT. Okay, well, not really, but EXTENDED STRING CLASS METHOD SYNTAX!
Like...
myString = 'foo'
otherString = myString.doHorribleThing('bar')
print 'good'.grief()
So...
You could do the same. What you want, perhaps, might be much easier than what I did. In fact, you could fork my project and add what you want to it. I'm already parsing the language reasonably well, which is arguably one of the difficult parts.
You don't always have to wait for a language's maintainers to get off their butts to address shortcomings or instantiate new goodies. Or eventually not do anything at all. There are other paths to nerdvana.
I've fallen off your lawn, and I can't get up.
As long as your process does jump from core to core, there is no cache invalidation, so GIL will not hinder you (because it will work as if it was running on a single-core machine).
should have been
" As long as your process does not jump from core to core, there is no cache invalidation, so GIL will not hinder you (because it will work as if it was running on a single-core machine)."
But, as always, you cannot edit a Slashdot comment after it's published.
Any guest worker system is indistinguishable from indentured servitude.
Yes, the OS supports shared memory; it's Python's support that is poor.
That's bullshit. Numerical and AI applications benefit greatly from sharing memory that way, compared to other IPC mechanisms.
Instead of translating to Go, why not directly translate to C, or even to language machine?
How is this different from what Cython has been doing for years? There's nothing new about the idea of translating Python to a statically compiled language to improve performance.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
Yeah, I'd have to agree. But kids like to pretend they made something new.
I really have to vehemently disagree with you contention of:
A simple example would be C and C++'s never ending supply of bugs related to dangling pointers, memory leaks etc., many of which are caused by inherent default choices in the language.
If I were to judge your age based on your user ID number, I would day you have been around long enough to know the old saying, "I malloc, therefor I free"
C and to a lesser extent, C++ are quite excellent tools, and for that matter most all other tools are written in those two languages by competent software engineers who actually know what they are doing.
Don,t blame the tool, place the incompetent tool user.
Hey KID! Yeah you, get the fuck off my lawn!
Sharing a framework is fine on servers running *n?x, not so fine on desktop or smartphones.
On Windows, the most widespread desktop and laptop computer operating system in the Western industrialized world, there exists no system-wide automated way to install and update frameworks separately from an application. So when you try to install an application whose framework comes separately, it won't run because the framework isn't already there.
And on both macOS and iOS, Apple requires applications sold in the App Store to be "self-contained". A "self-contained" app either is statically linked or has a local copy of any frameworks that aren't shipped with the OS. I imagine that Apple instituted this policy to avoid the "DLL hell" problem on Windows, where an update to a framework would break applications that inadvertently depended on implementation-defined, unspecified, or undefined behaviors of said framework.
Besides, address space layout randomization (ASLR) is likely to diminish any RAM-sharing benefits of a shared framework.
Then what's a better word for "compile" that specifically connotes the target language being higher level than assembly language?
(Here, I define "assembly language" as a human-readable programming language with a nearly one-to-one correspondence with a CPU's native machine language or a JIT-compiled bytecode.)
What's a better word for "execute" that specifically connotes the execution taking place on little-endian powerPC architecture?
Lillpowcecute! WHEEEE!! New words are FUN!
But nobody needs it.
Google compiled some code from Python into Go. Anyone in the industry should know what you're talking about. Anyone that tries to act like a know it all and claim that compiling only refers to the trip to binary deserves to be corrected.