Slashdot Mirror
Google Open-Sources GPipe, a Library For Training Large Deep Neural Networks (venturebeat.com)
An anonymous reader quotes a report from VentureBeat: Google's AI research division today open-sourced GPipe, a library for "efficiently" training deep neural networks (layered functions modeled after neurons) under Lingvo, a TensorFlow framework for sequence modeling. It's applicable to any network consisting of multiple sequential layers, Google AI software engineer Yanping Huang said in a blog post, and allows researchers to "easily" scale performance. As Huang and colleagues explain in an accompanying paper ("GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism"), GPipe implements two nifty AI training techniques. One is synchronous stochastic gradient descent, an optimization algorithm used to update a given AI model's parameters, and the other is pipeline parallelism, a task execution system in which one step's output is streamed as input to the next step.
Most of GPipe's performance gains come from better memory allocation for AI models. On second-generation Google Cloud tensor processing units (TPUs), each of which contains eight processor cores and 64 GB memory (8 GB per core), GPipe reduced intermediate memory usage from 6.26 GB to 3.46GB, enabling 318 million parameters on a single accelerator core. Without GPipe, Huang says, a single core can only train up to 82 million model parameters. That's not GPipe's only advantage. It partitions models across different accelerators and automatically splits miniature batches (i.e., "mini-batches") of training examples into smaller "micro-batches," and it pipelines execution across the micro-batches. This enables cores to operate in parallel, and furthermore accumulate gradients across the micro-batches, thereby preventing the partitions from affecting model quality.
Most of GPipe's performance gains come from better memory allocation for AI models. On second-generation Google Cloud tensor processing units (TPUs), each of which contains eight processor cores and 64 GB memory (8 GB per core), GPipe reduced intermediate memory usage from 6.26 GB to 3.46GB, enabling 318 million parameters on a single accelerator core. Without GPipe, Huang says, a single core can only train up to 82 million model parameters. That's not GPipe's only advantage. It partitions models across different accelerators and automatically splits miniature batches (i.e., "mini-batches") of training examples into smaller "micro-batches," and it pipelines execution across the micro-batches. This enables cores to operate in parallel, and furthermore accumulate gradients across the micro-batches, thereby preventing the partitions from affecting model quality.
"Now they want us to switch to expensive, inconsistent, polluting GPipe!" -Confusedative
Hard pass.
python3 -m pip search gpipe
gpipe (1.0.0) - GridPipe
torch-gpipe (0.0.0a0) -
https://pypi.org/project/gpipe/
https://github.com/east301/gpipe
Nope!
Here it is: https://github.com/tensorflow/lingvo/blob/master/lingvo/core/gpipe.py
I envision Alphabet will sue for the gpipe name.
Makes you wonder what good will come out of this, and even more what bad things will follow.
This software might be very useful for developing computer systems to solve difficult tasks. Sure, whatever... but we can't let that blind us to what's really important here.
What's truly important here is dissecting exactly what to call these systems. Using the term "neural network" makes sense on pretty much every level, and it would allow us to communicate clearly about what type of algorithms are being used, but that term (and especially "deep neural network") make all my personal bugaboos about AI flare up. These are computers not brains, so there's no neurons, so we can't use that word.
And yeah, obviously I recognize that, for most people, Google's efforts here are exactly what people mean when they talk about AI - but that makes me angry so nobody should do it. I've decided, for no good reason, that the term AI should only be used when describing an intelligence that works just the same as a human. I have weird quasi-spiritual hangups about all this, which I think you're all obliged to respect.
PS: Also, just a reminder, Google has accomplished nothing, these systems are useless and aren't improving. All just unimpressive hype. I hate technology and change, please stop. Thanks in advance for never mentioning something like this again.
Let's not stir that bag of worms...
laying some gpipe tonight for sure!!
enabling 318 million parameters on a single accelerator core. Without GPipe, Huang says, a single core can only train up to 82 million model parameters
Is that what's really going on in the human brain -- hundreds of millions of "model parameters" are getting trained up?
I doubt it. With this approach, AI researchers are following a road to something completely different from human intelligence. And I'll bet that something will be far more limited than human intelligence.
That that is is that that that that is not is not.