Slashdot Mirror
Google Open-Sources GPipe, a Library For Training Large Deep Neural Networks (venturebeat.com)
An anonymous reader quotes a report from VentureBeat: Google's AI research division today open-sourced GPipe, a library for "efficiently" training deep neural networks (layered functions modeled after neurons) under Lingvo, a TensorFlow framework for sequence modeling. It's applicable to any network consisting of multiple sequential layers, Google AI software engineer Yanping Huang said in a blog post, and allows researchers to "easily" scale performance. As Huang and colleagues explain in an accompanying paper ("GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism"), GPipe implements two nifty AI training techniques. One is synchronous stochastic gradient descent, an optimization algorithm used to update a given AI model's parameters, and the other is pipeline parallelism, a task execution system in which one step's output is streamed as input to the next step.
Most of GPipe's performance gains come from better memory allocation for AI models. On second-generation Google Cloud tensor processing units (TPUs), each of which contains eight processor cores and 64 GB memory (8 GB per core), GPipe reduced intermediate memory usage from 6.26 GB to 3.46GB, enabling 318 million parameters on a single accelerator core. Without GPipe, Huang says, a single core can only train up to 82 million model parameters. That's not GPipe's only advantage. It partitions models across different accelerators and automatically splits miniature batches (i.e., "mini-batches") of training examples into smaller "micro-batches," and it pipelines execution across the micro-batches. This enables cores to operate in parallel, and furthermore accumulate gradients across the micro-batches, thereby preventing the partitions from affecting model quality.
Most of GPipe's performance gains come from better memory allocation for AI models. On second-generation Google Cloud tensor processing units (TPUs), each of which contains eight processor cores and 64 GB memory (8 GB per core), GPipe reduced intermediate memory usage from 6.26 GB to 3.46GB, enabling 318 million parameters on a single accelerator core. Without GPipe, Huang says, a single core can only train up to 82 million model parameters. That's not GPipe's only advantage. It partitions models across different accelerators and automatically splits miniature batches (i.e., "mini-batches") of training examples into smaller "micro-batches," and it pipelines execution across the micro-batches. This enables cores to operate in parallel, and furthermore accumulate gradients across the micro-batches, thereby preventing the partitions from affecting model quality.
"Now they want us to switch to expensive, inconsistent, polluting GPipe!" -Confusedative
Ty Cobb -- no, not that Ty Cobb -- isn't a household name outside of Washington legal circles.
But Cobb, who spent almost a year from July 2017 to May 2018 as the White House lawyer leading the response to special counsel Robert Mueller's probe, may have, in retrospect, made one of the most consequential decisions in the presidency of Donald Trump: To cooperate fully with Mueller's investigation.
Cobb's thinking, according to reporting over the past two years, was that by fully cooperating with Mueller's investigation, Trump and his broader White House would ensure that the probe was short-lived.
"I'd be embarrassed if this is still haunting the White House by Thanksgiving and worse if it's still haunting him by year end," Cobb told Reuters in August 2017(!), adding: "I think the relevant areas of inquiry by the special counsel are narrow."
To that end, Cobb urged Trump not to publicly attack Mueller -- the former director of the FBI -- or the special counsel probe more broadly. And most importantly, Cobb advised Trump to allow administration officials to broadly cooperate with asks from the special counsel.
"I was the one that advised it," Cobb acknowledged in an interview this week with ABC News of the open-book approach to Mueller. "But the President did make the decision."
One critical example of why Cobb's advice -- and Trump's initial decision to listen to it -- mattered so much: White House counsel Donald McGahn spent more than 30 hours in conversations with the special counsel's office in which he shared "detailed accounts about the episodes at the heart of the inquiry into whether President Trump obstructed justice, including some that investigators would not have learned of otherwise," according to The New York Times, which broke the McGahn story back in August 2018.
That same story included these critical lines:
"Mr. McGahn's cooperation began in part as a result of a decision by Mr. Trump's first team of criminal lawyers to collaborate fully with Mr. Mueller. The president's lawyers have explained that they believed their client had nothing to hide and that they could bring the investigation to an end quickly."
Obviously, Cobb's theory of the case didn't come to pass.
Mueller's probe, which began in May 2017, is still ongoing -- although recent reports suggest it is nearing an end. Trump's frustration with Cobb's go-along-to-get-along-approach led to the lawyer's departure from the White House in May 2018 -- and to the installation of a much more aggressive legal team led by former New York City Mayor Rudy Giuliani, who has spent the past 10 months savaging Mueller and the investigation more generally.
As CNN reported at the time:
"A source familiar with Cobb's departure said the former federal prosecutor, who joined Trump's legal team in July 2017, had been clashing with the President in recent weeks over Trump's combative posture with the special counsel's investigation. Trump has intensified his public attacks on Robert Mueller's probe in recent weeks, and on Wednesday, suggested that questions by Mueller's team about whether he obstructed justice amount to a 'setup & trap.'"
By the time Cobb left the White House, however, the die was cast. The White House had spent the better part of a year cooperating fully with Mueller's probe. Critically, that cooperation came in the first year of Mueller's investigation -- when his team, presumably, were in an information-gathering and dot-connecting mode. They didn't know what they didn't know. And through Cobb's policy of full cooperation, Mueller's team was able to not only gather scads of information but also begin to put the pieces in place for a much broader investigation that now deals not only with Russian's interference in the 2016 election but also potential obstruction of justice by the President.
Without that initial suggestion by Cobb -- and Trump's decision to acquiesce to it -- it's uniquely possible we could be talking about a different, and maybe significantly narrower, special c
Hard pass.
python3 -m pip search gpipe
gpipe (1.0.0) - GridPipe
torch-gpipe (0.0.0a0) -
https://pypi.org/project/gpipe/
https://github.com/east301/gpipe
Nope!
Here it is: https://github.com/tensorflow/lingvo/blob/master/lingvo/core/gpipe.py
I envision Alphabet will sue for the gpipe name.
Makes you wonder what good will come out of this, and even more what bad things will follow.
This software might be very useful for developing computer systems to solve difficult tasks. Sure, whatever... but we can't let that blind us to what's really important here.
What's truly important here is dissecting exactly what to call these systems. Using the term "neural network" makes sense on pretty much every level, and it would allow us to communicate clearly about what type of algorithms are being used, but that term (and especially "deep neural network") make all my personal bugaboos about AI flare up. These are computers not brains, so there's no neurons, so we can't use that word.
And yeah, obviously I recognize that, for most people, Google's efforts here are exactly what people mean when they talk about AI - but that makes me angry so nobody should do it. I've decided, for no good reason, that the term AI should only be used when describing an intelligence that works just the same as a human. I have weird quasi-spiritual hangups about all this, which I think you're all obliged to respect.
PS: Also, just a reminder, Google has accomplished nothing, these systems are useless and aren't improving. All just unimpressive hype. I hate technology and change, please stop. Thanks in advance for never mentioning something like this again.
Let's not stir that bag of worms...
laying some gpipe tonight for sure!!
enabling 318 million parameters on a single accelerator core. Without GPipe, Huang says, a single core can only train up to 82 million model parameters
Is that what's really going on in the human brain -- hundreds of millions of "model parameters" are getting trained up?
I doubt it. With this approach, AI researchers are following a road to something completely different from human intelligence. And I'll bet that something will be far more limited than human intelligence.
That that is is that that that that is not is not.