Deep Learning Is Eating Software (petewarden.com)

Pete Warden, engineer and CTO of Jetpac, shares his view on how deep learning is already starting to change some of the programming is done. From a blog post, shared by a reader last week: The pattern is that there's an existing software project doing data processing using explicit programming logic, and the team charged with maintaining it find they can replace it with a deep-learning-based solution. I can only point to examples within Alphabet that we've made public, like upgrading search ranking, data center energy usage, language translation, and solving Go, but these aren't rare exceptions internally. What I see is that almost any data processing system with non-trivial logic can be improved significantly by applying modern machine learning. This might sound less than dramatic when put in those terms, but it's a radical change in how we build software. Instead of writing and maintaining intricate, layered tangles of logic, the developer has to become a teacher, a curator of training data and an analyst of results. This is very, very different than the programming I was taught in school, but what gets me most excited is that it should be far more accessible than traditional coding, once the tooling catches up. The essence of the process is providing a lot of examples of inputs, and what you expect for the outputs. This doesn't require the same technical skills as traditional programming, but it does need a deep knowledge of the problem domain. That means motivated users of the software will be able to play much more of a direct role in building it than has ever been possible. In essence, the users are writing their own user stories and feeding them into the machinery to build what they want.

Comments

Nice Advertisement

[prefec2]

Most software today and in the coming decades is designed and developed to support business processes or data flow and execution in scientific processes. These systems need a deterministic and foreseeable behavior. Yes, you may use "learning" classification mechanisms such as neural networks to support some tasks, but this is not changing how we develop software. Especially, developing software is usual a technical and social process, as you have to understand the demands and needs of users, which require interviews and discussions with users. You also need to communicate with UI designers to develop together with users and UI designers useful and easily to understand interfaces. And yes, you have to map all this onto technology.

Re:Accessible

[jellomizer]

Except for the fact the deep learning systems, may not be efficient enough. There are some tasks which it excels at, however some tasks there just isn't the volume or rewards for outcomes for the system to adapt fast enough to.

Lets say a flood control dam system. Which is is strictly coded if water goes above this line, open the dam and flood the down, because otherwise the damage is far greater than flooding the town. The reason why this is computer controlled is because it takes the faulty morals out of the picture. You wouldn't want to be the one to press the button and flood the town, even though it is for the greater good. A Deep Learning system, will want to take more variables into account, and will either decide that it is important to press the button before it hits the limit, as it would only create minor flooding problems, or too late figuring that it can wait a little longer creating a catastrophic failure.

Deep Learning often creates a lot of superstitious like behaviors (As shown in the funny walk of Googles Deep learning to walk video) Where it learned to do the task, but not at optimal way. This may just be a mater of time for the technology to get better, however it isn't a time to panic and toss out your CS Degree and get a job at your local Walmart as a greeter.
 

linear regression

[dmitrygr]

And by "deep learning" in most cases they mean "linear regression on cleaned-up data"

Re:linear regression

[serviscope_minor]

No, not really, and I wouldn't call this insightful.

Deep learning is not especially well defined, but it's not linear regression. I've seen several competing/complementary definitions.

1. A neural net (much) greater than 3 layers deep. A sufficiently wide 3 layer net has enough capacity to run any function, so a lot of ANN learning focussed on these in the past. Turns out having lots more layers makes training much more tractable (particularly with stochastic gradient descent an batchnorm).

2. Convolutional nets (where you're basically learning convolutional kernels, which saves a huge number of parameters compared to a normal net especially if you have many layers) with many layers.

3. Something which learns the low-level features in the same optimization as everything else. Traditional ML algorithms were often structured as a feature extraction stage which takes the data and extracts some features in a human designed manner. Then applies ML, but if the ML can't optimize the right loss, you go for the closest loss you can, then get the thing you want with post-processing.

A nice example would be the Viola-Jones face detector. The features are a bunch of zero mean box filter combinations applied to an image, combined with a threshold, each one giving a different binarisation of the image. Those were hand designed. a modified Adaboost[*] is then learned to to get a good selection and weighting of the features to give a pixelwise classification of the image. You want a bounding box, so the final stage is to extract a bounding box from the binarization.

The "problems" with that are that there's nothing to say those features are optimal, and the pixelwise loss is the wrong thing to optimize. A deep system takes pixels in and spits out a bounding box (or several). The point is you can compute derivatives with respect to all the stages so you optimize everything agains the loss you're actually interested in.

[*] They actually use a cacade (degenerate decision tree) with a biased adaboost classifier at each node. Either way it's an ML algorithm with largely the same properties.