Deep Learning Is Eating Software (petewarden.com)

Pete Warden, engineer and CTO of Jetpac, shares his view on how deep learning is already starting to change some of the programming is done. From a blog post, shared by a reader last week: The pattern is that there's an existing software project doing data processing using explicit programming logic, and the team charged with maintaining it find they can replace it with a deep-learning-based solution. I can only point to examples within Alphabet that we've made public, like upgrading search ranking, data center energy usage, language translation, and solving Go, but these aren't rare exceptions internally. What I see is that almost any data processing system with non-trivial logic can be improved significantly by applying modern machine learning. This might sound less than dramatic when put in those terms, but it's a radical change in how we build software. Instead of writing and maintaining intricate, layered tangles of logic, the developer has to become a teacher, a curator of training data and an analyst of results. This is very, very different than the programming I was taught in school, but what gets me most excited is that it should be far more accessible than traditional coding, once the tooling catches up. The essence of the process is providing a lot of examples of inputs, and what you expect for the outputs. This doesn't require the same technical skills as traditional programming, but it does need a deep knowledge of the problem domain. That means motivated users of the software will be able to play much more of a direct role in building it than has ever been possible. In essence, the users are writing their own user stories and feeding them into the machinery to build what they want.

Comments

Neural networks are fragile

[bradley13]

Once upon a time, I did my doctorate in machine learning. The machines were less powerful, but the algorithms? Basically the same as they are today. Sorry, the stuff most widely in use is still the same back-propagating neural networks. The machines are just faster, so the networks can be bigger. That's it.

Neural networks can work really well on specific problem domains. The problem is: You have no idea what they are actually learning. The features that a network identifies within its layers are not really accessible to us. The problem lies, imho, in the total lack of domain knowledge. Since the network doesn't understand what the objects in those pictures are, they are doing a purely mechanical analysis of some (and who knows which) aspects of the pictures. They can learn some really weird things.

In a well-trained network, the results mostly coincide with our expectations. In a completely isolated domain, like chess or go, a network can be trained sufficiently to perform quite well. However, in open domains, they are fragile: we have no idea when they will break. Look at the video of the turtle being identified as a rifle (in the link above). Why does the identification jump seemingly at random? When will a cat will suddenly be guacamole? When will a pedestrian crossing the road will suddenly be just a pile of leaves? We have no idea, none.

It is certainly true that selecting and managing training data is a very different task from classic programming. However, it doesn't really take much domain knowledge. In most domains, gathering training data is tedious, not difficult. The hard part comes in figuring out how to make the best use of that data to train and test a network - and that requires a deep understanding of how the neural networks work (and how they don't work). Plus, frankly, a huge pile of trial and error, because there aren't many rules on how to best structure a net for any particular task.