Slashdot Mirror

← Back to Stories (view on slashdot.org)

Deep Learning Pioneer On the Next Generation of Hardware For Neural Networks

Posted by samzenpus on from the listen-to-the-expert dept.

An anonymous reader writes: While many recognize Yann LeCun as the father of convolutional neural networks, the momentum of which has ignited artificial intelligence at companies like Google, Facebook and beyond, LeCun has not been strictly rooted in algorithms. Like others who have developed completely new approaches to computing, he has an extensive background in hardware, specifically chip design and this recognition of specialization of hardware, movement of data around complex problems, and ultimately core performance, has proven handy. He talks in depth this week about why FPGAs are coming onto the scene as companies like Google and Facebook seek a move away from "proprietary hardware" and look to "programmable devices" to do things like, oh, say, pick out a single face of one's choosing from an 800,000 strong population in under five seconds.

45 comments

  1. The problem with neural networks by Viol8 · · Score: 2

    Is that *in theory* you could understand why they come to a particular result, but in practice it could be potentially very hard with a large network for any person to get their head around the processes leading up to the output. This means that unless safety rules are changed we won't be seeing these things driving cars or flying aircraft anytime soon since the software needs to be verifiable and neural networks are not.

    1. Re:The problem with neural networks by Sneeka2 · · Score: 2

      Or, arguably, we need to change our definition of "verifiable"... For complex activities such as driving cars, we're reaching the limits of traditionally programmed computers. A human programmer cannot possibly think of every possible situation a car might encounter on the street and pre-program an appropriate response into the car. Neural networks and "artificial intelligence" doesn't have a pre-programmed response, but could come up with one based on patterns it knows. So it becomes more about giving the machine a robust basis to work on and then training it... just like a human gathers experiences and then applies them to new situations.

      --
      Bitten Apples are still better than dirty Windows...
    2. Re:The problem with neural networks by ziggystarsky · · Score: 4, Insightful

      Fortunately we can understand the processes within real people that lead to their actions. This is the reason that we safely let them drive cars, trains or fly planes.

    3. Re:The problem with neural networks by ShanghaiBill · · Score: 1

      ... since the software needs to be verifiable ...

      The software does NOT have to be "verifiable". It just has to be thoroughly tested, and in practice, shown to be better than humans. It doesn't have to be perfect, it just has to be an improvement.

      Only trivial programs can be mathematically verified. Even for mission critical programs that make life and death decisions, very few can be proven correct. And even then, are you sure you trust the proof?

      There are techniques for making ANNs more reliable. One technique is "boosting": Independently train two neural networks to solve the same problem, then train a third network on the sample cases where the first two disagree. Then use 2-out-of-3 voting. This can reduce the error rate by a factor of 10.

    4. Re:The problem with neural networks by Viol8 · · Score: 1

      But then one day the neural net has a "senior moment" and drives the car off a cliff. And no one can figure out why. At least with a program you'll eventually figure out where the failure is. But I take your point about pre-programmed responses and you're right. I'm not really sure what the solution is - maybe use a neural network but have a normal program acting as a watchdog?

    5. Re:The problem with neural networks by Viol8 · · Score: 1

      "Fortunately we can understand the processes within real people that lead to their actions. "

      Since when? Psychiatrics have been claiming that for years but I see little evidence for it beyond simple actions. Sometimes even the person themselves doesn't understand why they do something if it was subconsious.

    6. Re:The problem with neural networks by fuzzyfuzzyfungus · · Score: 1

      I certainly wouldn't want to be the one leading the charge to get this approved; but we currently let neural networks drive cars after a relatively pitiful 'black box' verification where we subject them to maybe 30 minutes, 45 at most, of approximately real-world stimuli and then evaluate their responses.

      This arrangement does end up with ~30,000 fatalities a year; but seems to enjoy broad support.

    7. Re:The problem with neural networks by Anonymous Coward · · Score: 1

      A human programmer cannot possibly think of every possible situation a car might encounter on the street and pre-program an appropriate response into the car. ...
      it becomes more about giving the machine a robust basis to work on

      A serious question: is ensuring that a training data set is "robust" (i.e. all possible relevant scenarios are somehow in it) that much easier for humans to do than "thinking of every possible situation"?

      Perhaps it is, but that is not obvious to me. It seems like the two tasks are both very difficult.

    8. Re:The problem with neural networks by Viol8 · · Score: 1

      And your alternative would be what? Not have allowed anyone to drive in the last 100 years?

    9. Re:The problem with neural networks by Anonymous Coward · · Score: 1

      Whoosh!

    10. Re:The problem with neural networks by MrL0G1C · · Score: 1

      A human programmer cannot possibly think of every possible situation a car might encounter on the street and pre-program an appropriate response into the car.

      And they don't have too, all they have to do is make sure cars are substantially better than humans at not driving into things. What to drive into and what not to drive into in the event of an unavoidable accident will be determined by a simple scoring system that determines each possible route and picks the one with the best score. The scoring system doesn't have to be perfect or even great because autonomous cars will very rarely ever drive into things by accident. When you look at the reasons humans drive into things and rule out the reasons that won't apply to autonomous cars, you won't be left with much - crashes due to mechanical failure are about 10-13% of the total.

      --
      Waterfox - a Firefox fork with legacy extension support, security updates and better privacy by default.
    11. Re:The problem with neural networks by Anonymous Coward · · Score: 0

      > The software does NOT have to be "verifiable". It just has to be thoroughly tested, and in practice, shown to be better than humans.

      Wow. So much handwaving that you inadvertently swatted a couple of flies in the process.

      - The software does NOT have to be "verifiable"
          Says who? You? Good luck in getting the rest of society onto your Disneyland boat.

      - It just has to be thoroughly tested
          For some value of "thoroughly".

      - and in practice, shown to be better than humans.
          How much practice? How many humans?

      Remember: we have a finely balance system of "liabilities" around a driver misbehaving (be it intentionally or not, with clear distinctions on those criteria). This system is already stressed out when difficult-to-tackle technological aspects enter the equation (Toyota acceleration anyone?).

      What do you do when (not if, *when*) such a deep network "seemingly intentionally" runs over a pedestrian? Rehab?

      (captcha was "hospital". Ewww...)

    12. Re:The problem with neural networks by fuzzyfuzzyfungus · · Score: 2

      Well, I think that the standards for driving tests could use some modification; but I was actually aiming at exactly the opposite point: There isn't any particular reason to believe that we need to, or will, demand that machines that control vehicles be submitted to some sort of profound understanding and formal verification, given that we accept black-box testing(and pretty shoddy testing at that) for human operators.

      The initial ,lobbying might be a fairly ghastly pain; but I see no reason why there would be any long-term resistance to complex systems(neural network or otherwise) that are effectively beyond human understanding; so long as they pass black box tests of their abilities. I say this both because that's what we do when dealing with people; and because, in practice, even today's tech is complex enough that effectively nobody outside of very specialized software dev and test outfits knows what the hell is going on; and people basically accept that, because the alternative involves being restricted to radically simpler technology or radically more expensive tech support.

    13. Re:The problem with neural networks by Sneeka2 · · Score: 3, Informative

      Sure, it's all extremely difficult. I'd think with neural networks you can use an evolutionary approach and eventually choose the program which has evolved and performed best over a series of X million of tests. The question "when is the program done" doesn't mean "when has the programmer thought of every last possibility" anymore, but rather "when are we satisfied enough with the statistics that we trust this program enough?"

      --
      Bitten Apples are still better than dirty Windows...
    14. Re:The problem with neural networks by orasio · · Score: 2

      You make a very interesting point.
      With automation, it's a lot easier for us to accept a given amount of understandable failure, than a much smaller amount of inexplicable failure. That might be a roadblock against some forms of automation.

      In any case, there's also economics, which do like statistics, and will make you choose the strategy that fails less, overall. For example, insurance companies might favour driving algorithms that crash less often vs ones that crash a bit more often, but for better known causes.

    15. Re:The problem with neural networks by bigpat · · Score: 1

      Is that *in theory* you could understand why they come to a particular result, but in practice it could be potentially very hard with a large network for any person to get their head around the processes leading up to the output. This means that unless safety rules are changed we won't be seeing these things driving cars or flying aircraft anytime soon since the software needs to be verifiable and neural networks are not.

      I would agree that neural networks shouldn't be in a learning mode while they should be in a fixed operational mode, but once they are trained and the neural network is no longer being modified to fit the training set then a neural net is like any other algorithm and will output predictable results.

    16. Re:The problem with neural networks by Anonymous Coward · · Score: 0

      *in theory* you could understand why they come to a particular result, but in practice it could be potentially very hard with a large network for any person to get their head around the processes leading up to the output.

      Equally, *in theory* you could train a second neural net to explain the first one in a way that humans could get their head around.

    17. Re:The problem with neural networks by meta-monkey · · Score: 1

      Well, how hard can it be?

      --
      We don't have a state-run media we have a media-run state.
    18. Re:The problem with neural networks by orasio · · Score: 1

      That's just not true.
      Humans, specially urban dwellers, are known to have a certain set of capabilities, in general.
      Also, they are known to behave in a certain fashion, and to abide by certain rules.
      For example, a human with tendency to kill everyone in his path, would just not be able to apply for a drivers license, he would be in jail, dead, or something similar.
      That black box testing is only verifying very specific knowledge and ability. It doesn't do a great job at that, but its task is a lot easier than testing an AI from scratch. You could do that, if there was some "human like" validation test, you could take prior to getting a license.

    19. Re:The problem with neural networks by ShanghaiBill · · Score: 1

      The software does NOT have to be "verifiable"

      Says who?

      Says everyone who uses current software, which is everyone. Almost none of it has been formally verified. Why should NNs be held to a different standard?

    20. Re:The problem with neural networks by kaiser423 · · Score: 1

      It's not impossible. More complex, but not impossible. You take the input from that caused that (car's black box), see which parent and child neurons fired indicating the undesired action and then check those neurons against the training set to see what aspects of the training set conditioned that behavior (aka, made those neurons respond to that image). You know now what caused that. If you want, you can then modify the network. Or you can take the inputs that caused the incorrect response, make some small changes to it to so that it is representative of other similar situations to come up with a 1,000 item set, and then add it to your training data as a negative set (aka what not to do). Then re-verify after training that those neurons no longer fire in that situation and cause it to drive off of a cliff. Actually, it's almost more intuitive than finding some threading issue, race condition, buffer overflow or multitude of other small, nasty bugs that you have to deal with on a normal basis. It just takes a lot longer to run the tools to see what the issue is and retrain the network.

    21. Re:The problem with neural networks by Anonymous Coward · · Score: 0

      This is the assumption ,let me make this clear to you one that is not true, that the government will use to convince you neural nets are reliable and need for figuring out a person has committed a pre crime or not.

    22. Re:The problem with neural networks by epine · · Score: 1

      But then one day the neural net has a "senior moment" and drives the car off a cliff.

      It's actually your geek pride that just plunged to astounding depths.

      Computers don't beat humans at chess by playing human chess better than humans. They beat humans by having a deeper view of the combinations and permutations and by making very few mistakes.

      A momentary "senior moment" in a self-driving car (I wish I could have rendered that in priapismic scare quotes, but Slashdot defeats me) would just as likely be followed by a Mario Andretti moment 100 ms later as it recomputes several of the box-within-box outer safety profiles ab initio with fresh camera and sensor data. It's so unlike a senior moment as to make my jaw drop (unless you count those senior moments in Quake 3 where you could momentarily see through a solid wall if your POV landed on just the right surface boundary).

      You had the whole time you were writing that paragraph to reverse out a bad rhetorical gambit, and never bothered.

      What's next in the self-driving car? Liver spots? Bladder failure?

    23. Re:The problem with neural networks by willworkforbeer · · Score: 1

      A human programmer cannot possibly think of every possible situation a car might encounter on the street and pre-program an appropriate response .

      I watched a talk by a Google self-driving car engineer; the funniest moment (and an example of your point about pre-programming) was the video showing the time when a Google car came across a woman driving a motorized wheelchair around chasing ducks in the middle of the street.

      --
      Pretending this is my office full of bitter coworkers..
    24. Re:The problem with neural networks by braindrainbahrain · · Score: 1

      According to this article neural networks do make mistakes, sometimes very big mistakes, like the ANN that confused a school bus with a football jersey. The root cause is not easy to determine, and hence the fix is not easy. 'Twould appear that the reasons neural networks fail is a hot topic in neural network research.

    25. Re:The problem with neural networks by Anonymous Coward · · Score: 0

      Why should the software be verifiable? Electronics are ubiquitous, but they aren't verifiable. And of course humans aren't verifiable at all. Hardly any practical software is 100% verifiable, and software can definitely not be verified if it's not fully open source.
      I think the requisite should be that any new technology needs to be safe enough (at least as safe as a good human replacement), which can be tested empirically, and there are indications that for limited tasks neural networks driving a car are already safer than most human drivers. And liability needs to be regulated.

    26. Re:The problem with neural networks by Anonymous Coward · · Score: 0

      You can understand the processes within real people? Wow, you seem to have solved one of the greatest mysteries of science. Please enlighten us.

    27. Re:The problem with neural networks by Big_Breaker · · Score: 1

      You wouldn't typically use a single neural network from input (LIDAR, video, gyros, accelerometers, etc) to output (steering and pedals)

      More typically you'd use different neural networks to tackle steps in the chain. One might identify the borders of the road and the median. Another might pick out cars. Another might project the position of the cars in the future. Each would be easier to test individually. You might also have a "supervisor" that looked for disagreement or inconsistencies between the systems and notify the driver to take over.

    28. Re:The problem with neural networks by ExekielS · · Score: 1

      So make it pass a drivers test? Maybe a couple dozen? Send it cross country a few times to prove it can handle an enormous variety of situations? Sounds like what they are already aiming to do.

      --
      ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn
    29. Re:The problem with neural networks by orasio · · Score: 1

      No.
      That's what the GP proposed.

      For a human, a skill test is OK, because we already know he's a human, and cities are built around humans. We can expect him to behave in a certain way, and we kind of know his possible range of abilities and limitations, even if not in a formal way.

      There's a reason why we require other things, like a minimum age, because being a responsible adult is precondition to the test.

      What they are doing right now is different. Still a black box test, but much more comprehensive. They are going to gather so much data that statistics are going to start kicking in, bc they should be able to somewhat "prove" that these cars are safer than regular cars.

    30. Re:The problem with neural networks by Anonymous Coward · · Score: 0

      you are allowed to drive a car (I hope).
      is your software verifiable?

      once you hit a certain quality of results, you can consider a problem solved for all practical purposes, even if the results are not correct in 100% of the cases.

    31. Re:The problem with neural networks by ziggystarsky · · Score: 1

      You need to have your sarcasm detector checked.

    32. Re:The problem with neural networks by tehcyder · · Score: 1

      A human programmer cannot possibly think of every possible situation a car might encounter on the street and pre-program an appropriate response .

      I watched a talk by a Google self-driving car engineer; the funniest moment (and an example of your point about pre-programming) was the video showing the time when a Google car came across a woman driving a motorized wheelchair around chasing ducks in the middle of the street.

      Where I live I have seen motorized wheelchairs in the middle of the road, and also ducks crossing the road, although not this precise scenario. The point is that while it might seem unusual to see this on a Californian desert freeway, it isn't really that difficult to enumerate most possible hazards.

      If your automatic car crashes because of trans-dimensional anti-matter vampire bats or something, I hardly think anyone's going to worry about the programming missing out on that possibility.

      --
      To have a right to do a thing is not at all the same as to be right in doing it
    33. Re:The problem with neural networks by tehcyder · · Score: 1

      "Fortunately we can understand the processes within real people that lead to their actions. "

      Since when? Psychiatrics have been claiming that for years but I see little evidence for it beyond simple actions. Sometimes even the person themselves doesn't understand why they do something if it was subconsious.

      But in this context, it's usually something along the lines of "I was texting on my phone while eating a burrito and slapping my kid's face in the seat behind me, which is why I failed to see the red light and hit the schoolbus without even braking".

      It's not really a question of subtle psychological explanations.

      --
      To have a right to do a thing is not at all the same as to be right in doing it
    34. Re:The problem with neural networks by tehcyder · · Score: 1

      The software does NOT have to be "verifiable"

      Says who?

      Says everyone who uses current software, which is everyone. Almost none of it has been formally verified. Why should NNs be held to a different standard?

      I think you're using a different definition of "verifiable" than the rest of us.

      What we mean is that you can reproduce the results given a certain set of inputs, which you most certainly can do with most software.

      You seem to be thinking of some form of pre-approval testing for 100% accuracy, which is a different question.

      --
      To have a right to do a thing is not at all the same as to be right in doing it
    35. Re:The problem with neural networks by fuzzyfuzzyfungus · · Score: 1

      I don't mean to suggest that "anything goes" will become the motto of software testing(at least not more than it is today); but unless all the neural networks deliver extraordinarily erratic behavior despite all effort to the contrary, I agree that it will be a difference of degree(we will test them a lot more rigorously than humans); but not of kind(humans also experience 'edge case' behavior, whether it be an aneurysm hitting them at the hardware level, a psychotic break, a murder-suicide, some ill-conceived plan that involves plowing into a group of pedestrians, whatever).

      We have less historical familiarity with these hypothetical computer systems, so we'll want to test them more carefully; but unless something makes them fundamentally more unknowable and unpredictable than people, my suspicion is that we'll decide to put up with some uncertainty if the alternative is not getting to use the new toys. We don't(and rightly so) put up with uncertainty in life critical systems merely because we want to save a few bucks or shave a few months off development; but if the only entrants are big, hairy, not-really-comprehensible designs that isn't the choice. It will be "you can have this now, or you can hope that we actually understand the problem at a much deeper level within not more than a few generations." There will be grumbling; but unless a real breakthrough allows us to have our toys and understand them, we'll opt for the toys.

  2. If The Neural Net Met The FlimFlamMan by Anonymous Coward · · Score: 0

    They would beget the rubberband man, man.

  3. Can this bubble burst already? by Pinky's+Brain · · Score: 0

    We've been through all this before ... spending 100s of millions just to re-realize that perceptrons are a dead end is a poor use of resources.

    1. Re:Can this bubble burst already? by ogikepenk · · Score: 0

      I agree. http://a-door.com/

    2. Re:Can this bubble burst already? by lorinc · · Score: 2

      I don't know. There is a part of me that says "yeah this is a bubble that's going to burst soon", and another part that says "wait, you've never seen that much improvement on such complex tasks before". Probably the future is in between, and parts of the deep conv nets are here to stay, while some others parts will rapidly be forgotten. But frankly, I don't know, which is a bit scary.

      --
      Video of some good progressive thrash music
    3. Re:Can this bubble burst already? by raftpeople · · Score: 1

      Why do you think perceptrons are a dead end and poor use of resources? A dead-end for what goal?

    4. Re:Can this bubble burst already? by Shortguy881 · · Score: 1

      I've written some pretty crazy decision tree algorithms and some deep learning neural networks. There are use cases of neural networks, including perceptrons, that no traditional algorithm can solve.

      --
      Brilliance without wisdom, power without conscience. Ours is a world of nuclear giants and ethical infants.
  4. Neural networks are for cows. by Anonymous Coward · · Score: 0

    You are all cows. Cows say moo. MOOOOOOOO! MOOOOOO! Moo cows MOOOOO! Moo say the cows. YOU COWS!!

  5. future? by l3v1 · · Score: 1

    Well, FPGAs being the choice for NN implementations is just as a reiteration as the whole deep learning and convnet field is - which is quite OK, since we have now computational tools and resources that we never had before, thus a lot of the NN/convnet/deeplearning theory suddenly became applicable. However, FPGA implementations of artificial/cellular neural networks and convnets dates back something like 20-25 years now, so it doesn't sit well to suggest it's a new direction. What's new however, is that while they could only do max. ~30 fps template matching with FPGA-based NNs back in the days, on very low resolution images, today's FPGAs are real monsters and we can do a lot more now.

    --
    I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
  6. Neural Net Alternatives by Anonymous Coward · · Score: 0

    Neural nets have their place. What I can't stand is the current popular position that they are the be-all, do-all solution for every problem. From my experience, many people use neural nets simply because (a) they've heard the buzz and (b) they don't know what alternatives exist. If all you have is a hammer, everything looks like a nail.

    There are so many learning, pattern recognition, and classification techniques out there -- path-finding algorithms, metaheuristic search techniques (genetics, PSOs), SVMs (especially in combination with high-dimensional kernel spaces), hundreds of flavors of Bayes, etc, just to broadly name a few -- and each is uniquely suited for a particular task. They can also be used in endless different combinations, and selected based on limited resources and/or desired optimality conditions (accuracy, completeness, time constraints, etc.) It all comes down to what works best for a particular problem and a particular task.

    I had a colleague a while back doing some work on problem that was (almost) provably NP-hard. I introduced him to a particle swarm problem that cut down his computation time 100-fold and he was thrilled -- until he realized that a PSO doesn't guarantee global optimality. However, as it turned out, for his particular problem, "good enough" was perfectly acceptable 100% of the time.