Slashdot Mirror

There's a lot of rounding error with FP16.

Sure, but it doesn't matter. Backprop, learning rate, denoising, etc. all just heuristics anyway. So what if your mantissa is off by one bit? You get better accuracy by going wider, adding layers, and (most importantly) using more data. But you can't afford to do that if half your bandwidth is sucked up transmitting meaningless precision.

Also, do you have a good citation that FP16 neural networks are, overall, more effective than FP32 networks, as you've described?

They are not necessarily more effective, just more efficient. If you have infinite resources, you might even get better results using FP32. But resources are never infinite. Here is a guy who claims that even 8 bits is enough for deep NNs.

http://petewarden.com/2010/02/06/how-to-split-up-the-us/

I'm pretty sure Apple had motion estimation happening on the GPU pre-October 2004. Here's a May 10, 2005 post by Pete Warden about how he implemented it. I believe he implemented it in the "Movement Blur" filter in Motion before making that post, though I'm not positive:

http://petewarden.com/notes/archives/2005/05/gpu_optical_flo.html#more