Slashdot Mirror
Casting Doubt On the Hawkeye Ball-Calling System
Human judgment by referees is increasingly being supplemented (and sometimes overridden) by computerized observation systems. nuke-alwin writes "It is obvious that any model is only as accurate as the data in it, and technologies such as Hawkeye can never remove all doubt about the position of a ball. Wimbledon appears to accept the Hawkeye prediction as absolute, but researchers at Cardiff University will soon publish a paper disputing the accuracy of the system."
The decision of which system to use: human, computer, human with computer check, computer with human check, committee vote, or what-not should be based on which has the lowest uncorrected error rate within limited time constraints.
This assumes there is another method, such as post-analysis of videotape, that can find almost all uncorrected errors or at least give some good indication of the uncorrected error rate.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Can this be applied to something useful. You know besides whether or not someone was out in a game of tennis?
So Hawkeye has to complement the equations with an ARBITRARY rule, eg least squares, and this arbitrariness makes the Hawkeye estimate neither more accurate nor less accurate than humans, just different. FYI, there are plenty of other arbitrary rules that work, eg least absolute errors, maximum entropy, etc.
Assuming you could build a radio transmitter tough enough to handle it...
With tennis balls, I imagine there would be problems with balance and the response of the ball. Especially with such a small ball, mounting a rugidtized radio transmitter (a ball probably has to go through 20gs or something) would probably mess with the balance and how the ball deforms. Not to mention, unless you can mount the system directly in the center of the ball, then you still have a margin of error the diameter of the ball. I imagine that would be fairly significant amount of error in tennis (perhaps on the same level as this Hawkeye system?) when calling the lines.
Further to that, if the transmitter can't survive in a soccer ball (where a well-struck shot probably moves around 120-130 kph) then there's no way it will handle travelling over 200 kph after a serve, followed by a (at least) 100 kph forehand return (a net >-300 kph in a fraction of a second!).
Also, a radio transmitter cannot account for the distortion of a ball upon impact, which will depend on velocity, angle of rotation, angle of impact, surface being played on, etc etc etc...
Triangulation of radio signals is not accurate enough to give sub-centimeter accuracy and the added mass to the tennis ball would probably cause the players to have some objection to adding a radio transmitter into the ball.
The claim that the Hawkeye system gives an average of about four millimetres of error seems somewhat reasonable, given that we're getting accuracy greater better than two centimetres on detecting objects with a single camera with optics as large as the last segment of a typical pinky. (FWIW, here's a short demo of what we're working on for our autonomous underwater vehicle)
However, the suggestion to display the error range for a particular shot and leaving the final decision to a human from TFA is quite reasonable and is how it should be. Blindingly trusting technology or discarding it altogether is unreasonable.
Jean-Francois Im's blog
I still don't understand why there isn't more research on developing a surface for the out-of-bounds area that temporarily registers the exact impression of any impact on it.
I envision something that looks like a big LCD touch screen (but more durable). Every time something made contact with the active surface, a record of the ball's "footprint" could be recorded (and even temporarily displayed wherever it touched the surface). That would allow for highly precise measurement of the ball's landing position, and it wouldn't need to incorporate any new materials into the ball itself. The active surface would only need to be in the out of bounds area, and even then, it would only need to be half a foot wide in order to cover the important zone where the ball's landing position is questionable.
I thought it was about University of Iowa football for a while.
That is only true if you assume the two players are making the same level of mistakes. If both players are regularly hitting the same shots witht he same amount of error, yes everything will even out. But let's say player A can consistently serve and hit the ball to within 2 cm of the out line, but player B often misjudges and goes 1 cm over. In this case, having player A's shots consistently called 'out' or player B's shots consistently called 'in' would be consistent, but it would also make a major change in the outcome of the match. And not the type of change that gets statistically evened out over games played.
Combining observations isn't arbitrary, its based on prior knowledge of the underlying statistics and measurement methods. If the multiple measurements are identical with normally distributed error, for example, the mean can be used. If the measurement is subject to random catastrophic failure (e.g. bit flipping), then the median is a good choice. In the Bayesian method you form a composite probability distribution by combining conditional or joint probabilities. In fact, if you do it wrong, you can make the final answer *worse* than any of the original measurements (this is called catastrophic fusion). The method is NOT arbitrary--making that assumption will get you into big trouble fast.
By the way a system like this has potentially many, many more observations than just five--since it also uses position and velocity estimates from previous frames to compute the most likely next position of the ball. With five high-speed cameras combining data into a Kalman filter you are looking at hundreds if not thousands of measurements of the ball trajectory, which will give you enough data to estimate subtle qualities like the spin of the ball and so on (by extension the number of variables is by no means limited to three, since one can estimate any number of higher order features--e.g., velocity, acceleration, angular velocity, wobble, etc).
It isn't hard to engineer machines that surpass the accuracy of a human in a variety of tasks, and the question of "which one is right" is not merely subjective but described up by a body of math known as signal detection theory. This math by the way came out of the subfield of psychology dedicated to measurement of the thresholds of discrimination by human judgement with respect to physical phenomena--psychophysics. The resolving power of a measurement system can be quantified by its discriminability index, and decision-making processes based on that information are described by positions along the corresponding ROC (receiver operating characteristic) curve.
For a system like Hawkeye to be useful, it doesn't need to be perfect. It just needs to consistently be more accurate and impartial than a referee can be.
Nor is it required for the system to be fully automatic and autonomic. Referees can sit in front of their monitors, observe the cameras from all angles, with any time slowdown, and ultimately come to a better decision than a single person could make while the ball buzzes past them at Warp 9.
But from the social aspect, one has to decide on what is the referee's role, and what kind of influence, if any, do we want to delegate to a computer. And that depends on the type of sport.
For non-interactive sports such as sprinting, an automatic system works very efficiently, and most people readily accept it as better than a human time tracker.
But for many GAME sports (soccer and boxing come to mind) many people consider that a referee is PART of the game rather than just an observer. As long as a referee is comparatively competent, and acts in good faith, he has the authority to judge events in the game, and while mistakes are unavoidable, they are considered part of the game as well.
I'm not sure why this position is popular in these kinds of sports. Maybe it's the whole "humans should be judged by humans and not machines" aspect. Or maybe it's because having a Review Comission in front of CCTV monitors be judging every little move just feels too 1984-rish for spectators and players alike. Or maybe its something else. But this is a rather popular feeling.
Depending on the features and benchmarks of the electronic system, it may or may not be more accurate than a human observer. In the long term, a joint human-computer analysis system would be certainly more accurate than a human referee alone, especially in team or high-speed sports. But one has to ultimately question, whether, by gaining mathematical precision, we lost some human touch of sport that makes it enjoyable to play and watch. Fun can't be generated with a mathematical formula. And sometimes sitting on the couch and thinking "OMG that referee is such a dumbass" is part of the fun as well.
I would accept the computers call over umpires any day of the week!
rubbish! then what would we all have to argue about afterwards in the Pub?
Actually Hawkeye is pretty poor for cricket.
Hawkeye cannot 'hear' a snick to give a 'caught behind'.
Hawkeye cannot (as far as I can tell) decide if a ball is caught or if the fielder let it slip through his fingers as he scoops it up the ground.
Hawkeye cannot tell if a Leg Bye or simple bye was scored.
I don't believe it can decide a 'wide' as there is no fixed length rule.
Hawkeye cannot tell if a ball was caught inside or outside the boundary.
Hawkeye cannot decide a run out.
Hakweye cannot tell if the ball hits the helmet often left behind the wicket keeper (5 runs)
Hakweye cannot even decide a no ball yet.
and so on
The only thing Hawkeye was/is used for is to decide an LBW decision which is a small percentage of 'outs' in a given game, and also to show where balls have been pitched for a given bowler.
Umpires in Cricket are going nowhere.
Yes, and this discipline depends on something called decision theory, which in turn depends on an arbitrary choice of loss functions.
None of this matters one bit if these reasons are not compatible with the sporting rules in the problem at hand. To be pedantic, if the rules say that an umpire has the final word (for example), then a tracking system which doesn't optimize for the same criteria that the umpire uses himself is irrelevant. To be even more pedantic, if one claims that the tracking solution is superior to an umpire's criteria if those criteria differ, one is merely trying to change the rules of the game.
Thanks for the links, but I am actually familiar with (and have used) Kalman filtering in the past. The issue I raised is not an engineering one. It occurs before engineering begins, namely in the problem specification. More precisely, I responded to a post claiming that the Hawkeye system was obviously more accurate than human referees, which I consider far from obvious and said so. Perhaps I could have talked about loss functions instead of overdetermined systems, but the gist of my point remains.
My understanding is that the umpire is the final arbiter. People are free to come up with a model and a methodology to compute their own best version of the head judge's decision, but unless the rules specify the methodology completely, it's merely an academic exercise with no intrinsic validity for the game.
But certainly, to talk about a system being more accurate merely because it uses engineering methodologies when the problem is not fully specified to begin with is ridiculous, and verges on technology worship.
Long answer: The ranges calculated in GPS are estimates, because the clocks in the receiver aren't very precise. A small offset in the timing can cause a large error in the calculated distance (if the clock is off by 1/1000th of a second, you're actually 200 miles away from where you think you are). This is why GPS usually uses 4 satellites. If the receivers all had atomic clocks on them, every set of measurements from any three satellites would end up at the same exact point, because the clocks are so precise. The quartz clocks in GPS receivers drift out of sync with the clocks on the satellites, and this drift is enough to cause pretty large inaccuracies. In other words, if you measure the ranges from three satellites, and then subsequently measure the range from a fourth, the fourth satellite's measurement will not align with the other three. When this happens, the GPS unit makes adjustments to the 4 measurements until they all align in a single point. This effectively eliminates the clock issue. To get an even more accurate measurement, the GPS receiver will try to acquire as many satellites as possible and take measurements in groups of 4. This helps eliminate other errors caused by interference, atmospheric anomalies, highly reflective goats, etc.
Show me on the doll where his noodly appendage touched you.
Look, statistics is much more complicated than averaging. Do you know where the averaging rule that you use comes from? It's the maximum likelihood estimator: It's a function of the observations which is obtained under certain assumptions on the physical process (which in your case would typically be a Gaussian distribution of errors, all independent).
It makes no sense to claim that accuracy is improved by averaging without subscribing to those or similar assumptions. In fact, there are other rules, for example you might add some extra dummy values to your measurements if you have a Bayesian prior assumption, etc. The point is what works in your case need not work in other, superficially similar, problems, especially if the risk function is different.
When one solves an overdetermined system, one implicitly includes some assumptions. The answers one gets are only as good as the assumptions one puts in, garbage in garbage out etc. There simply is no universal solution to an overdetermined system.
Why not stick the sensor on the pitch itself, along the relevant lines. Then it would detect exactly where the ball bounced.