Casting Doubt On the Hawkeye Ball-Calling System

Human judgment by referees is increasingly being supplemented (and sometimes overridden) by computerized observation systems. nuke-alwin writes "It is obvious that any model is only as accurate as the data in it, and technologies such as Hawkeye can never remove all doubt about the position of a ball. Wimbledon appears to accept the Hawkeye prediction as absolute, but researchers at Cardiff University will soon publish a paper disputing the accuracy of the system."

Comments

Why not use...

[Kagura]

Why not use a radio transmitter in the tennis ball (or soccer ball or whatever) to record its exact position? I am certain this has been discussed and I wouldn't be surprised if it's already in use. The article's "Hawkeye" just works by optical analysis.

Re:Why not use...

[Bun]

Why not use a radio transmitter in the tennis ball (or soccer ball or whatever) to record its exact position? I am certain this has been discussed and I wouldn't be surprised if it's already in use. The article's "Hawkeye" just works by optical analysis.

It's been tried in soccer. The latest attempts were prior to the last couple of World Cups IIRC, but the systems were plagued with problems, not the least of which was the survival of the transmitter.

http://www.gizmag.com/go/2790/

Re:Why not use...

[icegreentea]

Assuming you could build a radio transmitter tough enough to handle it...

With tennis balls, I imagine there would be problems with balance and the response of the ball. Especially with such a small ball, mounting a rugidtized radio transmitter (a ball probably has to go through 20gs or something) would probably mess with the balance and how the ball deforms. Not to mention, unless you can mount the system directly in the center of the ball, then you still have a margin of error the diameter of the ball. I imagine that would be fairly significant amount of error in tennis (perhaps on the same level as this Hawkeye system?) when calling the lines.

Re:Why not use...

[InadequateCamel]

Further to that, if the transmitter can't survive in a soccer ball (where a well-struck shot probably moves around 120-130 kph) then there's no way it will handle travelling over 200 kph after a serve, followed by a (at least) 100 kph forehand return (a net >-300 kph in a fraction of a second!).

Also, a radio transmitter cannot account for the distortion of a ball upon impact, which will depend on velocity, angle of rotation, angle of impact, surface being played on, etc etc etc...

Re:Why not use...

[jfim]

Triangulation of radio signals is not accurate enough to give sub-centimeter accuracy and the added mass to the tennis ball would probably cause the players to have some objection to adding a radio transmitter into the ball.

The claim that the Hawkeye system gives an average of about four millimetres of error seems somewhat reasonable, given that we're getting accuracy greater better than two centimetres on detecting objects with a single camera with optics as large as the last segment of a typical pinky. (FWIW, here's a short demo of what we're working on for our autonomous underwater vehicle)

However, the suggestion to display the error range for a particular shot and leaving the final decision to a human from TFA is quite reasonable and is how it should be. Blindingly trusting technology or discarding it altogether is unreasonable.

Re:Why not use...

[Sethumme]

I still don't understand why there isn't more research on developing a surface for the out-of-bounds area that temporarily registers the exact impression of any impact on it.
I envision something that looks like a big LCD touch screen (but more durable). Every time something made contact with the active surface, a record of the ball's "footprint" could be recorded (and even temporarily displayed wherever it touched the surface). That would allow for highly precise measurement of the ball's landing position, and it wouldn't need to incorporate any new materials into the ball itself. The active surface would only need to be in the out of bounds area, and even then, it would only need to be half a foot wide in order to cover the important zone where the ball's landing position is questionable.

Re:Why not use...

[Drathos]

Fox tried to do that with hockey back in the 90s in order to make the puck easier to see on TV (personally, I've never had a problem seeing the puck). The Glow Puck was horrible. When there was a jam up in the corner, it would literally be bouncing all over the screen. It also changed the way the puck performed on the ice. Because of the electronics and battery inside, they couldn't freeze the puck like they normally do, causing it to bounce a lot more and not slide on the ice as easily.

In a hollow sphere like a tennis ball, how would you keep the dynamics of the ball the same as they are when you add a transmitter to it? If you adhere it to the side, the ball will be off balance. If you create some internal structure/support to keep it centered, you change the deformation during a bounce/hit.

Re:Why not use...

[Don_dumb]

At Wimbledon you could have chalk for the lines and if you were unsure if the ball hit the line one of the competitors could point out that

"the chalk flew up"

Re:Why not use...

[rant64]

Also, a radio transmitter cannot account for the distortion of a ball upon impact

I seriously doubt that an umpire can.

Hawkeye's also being used in snooker now, and it actually looks very accurate. The refs always re-spot the ball at least 2 inches away from the spot where it was, and I don't see why they're not using this more often.

Honestly, even if the Hawkeye system is off by a few millimeters, if I were a pro tennis player then I'd rather have a call which is at most 3mm off than being called by an umpire who maybe wasn't paying close attention and calls whatever he thinks is right.

The only solution

[Anonymous Coward]

And ultra-accurate GPS like system that tracks the position of balls in nanosecond detail. They can call it Your Object Universal Remote Movement Observance Mechanism, or YOUR MOM for short.

Anonymous Coward

[Anonymous Coward]

They're reproducing stuff that's already known. Yes, Hawkeye can be inaccurate. However, it's MORE accurate than linesmen and certainly the chair umpire. That's why it's used as the definitive word.

I'd certainly prefer it to be used otherwise - the best way would be to give the chair umpire the information from HawkEye and then let him decide whether to use it or not at any given time, properly educated about the types of errors the machine can make - but that wouldn't be as flashy, would it. So the advertisers wouldn't go for it.

Re:Anonymous Coward

[the_other_chewey]

The accuracy has absolutely nothing to do with the overdetermination of the system.
If it had, it would be simple to reduce the number of cameras to three, and boom - perfect position.
That's obviously not how it is.

And of course does the number of cameras increase the precision of the computed position - the principle
is exactly the same as for GPS, where more satellites are better as well.

Using a certain fitting method (least squares, least absolutes etc.) has nothing whatsoever to do
with something like "complementing the equations", that's just necessary because no measurement is perfect -
You are arguing that multiple measurements do not increase the accuracy of a computed average because there
are multiple averaging algorithms to choose from.

Bullshit.

Re:Anonymous Coward

[DrJimbo]

Since we're only dealing with three dimensions, why would any number of satellites > 3 be more precise for GPS?

If the errors are random and follow a normal distribution (two big ifs, I admit) then even in one dimension, the error is reduced by a factor of 1/sqrt(N) where N is the number of measurements.

The same general idea applies to higher dimensions. If you can avoid systematic errors then the more measurements you take, the more accurate your final result will be. If you are interested in the gory details of the higher dimensional case, you should take a look at singular value decomposition.

Re:Anonymous Coward

[the_other_chewey]

Since we're only dealing with three dimensions, why would any number of satellites > 3 be more precise for GPS?

Because we are dealing with reality as well - where no measurement is perfect.
Geometrically, three sats indeed are enough, but in reality:
More measurements -> smaller error bars -> better position.
The alternative to more sats would be not to move and to take more measurements over time.
But that would render GPS useless for most applications ;-)

Additional trouble with the "stay and wait" method: Those nasty satellites move over time,
introducing different errors that can not be eliminated as easily by simple averaging.

That's also why ultra precise GPS surveying records the satellite data and waits for the week it takes
to make the actual orbital data (as measured, and not just as predicted) available before computing
the position, thereby elimiating (well, at least reducing) another source of error.

In statistics, the only thing beating multiple measurements is even more measurements.

Re:Anonymous Coward

[SnowZero]

A system such as Hawkeye CANNOT BE MORE ACCURATE than humans. From the link in the article, the Hawkeye system uses 5 cameras to compute the 3D position of the ball. That's an overdetermined system of equations, which cannot have a unique solution due to observation errors in the camera views.

Luckily there's a 100+ year old discipline called statistics, and 60+ years of literature on tracking to help you out in these cases.

So Hawkeye has to complement the equations with an ARBITRARY rule, eg least squares and this arbitrariness makes the Hawkeye estimate neither more accurate nor less accurate than humans, just different. FYI, there are plenty of other arbitrary rules that work, eg least absolute errors, maximum entropy, etc.

While I can't speak for the designers of the Hawkeye, in tracking there are very good reasons to choose one form of error minimization versus another. It only seems arbitrary because you are not informed on the subject, but there's plenty of free papers out there to read and discover.

To explain current methods, please start out with this paper, in particular Figure 2, you'll see that the sort of errors you get from a camera are indeed well fit by a Gaussian. While a camera's perspective transformation is not purely linear (and various forms of distortion make it also non-linear), a good camera with a decent lens estimating the ball location within a limited area is well approximated by a linear model (and you can characterize just how much the error is). Now, a bunch of cameras with a Gaussian error distribution in the image plane with a linear projection out into the world is still a Gaussian (with a transformed covariance matrix). You can then multiply the independent measurements from multiple cameras to get a better estimate. Add a time series to that and apply this recursively and you get a Kalman filter, something invented for aerial tracking and still in widespread use today. If something is good enough for missiles to intercept other missiles, it ought to be good enough for a tennis match.

If the linear approximation not good enough for you, you can use a Rao-Blackwellized Kalman filter. If that's still not good enough because you want to use another error distribution or non-linearizable dynamics, set up a particle filter with a whole lot of particles and enough CPU to simulate it. The point is that what you call arbitrary is a well studied field which is many decades old. You'd be best served by learning about it first before you cast away all that work. I'm not a "tracking" person, just a user of there work. When a field of science has done its job well enough that it has become common engineering, and you can go look up whatever you need in books, with all the derivations, caveats and tradeoffs laid out there for you to see, I would say that that field has done a pretty good job.

The whole media story around this paper is ridiculous. It's a paper from a social sciences department about how the public does not understand the fallibility of these machines due to noise. That's all this paper is about: Hawkeye has error. I hate to break it to the uninformed, but all measurement systems have error. From Galileo to Gravity Probe B, your results can only be as accurate as your measurements, calculations, and statistical models will allow. You can decrease error with various methods, but you can never completely eliminate it. People should not be able to get out of high school without understanding accuracy on measurements, and some rudimentary statistics, but unfortunately our education system hasn't been able to reach that goal. As a result, the public doesn't understand error, and might come to believ

Re:Anonymous Coward

[Moridineas]

I'm willing to concede that you are talking theory at some level I don't fully grok. What I think you're completely missing in this discussion stems from your original statement that"system such as Hawkeye CANNOT BE MORE ACCURATE than humans", which does not seem to be possibly true by any standard definition of these words that I am familiar with.

You can talk about "error criterions" and odd offtopic tangents about targeting algorithms etc, but the bottom line is, your original statement is completely wrong.

You say "So Hawkeye has to complement the equations with an ARBITRARY rule, eg least squares, and this arbitrariness makes the Hawkeye estimate neither more accurate nor less accurate than humans".

That's both wrong and illogical. Yes, Hawkeye is estimating a solution, and using a "arbitrary" (again, this is utterly bizarre and incorrect word choice--the makers of Hawkeye have presumably done a great deal of testing to pick an algorithm, which is NOT arbitrary) method to estimate. However, if Hawkeye ESTIMATES the correct answer more often than a human judge then, Hawkeye is more accurate than a human judge. The methods it uses are really completely irrelevant to the final answer.

So in short, it seems that this is a discussion in your usages of "accurate," "error," "arbitrary," etc are different than the rest of the people in the thread.. Please let me know if I'm misinterpreting something though!

Re:Anonymous Coward

[Rakishi]

Just because an umpire is the final word doesn't mean that a system can't do better than him, That is because the umpire is in fact he trying to measure something with a right/wrong answer. Specifically the umpire is the person who decides if event X happened or not which means that the goal is to see if X happened or not (not to see if the umpire thought X happened or not). The umpire isn't an inherent part of the rules but simply a judge to determine if something specified in a certain rule happened or not. As a result it's a perfectly valid problem to predict this event X in a method that is better (ie: lower misclassification) than the umpire. Finding the winner in a horse raise is one example of where technology is more accurate despite the rules likely having a person originally be the final judge.

One problem is that sometimes one can't measure the true answer in some way so there is no way to truly measure accuracy for a problem. That is a valid problem however I have no clue if that or something else is the actual problem you're so concerned about (your posts are as clear as black mud). In this case there probably are more accurate systems of measuring the truth although these take excessive money, time or preparation. One could for example cover the ground around the line with wet paint (or some such) and then check for breakages, or simply cover the ground with pressure sensors. The article implies they can measure the accuracy of the system compared to the true impact point which means that one can devise experiments in which one can measure the truth of where the ball lands.

Consistency, is more important than accuracy!

[Anonymous Coward]

Hawkeye and the like deliver a consistent result. It matters not at all if the ball is in by two Centimetres but is called out, provided that error is consistent throughtout the match.
If both players, or teams, are playing by the same margin of error, the contest is fair.
In cricket for instance, I would accept the computers call over umpires any day of the week!

Re:Consistency, is more important than accuracy!

[Telvin_3d]

That is only true if you assume the two players are making the same level of mistakes. If both players are regularly hitting the same shots witht he same amount of error, yes everything will even out. But let's say player A can consistently serve and hit the ball to within 2 cm of the out line, but player B often misjudges and goes 1 cm over. In this case, having player A's shots consistently called 'out' or player B's shots consistently called 'in' would be consistent, but it would also make a major change in the outcome of the match. And not the type of change that gets statistically evened out over games played.

Re:Why not, it works for shopping carts

[Anonymous Coward]

So if one of the players tries to steal a tennis ball, they won't get very far?

Summary Miseleading? No Wai!

[Zackbass]

For those that didn't care to RTFA, the study is in the journal 'Public Understanding of Science' and (gooly who would have guessed) doesn't have anything to do with the summary written. They argue that uncertainties in measurement that normally don't impact the layman now need to be presented in an understandable way. They worry that people will wrongfully become too trusting of the systems that do have appreciable error in rare circumstances.

To inject my own opinion on the matter, I've had a little bit of experience with Vicon motion capture systems which appear to use similar technology to the Hawkeye system. The main problem with the system (when it works) isn't any problem with accuracy or precision. In fact, it's awesome. The problem is that the output is a little noisy and suffers from occasional jumps and hiccups. With proper filtering these are eliminated and the output is amazing. I can only imagine the problem is much easier when you're tracking a single ball rather than tens of tiny reflective makers such as with the Vicon system.

Read The Friendly Article

[MaliciousSmurf]

Because that's not the issue. You'll always have uncertainty in systems. The study argues that the public perceives these systems as infallible, and therefore believe that technology can provide a final, absolute arbitration. The study is commenting on this tendency in lay people (i.e., people without specialized knowledge of the system), and warns against the corollaries that stem from such assumptions. Also, the title is bad: they are merely looking at the issue through the lens of Hawk-Eye, they're not looking at Hawk-Eye specifically. You may note that there is no analysis of the hawk eye system beyond a basic discussion of its function.

Re:major league base ball umpires union does not l

[DriedClexler]

I'm confused. Why would umpires oppose a technology that can automate the refereeing of a game? It just doesn't make any sense.

Refereeing is by many considered PART of the game.

[SamP2]

For a system like Hawkeye to be useful, it doesn't need to be perfect. It just needs to consistently be more accurate and impartial than a referee can be.

Nor is it required for the system to be fully automatic and autonomic. Referees can sit in front of their monitors, observe the cameras from all angles, with any time slowdown, and ultimately come to a better decision than a single person could make while the ball buzzes past them at Warp 9.

But from the social aspect, one has to decide on what is the referee's role, and what kind of influence, if any, do we want to delegate to a computer. And that depends on the type of sport.

For non-interactive sports such as sprinting, an automatic system works very efficiently, and most people readily accept it as better than a human time tracker.

But for many GAME sports (soccer and boxing come to mind) many people consider that a referee is PART of the game rather than just an observer. As long as a referee is comparatively competent, and acts in good faith, he has the authority to judge events in the game, and while mistakes are unavoidable, they are considered part of the game as well.

I'm not sure why this position is popular in these kinds of sports. Maybe it's the whole "humans should be judged by humans and not machines" aspect. Or maybe it's because having a Review Comission in front of CCTV monitors be judging every little move just feels too 1984-rish for spectators and players alike. Or maybe its something else. But this is a rather popular feeling.

Depending on the features and benchmarks of the electronic system, it may or may not be more accurate than a human observer. In the long term, a joint human-computer analysis system would be certainly more accurate than a human referee alone, especially in team or high-speed sports. But one has to ultimately question, whether, by gaining mathematical precision, we lost some human touch of sport that makes it enjoyable to play and watch. Fun can't be generated with a mathematical formula. And sometimes sitting on the couch and thinking "OMG that referee is such a dumbass" is part of the fun as well.

Re:Why not use... LASERS!

[janrinok]

They have already experimented with this idea, but had problems keeping the sharks under control.

Re:Other applications?

[InfoHighwayRoadkill]

Yes, some people also want to use Hawkeye for some decisions in cricket, the sport that first used it. However the margin of error is far greater (approximately +- 2 inches) in cricket as the cameras have to be a lot further away due to the size of the pitch.

Also Hawkeye finds it hard to pick up swinging, seaming and spinning balls. Basically anything that deviates off its theoretical trajectory either in the air or off the playing surface. Both of which are vital in the LBW decisions where the TV companies and doubtless the Hawkeye people would want to see it used.

Obviously cricket is a far more useful game than tennis so does this answer your question?

Re:Hawkeye is rather redundant in cricket actually

[InfoHighwayRoadkill]

Hawkeye cannot 'hear' a snick to give a 'caught behind'.

the tv companies have a "snickometer" which puts up an analysis of the sounds picked up by a microphone in the stumps. Its only used for commentary. The umpire makes the decision himself

Hawkeye cannot (as far as I can tell) decide if a ball is caught or if the fielder let it slip through his fingers as he scoops it up the ground.

A good tv replay can show this but as cricket is a gentleman's game it is up to the fielder making the catch to say if he thinks he made a clean catch. There have been instances in test cricket where fielders have called back batsmen after the umpire initially gave them out.

Hawkeye cannot tell if a Leg Bye or simple bye was scored.

No but the umpire can, hawkeye finds it very hard to spot a ball that deviates from its theoretical trajectory at the best of times

I don't believe it can decide a 'wide' as there is no fixed length rule.

you answered your own question there

Hawkeye cannot tell if a ball was caught inside or outside the boundary.

Thats because its looking at where the ball is being bowled in the middle of the playing area, it doesn't cover the whole of the field

Hawkeye cannot decide a run out.

That is because it is used to approximate the trajectory of the ball as its being bowled. Not when its being throw to the stumps and the relative position of the batsmans feet and bat. TV slo mo replays decide run outs (if the umpire is unsure) and are ideal for the purpose

Hakweye cannot tell if the ball hits the helmet often left behind the wicket keeper (5 runs)

the normally loud noise the ball makes when it hits the helmet and the ball shooting off in a different direction often suffices.

Hakweye cannot even decide a no ball yet.

As previously stated hawkeye doesnt watch peoples feet it watches the ball

The only thing Hawkeye was/is used for is to decide an LBW decision which is a small percentage of 'outs' in a given game, and also to show where balls have been pitched for a given bowler.

Its only used for this purpose for the tv commentators to have something to talk about. The margin of error and the problems with picking up balls that swing in the air or move off line from the pitch make it impossible to give an accurate ruling on an LBW.

Umpires in Cricket are going nowhere.

Its because 90% of decisions made in cricket are made by the umpires without needing back up that makes cricket a fascinating sport.

Re:Other applications?

[stranger_to_himself]

Yes, some people also want to use Hawkeye for some decisions in cricket, the sport that first used it. However the margin of error is far greater (approximately +- 2 inches) in cricket as the cameras have to be a lot further away due to the size of the pitch.

The other key difference in cricket is that Hawkeye is used to predict where the ball would have gone had it not hit a pad, whereas in tennis it only needs to say where the ball actually was.

Re:It doesn't have to be perfect

[Fred_A]

This assumes there is another method, such as post-analysis of videotape, that can find almost all uncorrected errors or at least give some good indication of the uncorrected error rate.

Another method would be to use Radar instead of Hawkeye. Probably faster and more efficient as well.
(obscure reference).

Re:Why not, it works for shopping carts

[b4dc0d3r]

That's compatibility for legacy (analog) shopping carts with the one wheel stuck sideways.

O RLY?

[davidwr]

You Pierce me with your wit! It Burns! When you finally drag out your Winchester and kill me, you'll have to bury me in a field fit for a Potter because I can't afford anything better.