Slashdot Mirror
Giving Doctors Grades Has Backfired
HughPickens.com writes: Beginning in the early 1990s a quality-improvement program began in New York State and has since spread to many other states where report cards were issued to improve cardiac surgery by tracking surgical outcomes, sharing the results with hospitals and the public, and when necessary, placing surgeons or surgical programs on probation. But Sandeep Jauhar writes in the NYT that the report cards have backfired. "They often penalized surgeons, like the senior surgeon at my hospital, who were aggressive about treating very sick patients and thus incurred higher mortality rates," says Jauhar. "When the statistics were publicized, some talented surgeons with higher-than-expected mortality statistics lost their operating privileges, while others, whose risk aversion had earned them lower-than-predicted rates, used the report cards to promote their services in advertisements."
Surveys of cardiac surgeons in The New England Journal of Medicine have confirmed that reports like the Consumer Guide to Coronary Artery Bypass Graft Surgery have limited credibility among cardiovascular specialists, little influence on referral recommendations and may introduce a barrier to care for severely ill patients. According to Jauhar, there is little evidence that the public — as opposed to state agencies and hospitals — pays much attention to surgical report cards anyway. A recent survey found that only 6 percent of patients used such information in making medical decisions. "Surgical report cards are a classic example of how a well-meaning program in medicine can have unintended consequences," concludes Jauhar. "It would appear that doctors, not patients, are the ones focused on doctors' grades — and their focus is distorted and blurry at best."
Surveys of cardiac surgeons in The New England Journal of Medicine have confirmed that reports like the Consumer Guide to Coronary Artery Bypass Graft Surgery have limited credibility among cardiovascular specialists, little influence on referral recommendations and may introduce a barrier to care for severely ill patients. According to Jauhar, there is little evidence that the public — as opposed to state agencies and hospitals — pays much attention to surgical report cards anyway. A recent survey found that only 6 percent of patients used such information in making medical decisions. "Surgical report cards are a classic example of how a well-meaning program in medicine can have unintended consequences," concludes Jauhar. "It would appear that doctors, not patients, are the ones focused on doctors' grades — and their focus is distorted and blurry at best."
How could no one have foreseen the potential abuse and pitfalls of a system like this? Without even reading any further than "Giving Doctors Grades..." I immediately conjured images of a bunch of doctors huddled around each other saying, "I don't want that one." "Well I don't want that one either. My feedback is back at 85% and I can't risk another death screwing me over."
bad metrics lead to bad results. Who would've guessed?
Gotta go, must write a million lines of code so I am "productive".
That's the problem with using metrics as incentive. You'll find people caring more about the metrics rather than the outcomes that are actually important.
I think that this Dilbert comic captures the idea quite well.
In NY, where I live, we're now "grading" teachers based on how well their students do on standardized tests. Any teacher who strays from the "prep for the test" subject matter and uses inventive ways of helping their students learn is going to have students who might know more, but who will perform worse on the tests. Teachers who stick to the script and drill test preparation into their students will wind up with better scores even though their students will know less (except how to fill in bubbles).
Just like the Doctors example in the article, the "teacher grading" system is going to backfire. Talented teachers will be kicked out (test scores are tied to their jobs now, your students get low scores and you're out) and mediocre teachers will remain. It's almost like trying to take the jobs that teachers and doctors do and standardize their job functions across every student/patient they see doesn't work. Maybe because their jobs require using their brains and trying different techniques as opposed to an assembly line worker who just needs to perform the same task every time with no variation.
My sci-fi novel, Ghost Thief, is now available from Amazon.com.
Wasn't this addressed by the Scrubs TV show years ago?
competence isnt being measured here. the altruistic goals, "live" or "dead" instead are supplanting good science to determine which doctors are and are not performing well. Death is not objectively bad in cases where it is an unavoidable consequence of environment or genetics. Quality of care and quality of life, the two metrics doctors have always used, is a far better judge of performance. If a 78 year old chronic smoker dies from emphysema then it is of little use to chastise a surgical team or doctor for the death.
Good people go to bed earlier.
It's still a good idea, but the metrics need to be better thought-out to account for the patients that are being seen. A proper system will also "grade" each patient based on how bad their condition is, and then combine the mortality rates to come up with a metric that reflects how well the doctor is doing at improving outcomes where it is possible to do so.
This sounds tough, but how much of the high risk- low success operations being done contribute to the high cost of health care in the US? maybe in some of the high risk situations somebody needs to say no. sorry, but costs are out of control and we need some realistic assessments. On a similar note, its been some years since I've heard people say ' I don't care how much it costs, if it just saved one life it was worth it.'
Painfully obvious that a single metric like this would backfire. A better model is one where we assume (unless demonstrated otherwise) that everyone in the profession at hand is striving in good faith for excellence, then provide mechanisms to self-report errors and close calls without fear of punishment. The body handling this then uses the lessons learned to continually improve the systems and processes that the professionals interact with to lessen the likelihood of impact due to human factor errors in the future. Everyone's weaknesses and experiences in aggregate paint a much better picture of what the ultimate risk mitigation strategy looks like. Check out the airline industry. It works extremely well, and I'm underselling this.
Any system that people can game, will be gamed.
"They often penalized surgeons, like the senior surgeon at my hospital, who were aggressive about treating very sick patients and thus incurred higher mortality rates," says Jauhar.
It is true, some surgeons who are willing to treat very difficult cases would be adversely graded. But shouldn't there be some mechanism to apply brakes to the aggressive treatment? Some patients, and some of the relatives will be seeking treatment even when the situation is utterly hopeless. There are incentives for the doctors and the hospitals to pursue aggressive treatment. So, under these circs, is it really bad these grades are making them reevaluate the cases and be more realistic about the prognosis?
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Yeah, because Bill Gates is THE go-to guy for opinions on software quality.
Suppose a patient comes in for a routine checkup and the doctor finds an advanced cancer and the patient dies. The primary care doctor who had the patient "in for a routine checkup" should not be punished for the poor outcome.
I get the feeling that is not what you meant to happen when you said "losing a patient who was just in for a check-up should count HUGE", but that is what you said. It highlights the difficulty of doing this kind of thing correctly.
--PM