Slashdot Mirror
Computer Modeling Failed During the Ebola Outbreak
the_newsbeagle writes: Last fall, the trajectory of the Ebola outbreak looked downright terrifying: Computational epidemiologists at Virginia Tech predicted 175,000 cases in Liberia by the end of 2014, while the CDC predicted 1.4 million cases in Liberia and Sierra Leone. They were way off. The actual tally as of January 2015: A total of 20,712 cases in Guinea, Liberia, and Sierra Leone combined, and in all three countries, the epidemic was dying down. But the modelers argue that this really wasn't a failure, because their predictions served as worst-case scenarios that mobilized international efforts.
in the end.
Garbage in ... Garbage out - GIGO
That computer simulation failed simply because all the input that the program got fed with were erroneous
Which is just akin to crying wolf. Next outbreak: "They're predicting 2 million infected, but remember ebola? It never got that bad. Take your time."
People who work in population dynamics know that the models are based on a very crude understanding of the disease and ultimately general epidemiological assumptions. The fact that there are basic assumptions is sometimes disguised in the process of making fancy computer models.
These models may be the best predictions that people can make. However, GIGO (garbage in, garbage out) still applies. However, sometimes the best predictions are not good enough since they can be very misleading.
(I used to work on similar models and became disenchanted; I need to post anonymously.)
I've been involved in contracts that had public health modeling components. Being "way off" is not necessarily a proof the model is no good when you're modeling a chaotic process which depends on future parameters that aren't predictable. In our case it was the exact timing of future rainfall. In their case it probably had to do with human behavior. A small thing, like an unseasonable rainstorm, or an infected person showing up in an unexpected place, can have immense consequences.
You look at all the data you have, and you think, "Hey, this is a lot of data, I should be able to predict stuff from it," but the truth is while it looks like a lot of data it's a tiny fraction of all the data that's out there in the world -- and not even a representative sample. So you have to guess "plausible" values, and if they're wrong you might not see the kind of result that eventually happens, even after many model runs.
So in most cases you can't expect a computer model to have the power to predict specific future events. It can do other things, like generate research questions. One of our models suggested that having a lot of infected mosquitoes early in the season reduced human transmission of a certain mosquito borne disease later in the season, which was a surprising result. When we looked at it, it turned out that the reason was that the epidemic peaked in the animal population early in the season before people were out doing summer stuff and getting bit. Does that actually happen? We had no idea, but it sounded plausible. The model didn't give us any answers, it generated an interesting question.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Software people are unable to understand the real world. But they're convinced we can build Mars colonizing spaceships based on decades-old fantasies...
Scientists who do not admit they had made mistakes are pseudo-scientists (So are most politicians, too).
If nothing else, this shows just how bad an idea it is to put too much trust in computer models. There are always factors that we either don't know about or don't know how to include properly and getting even one of them wrong can throw the whole model off. Yes, computer simulations and models can be very, very useful but you have to take the results with a grain of salt and remember that they're only approximations at best.
Good, inexpensive web hosting
"Its not a bug...... Its a feature!"
This is like a car manufacturer claiming that their car will have 1,000 horsepower, and after several months/years the people who have preordered it finally get theirs and find out it has 20 horsepower and the manufacturer says its a good thing because it makes the car safer.
The problem with blaming prediction models after the fact is that the models were based on the assumption of current and continued support. Ebola just like the Y2K bug was everything the disaster it may have been had it not been for the efforts involved in preventing it.
End result, all the people who did the hard work making the world aware of the problem are blamed for crying wolf.
To quote my beloved Bryan Caplan (http://econlog.econlib.org/archives/2015/06/the_sum_of_all.html):
"I didn't think there was anything more to say about infamous doomsayer Paul Ehrlich. Until he decided to justify his career to the New York Times. Background:
'No one was more influential -- or more terrifying, some would say -- than Paul R. Ehrlich, a Stanford University biologist... He later went on to forecast that hundreds of millions would starve to death in the 1970s, that 65 million of them would be Americans, that crowded India was essentially doomed, that odds were fair "England will not exist in the year 2000." Dr. Ehrlich was so sure of himself that he warned in 1970 that "sometime in the next 15 years, the end will come." By "the end," he meant "an utter breakdown of the capacity of the planet to support humanity."'
Okay, here's Ehrlich's side of the story:
After the passage of 47 years, Dr. Ehrlich offers little in the way of a mea culpa. Quite the contrary. Timetables for disaster like those he once offered have no significance, he told Retro Report, because to someone in his field they mean something "very, very different" from what they do to the average person.
In the video interview, Ehrlich elaborates:
'I was recently criticized because I had said many years ago, that I would bet that England wouldn't exist in the year 2000. Well, England did exist in the year 2000. But that was only 14 years ago... One of the things that people don't understand is that timing to an ecologist is very very different than timing to an average person.'"
...maybe the crazy predictions opened enough eyes to get the ball rolling on containment...therefore nullifying the predictions?
THIS is how science is done these days. Computer model the result you want, ignore reality, and call people who question it names.
Seems to be happening a lot lately.
supporters say when their models failed in 30 or 50 years. Will they also argue that these really aren't failures, because their predictions served as worst-case scenarios that mobilized international efforts?
The media love it because scary headlines attract eyeballs, but scare tactics can backfire: Researchers might get more funding, but they also might get their research banned: http://victimsofdsto.byethost3...
When American Pharoah won the triple crown. Come to think of it, computer modeling usually fails.
FFS, it was even obvious at the time that they were basically just looking at a graph and, rather than using their pen to draw a line that "kinda looks the same", they ran through a list of dozens of algorithms until the computer spit out a bunch of fancy math supposedly justifying the drawing of a line that "kinda looks the same". There was nothing remotely scientific about it, and it has as much to do with predictive modeling as lighting a firecracker in a frog's behind has to do with putting a man on the moon.
everyone will just ignore the models, more people will die, and the predictions will be right. \o/ yay!
So the ends justifies the means. Got it.
Stupid sexy Flanders.
But the modelers argue that this really wasn't a failure, because their predictions served as worst-case scenarios that mobilized international efforts.
s/ebola/climate/g s/deny/assert/g s/assert/deny/g
I think computer models only work when the target data is too far out to verify.
I would rather we end up with fewer losses than expected, than more. We learned something from this problem, and the cost was not as bad as it would have been had the error been in the other direction.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Well Stated, Reasoned and Accurate. No surprise that someone modded you overrated.
Anyway just to add to the statement, Everyone has biases, computer models tend to amplify those biases to absurd proportions, You can only hope whoever is running the model is doing a good job of validation and doesn't have an agenda.
Computers are infallible. If a computer 'says' something we know that it is correct. The link between the assumptions that make the model and the model itself is seamless and is science. In this 'science' experiment is not needed and is possibly even harmful.
You learn the truth about the world from building your computer model. This isn't the 19th century anymore. Now we have computers. Actually doing something out there in meatspace to observe what happens when you try it is crude and unnecessary and in any case is subject to human error.
Don't question the assumption-model connection because computer models = science and the people who code the models are scientists. The fact that those people are 'scientists' also means they are correct. Who are you to question them? You'd better at least have a PhD in the relevant field.
If you question what human assumptions the model is built from you are anti-science and should be ignored as a religious nutjob and no it doesn't matter if you claim to be an atheist.
Quite an experience to live in fear, isn't it? That's what it is to be a slave.
I don't know if that is the worst case scenario. The worst case scenario is that is goes global and there is a huge fast selection pressure on our species and many/most of us die. If that happened it would be interesting to see if the children of the survivors had a natural immunity.
Is this Y2K all over again?
If math errors by computers due to hardware or software make us take the wrong assumptions? Especially in physics or other high end math fields.
"If any question why we died, Tell them because our fathers lied."
The way the model results are reported needs to change. The worst case results were presented to the public as the expected outcome. This is something between highly deceptive and unethical. (think yelling "Fire!" in a crowded movie theater.) The best, worst and average outcomes from the model need to be reported. Perhaps even two sets of best, worst and average outcomes. One with large scale intervention and one with zero intervention.
A very simple way to think about when you know the model has failed: The model has failed when it makes 100 predictions with 95% certainty and more than 5 of the actual outcomes are outside the bounds defined by the best and worst outcomes. Note: I said SIMPLE.
The modelers need to be careful about what they say. Next time they predict armageddon, no one will take them seriously.
What a bunch of idiotic posts. Model results are associated with predicted range of probabilities (eg 80 percent chance of rain). We depend on weather reporting even when they are wrong on occasion. Why? ...because our weather models are pretty good(despite chaos).
A Good Troll is better than a Bad Human.
Statistical simulations are suppose to include estimates of probabilities of events. That you never hear the numbers for these extremely long run (i.e. error-prone) forecasts is deliberate negligence for political purposes. Weather forecast (which use similar methods) are delivered like "40% chance of rain on Tuesday". That you are not told these numbers means you are being manipulated. I'll go ahead and make the jump for you: statistical computer simulation of Ebola spread -> statistical computer simulation of climate change.
The models were right. What they didn't expect is that during the week of October 14, 2014 the r-naught factor dropped by an order of magnitude.
http://www.nytimes.com/2014/09/06/health/ebola-immunity.html
http://en.ird.fr/the-media-centre/scientific-newssheets/337-possible-natural-immunity-to-ebola
http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0009126
Plos paper was so close. Man and bat eat the same fruit. Now ask yourself what that fruit was. If only they had.
People in Africa know what to do now. There is not now and can never be a vaccine for this disease, fortunately it's not needed.
The Doctor
THIS is how science is done these days. Computer model the result you want, ignore reality, and call people who question it names.
Seems to be happening a lot lately.
THIS is how greed is fed these days.
Now, if that greed happens to feed funding to ensure future outbreaks do not model the worst-case scenario, I'm all for it. Gets those in power to wake the hell up and spend more than a nickel on prevention.
Unfortunately, for the other 99% of models, greed and corruption fund the study, so you can imagine how accurate the results would be. You've already outlined the process.
Now is the time for Modellers to RISE.
RISE ... MY BEAUTY ... RISE
WANTON ... DOWN, WANTON ... DOWN. You must be patient and RESPECTFUL. [Smack ... smack ... smack.]
http://www.poetryfoundation.org/poem/180835
Computer modeling is vastly overrated. It is mostly based on the abstraction of trend lines. Which is the assumption that existing trends will continue. That is less a prediction of hte future than a picture of the present.
Look at the growth trend line of a six year old... then graph that out... in 10,000 years think how big he'll be!
Right?... that's what trend lines do... They're only useful if people that know what they are and how they work use them. Often as not, people that aren't educated or knowledgeable enough to deal with them get put in positions where they can make determinations about stuff using those trend lines.
And it generally leads to a shitstorm of stupidity.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
I love that this can't possibly be a case of the model helping the resolution of the outbreak.
if we let things go to shit like we normally do. Because of this we didn't let things go to shit and so the model was wrong...? How is this a bad thing, or a failure of the model. This is like when people say regulations aren't needed anymore because the abuses they were put in place to stop have stopped. They stopped for a reason people. Figure it out, it's not that difficult.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
No, it doesn't show that. The point of the computer models is not to predict exactly how bad the outbreak will be. What good would that do? All you have to do to find out how bad the outbreak will be is wait. What computer models do is give us some kind of idea of how seriously we should take the situation. For that, the models did a fine job. They probably shouldn't have been bandied about so much on the news, but that's not a problem with the science--that's a problem with the science reporting, which is a well known problem.
But it's really, really frustrating when people predict a possible bad outcome and suggest steps be taken to prevent it, and then steps are taken, and then the bad outcome doesn't happen, possibly because the steps were taken (it's never possible to know for sure) and then somebody says "you cried wolf." No. Crying wolf is when you lie about a threat you know doesn't exist. The Y2K threat wasn't crying wolf, and this wasn't crying wolf. What both things were were attempts to mitigate a very real risk the severity of which was uncertain. The fact that we didn't have a massive breakdown in 2000, and that we didn't have an Ebola pandemic, are both really good outcomes.
TWENTY THOUSAND PEOPLE died and you claim this as 'crying wolf'???
If the international community hadn't jumped on this, it could have been way, way, way worse.
-WolfWithoutAClause
"Gravity is only a theory, not a fact!"These kind of events often follow a power law for how bad they get. It's notoriously (maybe impossible) to guess the magnitude of a power law event in advance. Also, my guess is they are not able to accurately account for adaptation in behavior of normal people in the disease zone. That's the sort of factor that will almost by definition be under-predicted.
Play Command HQ online
But the modelers argue that this really wasn't a failure, because their predictions served
to increase their funding. So the process was 100% successful.
done precisely in order to encourage behavior that would change the inputs to the model.
Nobody looks at cigarettes today and says, "Gosh, nobody smokes anyway and death rates are coming down, there was no need for all that worry, smoke away!"
The whole point of the data in that case (and in this one) was to encourage the world to change behavior (i.e. alter the inputs) to ensure that the modeled outcome didn't occur.
To peer at the originally modeled outcome after the fact and say that it was "wrong" make no sense.
When we tell a kid, "finish high school or you're going to suffer!" and then they finish high school and don't suffer a decade down the road, we don't say "well, you didn't suffer after all, guess there was no point in you finishing high school!"
That would be silly. As is the idea that the modeling was wrong after the modeling itself led to a change in the behavior being modeled.
STOP . AMERICA . NOW
spend their days posting, "Why in the hell do we vaccinate against polio? It's a scam! After all, how many people do *you* know that have ever had it? Hmmmm? Case closed!"
STOP . AMERICA . NOW
I admit that this is off-topic.
I dissected a frog in school. I do not recall there being an entry hole large enough for a firecracker. I did not go on to study medicine or any icky biology field. But I have been curious for a long time (don't get me started on the raped ape thing) and have to ask. How, pray tell, does one actually go about sticking a firecracker into a frogs anus? Google is not forthcoming and I refuse to do an image search. This important, scientific (I believe), question has to be answered and we rely on you.
"So long and thanks for all the fish."
I really don't get how this was a failure; I'm damn sure the entire point of those simulations was to aid in fighting the disease and ensuring it didn't spread as far as it would otherwise.
To put it another way, imagine you are driving a car with a modified satnav; instead of telling you where to go, it tries to detect the route you're taking, and tell you the outcome of that choice. You come to a fork in the road, and it says you'll go right and get in a terrible accident, so with that in mind you turn left. You don't then say the satnav is faulty because you didn't crash your car!
It's not as if the CDC gets more funding if they terrify us with their nonsense about 'epidemics', is it?
...and first among children and young adults (5 - 24) are cars, with 33804 deaths in 2013 alone.
6510 of those being in the 15 - 24 years of age range.
Now... Maybe you're out there, in the night, prowling garages and parking lots, killing all those cars in their sleep in order to prevent further deaths.
Which would explain why you missed OP's point.
Which was that those "worst-case scenarios that mobilized international efforts" predicting "175,000 cases in Liberia by the end of 2014... [and] 1.4 million cases in Liberia and Sierra Leone" could result in a "crying wolf effect" when the next epidemic comes around.
Mit der Dummheit kämpfen Götter selbst vergebens
You really don't know how any of this works, do you? This is becoming more and more obvious with every post you make. Learned the difference between sea ice and land ice yet? :)
You really don't know how any of this works, do you? This is becoming more and more obvious with every post you make. Learned the difference between sea ice and land ice yet? :)
:D, why don't you show the good people my errors.
Just wondering if the models came with a prediction score or some measure of their accuracy. As with climate predictions, the untold story is that the models are no more accurate than their inputs and the validity of the theories used to create them. You might expect models to come with warnings, but they don't, at least nothing that gets transmitted to the public.
As the world embraces 'big data' and the modeling it spawns, this should be a bit of a lesson. The worry should be: how many times can models be used to 'cry wolf' before people start ignoring them?
"Consensus" in science is _always_ a political construct.
I'd like to buy your rock, Lisa.
But the modelers argue that this really wasn't a failure, because their predictions served as worst-case scenarios that mobilized international efforts.
Sure, this worked this time in everyone's favor. But what about the next epidemic? Let's say the modeling is better next time (which it should be) and it predicts another disaster. What then? People will look to the modeling on Ebola and say "it's not going to be that bad" and regard the next warning more lightly. This does nobody any good.
A good example of this is Hurricane Katrina. The Weather Channel makes every weather event look like the apocalypse because it's the Weather channel and they really only have one story they can run. They have to keep eyes on their channel to sell advertising time. So they exaggerate everything. And people become numb to the warnings - and look what that got us with Katrina.
No, it's always better to call things for what they are. I think they would be better off to say the modeling was off, call a failure a failure, and keep people's trust intact.
Weaselmancer
rediculous.
this shows just how bad an idea it is to put too much trust in computer models
What's this? What exactly did the output of their model harm?
If anything, it was a reality check reminding people who don't study the spread of disease just how bad things can get if something this harmful goes unchecked.
"I have no special gift, I am only passionately curious." - Albert Einstein
Makes me wonder if GL models are also wrong.
If you keep making Chicken Little the sky is falling predictions with computer models as your proof to bring about some desired political change. Then eventually you will lose all credibility and no one will listen to you anymore. You damage the reputation of the sciences and certainly of computer models when you do this. Knock it off.
Trust us with everything from the economy to your children!
"But the modelers argue that this really wasn't a failure, because their predictions served as worst-case scenarios that mobilized international efforts."
They apparently never heard of the story "The boy who cried wolf".
Also, it reminds me of this: http://www.getelastic.com/lisa-simpson-gets-why-correlation-does-not-imply-causation/
Basically, they're saying they're right when the prediction is right, and they're right when the prediction is wrong. They can't be proven wrong, thus. How easy (and self-serving) such predictions and claims are. One could just ignore it, except for the reasons mentioned above: do this kind of thing enough, and people won't believe you, EVEN if your prediction IS right.
The Ebola they modeled and the Ebola on the ground were not at all the Te same. The incubation period was longer, the death rate lower, and it took the survivors longer to become non-infectious after they recovered. All these made it spread much worse.
Red