Statistical Accuracy of Internet Weather Forecasts

markmcb writes "Brandon Hansen considers the statistical accuracy of popular on-line weather forecast sources and shows who's on target, and on who you probably shouldn't rely. Motivated by a trip to a water park that was spoiled with hail despite a 'clear sky' forecast, he does a nice job of depicting deviations, averages, and overall accuracy in a manner that stats junkies are sure to love."

Comments

The more the merrier?

[Reverse Gear]

What a nice piece of work.

I can't help but smile a bit that MSN weather in this test turns out to be the absolutely worst when it comes to accuracy in almost all categories.

I would think there is a lot of difference on how the forecasts are made in the different channels, some of them probably do get a lot of their information from meteorologist working on their own stations. I wouldn't wonder if MSN doesn't have a meteorologist (or maybe only one) working to provide their forecasts.
Computers and simulations play a big role in predicting the weather today, but human eyes are worth a lot still.

I don't myself live in the USA, so my primary use of these are to check on when there is severe weather in areas where I know someone.
I have gotten used to check on weather underground for this information, I haven't checked on many other weather channel, but I feel quite well capable of following what is going on in the USA with tornadoes and such here from Denmark.

For a long time we only had one weather forecast service here in Denmark, a national institute. Since a primarily private TV station (TV2) a few years ago started their own weather forecast service, I really feel the national institute have been pulling themselves together and have provided many services that they didn't provide until now. So even though some of the services provides terrible accuracy they might still serve a good purpose in giving the other services competition and thereby forcing them to improve also.

When I am really dependent on the forecast I tend to study the information behind the simple prediction of the given weather, that way I am also much better prepared for possible scenarios, knowing which front move where and can better "read the skies".

Re:The more the merrier?

[Gabrill]

It seams odd to me that he started the project because of rain, and then completely ignored rain in his observations. Otherwise, the study was very cool.

Re:The more the merrier?

[Hoi Polloi]

That's where I go for my weather online. Let's face it, all of the other weather services depend on NOAA for their base data anyway so you might as well go to the source. I do find that the NOAA predictions tend to err on the conservative side though (always predicting a chance of precipitation on dry weeks, etc).

Re:The more the merrier?

[AceGopher]

Just a note, MSN doesn't have any meteorologists on staff. The weather forecasts at weather.msn.com are provided by The Weather Channel.

-Ace

NOAA/NWS

[zoward]

I use the NWS website, mostly because I hate all the annoying flash ads on most of the other sites. I was also under the impression that most of these sites get their information from the NWS and pass it along to you (along with a bucket of ads). There was a lot of complaining amongst the popular weather sites when the NWS opened its own web site.

Re:NOAA/NWS

[limecat4eva]

I'll second that. Personally, I love the 48 hour graphs for being clean, simple, and easy to understand. And quick to load.

Re:NOAA/NWS

[istartedi]

Amen to that. NWS is one example where a government funded program actuallly WORKS. Clean, simple, Flash-free. It's been reliable enough for me. I was able to successfully plan for an outside project almost a week in advance with their long-range forecasts. You have to learn how to use these things a bit, based on your area. For example, here in DC during Summer, it may or may not rain in the afternoon, and nobody can predict if it will actually rain on a particular spot. That's because most of the rain comes from brief thunderstorms that pop up. In winter, they can tell you if a snowstorm will be nearby, but not if it will actually snow or how much. OTOH, sunny vs. rainy and general temperature predictions work pretty well. As an experienced user, I've learned to recognize which types of weather systems are predictable, and which aren't (e.g., Alberta Clipper -- easily predictable temperature drop vs. Gulf low snowstorm--extremely difficult to get the snow total in advance). Other areas have their own peculiarities too I'm sure.

What is your source?

[Lord Satri]

(I work at the Canadian Meteorological Centre, but I am not a meteorologist myself)

One thing that struck me is the 'abnormal diversity' of weather information sources. In Canada, weather models are computed in one place, a ~1000 processors computer in a basement which does only one thing: forecasting weather (the constant real-world observations that are ingested are used to adjust the models). Only one 'real' source (of course, there's the american, british, french, etc. official forecasting models to which we compare 'scores' on a daily basis). However, there's plenty of other canadian websites which will give you weather forecasts (one example). From what I know, these "other websites" have a significantly smaller workforce of meteorologists to interpret the models results than the Meteorological Service of Canada (the CMC is part of the MSC). That's why I would favor the 'original' source instead of a 'second-hand' source. I must however admit, commercial online sources of weather forecasting sometimes offer value-added products, such as the number of ski trails opened, offer general weather information capsules, etc.

And by the way, the official Environment Canada weather website is the most visited website in Canada (or at least, that's what they tell us, the employees! :-).

Re:What is your source?

[CmdrGravy]

Within half an hour ! That's pretty gosh darn impressive work there sir !

If you wouldn't mind going into a bit more depth I'd love to hear the details of how you knock up these forecasts, at the moment I can only predict the weather for definite maybe 20 - 25 minutes ( using basic optical observations ) in advance and I'd love to shave off those extra 5 minutes but I wonder if it's worth the cost of investing in something like a PC. I've seen portable weather stations you can install in your living room from hippy shops, do you think these would be suitable ?

Reliable forcasting method...

[AmIAnAi]

I remember some years ago a radio presenter saying that you could achieve greater accuracy than supposed weather forcasters simply by using the assertion: today's weather will be the same as yesterday. Have we moved on from this position?

Re:Reliable forcasting method...

[Bob(TM)]

In the forecasting vernacular, that method describes forecasting based on "persistence".

Persistence is the yardstick all forecasters use to determine if they should find another line of work (or be asked to do so by others). If you can't demonstrate an understanding of the processes and data such that extend beyond the data source everyone else has (ie., the weather their experiencing), it's just snake oil.

(IAAM)

Re:Reliable forcasting method...

[Anonymous Coward]

persistence is 75% reliable.

On any significant sample, weather reports were never worse than this.

Currently, models are able to make 85% or a little more accuracy.

This may sound paltry, but where this really works out is in longer term forecasts. At 75% you are probably wrong at 3 days forecast. Even if you take the assumption that forecasts are independent from day to day, 85% means you are probablt wrong after 5 days.

The extra two days you can predict for is what the money is going towards.

Statistics don't lie Statisticians do!

[wesborgmandvm]

Kudos to this guy for the work he put into the effort but it is really comparing apples and oranges. A forecast is a time sensitive product. You can't look at the forecast provided on day x from two different sources and compare them unless the forecast was provided at the same time of day.

The National Weather Service collects all the weather data used by forecasters, they also provide the 1st forecast. AccuWeather and others take the National Weather Service forecast then watch the new data (using National Weather Service provided data) to offer a refined forecast a few hours latter. Who do you think is going to be the most accurate the guy who provides the first forecast or the guy who waits for more data and then refines the for cast? AccuWeather's has statistics that show they are more accurate then the National Weather Service but if you used the AccuWeather forecast then waited for the next National Weather Service update I bet National Weather Service would be more accurate.

I am surprised that this guy used the weather.com and not the National Weather Service for the actually temp for all his calculations. (It doesn't matter b/c I am sure weather.com is right from National Weather Service data). He did point out that AccuWeather is the only one who provides forecasts > 10 days in advance.

My preference for weather forecasts is:

National Weather Service
AccuWeather (easy to understand graphics and 2 week forecasts)
The Weather Underground (Years ago they were the1st to provided free access to hurricane computer models)

Re:Statistics don't lie Statisticians do!

[Overzeetop]

Actually, you're only considering the accuracy of the scientific forecast. That is essentially useless to the end user - what they (I) want is accuracy of the reported forecast. The difference? I don't care when the actual prediction was made, I want to know that when I look at the forecast, it is likely to be correct. A very accurate forecast that is only updated once every three days is not nearly as useful as a farily accurate forcast updated every ten minutes. The former would be the best by your yardstick, but wouldn't necessarily help determine if the likilihood of a hail storm was high for this afternoon as much as the latter.

Can we believe the forecasts?

[gavink42]

Personally, I take weather forecasts with a couple of grains of salt.

However, the last cold blast that came through Memphis was forcast almost a week ahead of time. Weather radar of the middle part of the country showed about 90% clear of storms. So, I had a hard time with that one.

To my surprise (and right on time), down came the blast of cold air. Soon after was the promised snow/ice.

It still seems like an inexact science... with a touch of art and a pinch of luck thrown in for good measure.

Re:Can we believe the forecasts?

[nomadic]

To my surprise (and right on time), down came the blast of cold air. Soon after was the promised snow/ice.

I remember when a year and a half ago one of the hurricanes was in the Gulf of Mexico, heading almost straight west, and the meteorologists all insisted that the hurricane would make a complete 180 degree turn and head back east and smack into Florida. I didn't believe them. On more than one occasion I publicly stated that this was ridiculous.

I ate a lot of canned food that week.

Re:Can we believe the forecasts?

[Daniel Dvorkin]

What I find funny is that the entire practice of weather prediction is based on a logically fallacy. They take the data from previous years and say, ok, last time conditions looked like this x happened, so we predict x will happen again. Anybody who's taken an introductory logic class knows that you can't correlation does not equal causation.

And anyone whose understanding of correlation goes beyond "an introductory logic class" knows that in fact, as long as you're very careful about what you're doing, you can in fact very often use observed correlations to make valid predictions.

There's this whole field of study called "statistics," see. Not the "X% of people surveyed believe Y" type of thing you hear on the news, but an actual science, grounded in rigorous mathematical theory and growing more sophisticated all the time at producing useful knowledge from mountains of data. People get PhD's in it and stuff. Really. Maybe you ought to read about it some time. Maybe even take a class.

Or perhaps you'd rather remain secure in your prejudices, repeating "correlation does not equal causation" like a mantra, snickering at people whose knowledge you choose not to understand.

Re:rain?

[Wordplay]

You'd need a lot of data for that one, if you want to establish the accuracy of the probabilities. Unless I'm missing a more mathematically clever way to do it, I'd assume you'd require lots of 20% days to determine whether rain happened on anything near 20% of them. Similarly, 10, 30, 40, 50, etc would require their own groups of lots of days.

It'd be even trickier in, say, the SF Bay Area, where it only rains for two or three months a year, and then almost every day. Your 0% and 80-100% groups would be well-stocked, but not so much the other ones.

Whom

[mdsolar]

A preposition is awkward to end a sentence with. But, "whom" is the word "on" is followed by.
--
Solar follows the rules for grammer. http://mdsolar.blogspot.com/2007/01/slashdot-users -selling-solar.html

Re:Whom

[spun]

This is an outrage up with which we will not put.
-- Winston Churchill, on the practice of rearranging sentences to make sure they don't end in a preposition.

It is interesting to note the origin of such practices. Nothing in English forbids ending with a proposition. In Victorian England, the educated middle classes invented grammatical shibboleths to differentiate themselves from the uneducated lower classes. To do so, they took rules from Latin that had never before applied to English.

Rules such as this one and not splitting an infinitive were not originally part of the English language. They were invented for elitist reasons. People who insist on them should in their nuts be kicked.

Interpretation of the models is everything

[Aliks]

I am not a meteorologist, but I have worked with them a few times.

Generally the competing weather models will show a range of possible outcomes with various probabilities. You can average across all scenarios and come up with a 60% probability of rain, but the more days out you go more the scenarios diverge, so the less useful a single average will be.

Most people would not find it useful to hear that "there will be probably be thunder on Wednesday if it remains hot enough, but if it cools down on Tuesday then the thunderstorm will be off to the north somewhere"

Additionally, a lot of weather conditions are influenced by thin layers of cloud high up, so thin that precise measurements are critical so precise forecasts in one location more than 3 days out are difficult.

Possibly the best slashdotted error message ever.

[sammy baby]

"Total Freaking Database Error!"

Best 500 error I've ever seen. (Although I'm not sure it actually sent a 500.)

You can't fool mother nature

[TheHornedOne]

I work for Mother Nature; So I am really getting a kick out of most of these replies. Some of you guys are very good at making it sound like you know what you are talking about. But trust me.... You don't. I think you just want to make yourself sound smart, when in reality you don't know what you are talking about. This is how bad info gets passed around. If you dont know about the topic....Dont make yourself sound like you do. Cos some slashdotters believe anything they hear."

/wrong metasite
//slashies
//dont' kill me

Source Integrity

[wasted]

It always seems a bit odd to me, that when you model the weather, you need to model the entire world, and all the different models get their data from the same sources. Why then, don't we have a world weather computer? What I mean, is combining all these different computer resources into one huge model? I know that each nation does its own little tweaks to produce the "best" model, but surely the ability to throw even more machines at such a problem would produce better results? Are we heading this way, or is their just too much prestige for a country to work out its own weather?

One reason for countries to maintain their own weather forecast agency is to ensure the integrity of the data. This ensures that a country isn't receiving tainted data, or denied data. Models could be skewed to favor accuracy in one country over another, giving that country agricultural and energy trading competitive advantages. During many conflicts, countries where the conflicts occur cease dissemination of weather data so that the opposing force can't use the data. The US DoD maintains its own weather forecasting computers to ensure that access can't be denied, even if there is an NWS outage. If a country maintains its own systems, data integrity isn't in question.

A reason to use multiple models is that each model has different strengths. One model may tend toward forecasting precipitation over the midwest more often than it is likely to occur, and another may tend to forecast precipitation less often then actual. By using both models, we can get a better idea of the actual weather. In this case, if both forecast dry, it would likely be dry, and if both forecast precipitation, we would expect precipitation, and if they split, the forecasters would have to go back to old time forecasting techniques and get the coin and dartboard out. (Just kidding about the coin and dartboard. They'd really have to unfold their broaches, hats, and Pterodactyls, and start using the charts for what they were intended.)

My experience with the BBC weather forecast...

[OneSmartFellow]

for my area is that they are usually accurate down to a period of about 3 hours. As an example.

http://www.bbc.co.uk/weather/24hr.shtml?world=41 79 predicted that it would be snowing yesterday morning by 0600. Sure enough, I woke at six, and it was snowing. I awoke earlier in the night (about 0400) and it hadn't yet started.
Similarly, that site predicted that the snow would drop off by noon, and turn to sleet or rain by 1600. Again, this prediction came true, within an hour of the predicted time.

Generally speaking I find the BBC weather site to be accurate significantly more often than not (guesstimate 80% accuracy) with the 24 hour forecast being almost universally correct, and the 5 day forecast being the least reliable. (as expected)

This is a FAR cry from the weather predictions when I was a lad. Then the weather forecast on TV was simply a way to poke fun at the meteorologist, who clearly was doing the best he could, but invariably got it wrong.

Obligatory Stats Joke

[frank249]

Nice article but the sample only uses an 'n' of 14 days. I would have more confidence in the means, standard deviations and correlations if the author had used a bigger 'n'. For in stats, as in ethics, the n's do justify the means.

Large metro areas

[Anonymous Coward]

Geographically large metro areas -- especially those with hills or large bodies of water -- make a weather forecaster's job all the more difficult. The chance of rain may be higher on one end of town, but it's difficult for a TV or radio announcer (or a newspaper spread, for that matter) to present the distinctions clearly and quickly.

Too long ago, when I was an undergraduate taking Meteorology, we visited the weather department in a Twin Cities (MN) television station. The anchor on duty was pretty blunt: if there's a 100% chance of rain on one end of town and a 10% chance on the other end, the broadcast would distill that as a 55% chance of rain. He argued that it was the best his department could offer given the commercial realities of limited airtime and the mandate to serve the entire metro area.

ForecastWatch and ForecastAdvisor

[AceGopher]

I was quite curious about weather forecast accuracy as well. So three years ago I started collecting weather forecasts from the primary providers (Accuweather, Weather Channel, NWS, CustomWeather, Intellicast, etc.) and comparing them to actual observations. It's tougher than you might imagine, and there are a lot of factors that need to go into creating usable verification statistics.

I have a public site with some statistics for about 800 locations in the US available at ForecastAdvisor.com. There is also a blog with more in depth analysis (like how do temperature forecasts fare relative to how deviant the actual temperature is...in other words how well do forecasts do the further away from normal the actual is, and how to forecasts fare the further out they forecast for, and how does forecast accuracy compare over time.).

ForecastWatch.com is used by meteorologists and professionals. Accuweather, The Weather Channel, and several private meteorological companies use this system to help them understand and improve their weather forecasts.

And a geek note: ForecastWatch.com runs on Quixote (a Python web framework), while ForecastAdvisor.com runs on Ruby on Rails. The back-end forecast and actual collection, and calculations are Python with a MySQL database. Both sites are close to migrating to Django, a new Python web framework and ORM.

-Ace