Perfect Weather on the Net

ctwxman writes "Here in the East, we're having a heck of a weekend. The snow has been flying, accompanied by wind, accompanied by people on TV saying, 'stay home.' I'm one of those people. I forecast the weather for a living. It's a great job... a magnificent physics puzzle, solvable with high level math and some acquired skill. And, there's a new puzzle every day, tenuously linked to the one you solved yesterday. When I started doing this over 20 years ago, the data to accomplish the task was tough to come by. I remember how excited I got when I first was able to get data at 300 baud on a request/reply basis (I guess we call that interactive now). Now, nearly everything you need to forecast the weather is here, on the net. Yes, there are a zillion sites that already have the finished forecast, but this is Slashdot. We don't need no stinkin' forecasters!" Read on for ctwxman's notes on do-it-yourself weather forecasting.

ctwxman continues:

I always like to start at NCEP (National Centers for Environmental Prediction) for a look at the dynamic models. Each is run using somewhat different equations, making them often come up with different, quite contrary solutions. Some of these models, like the GFS are worldwide in their coverage and forecast out an amazing 16 days (note: the word accurately was not used in the last sentence). Once the dynamic models are through, we can massage them against past performance under similar circumstances at specific places. These are the statistical models, referred to as MOS (Model Output Statistic) models. Again, there are somewhat different solutions from different models. If none of these work for you, run your own. There are programs available to allow you to run your own model, specifying the domain, grid spacing, time interval, etc. The most commonly used research model of this type is the MM5, produced at Penn State University. Run it on your PC! Of course, it's freely available and supported. Sometimes, the data you want already exists, but not in the form you'd like to see it. That's where software like GrADS comes in. Put out by the Institute of Global Atmosphere and Society's Center for Ocean-Land-Atmosphere Studies, GrADS claims to be an 'interactive desktop tool that is used for easy access, manipulation, and visualization of earth science data.' I agree with all except 'easy.' I run a version of GrADS on my server in order to produce localized forecast graphics like this that wouldn't otherwise be available. Yes, looking at satellite imagery and radar is a lot of fun... but the real fun is knowing what will be there before you look. And, astoundingly enough, we are significantly more accurate (and I get assaulted significantly less often at the grocery store) than even a few years ago.

Comments

Run your OWN weather station

[caferace]

I bet I'm not alone doing this. Decent weather stations are cheap, and it's simple enough to hook it into your box and keep statistical models on a running basis. Using wx200d and an Oregon Scientific WM-918 (sold under other names as well) is a great solution for less than USD $200.

Granted, it's not true forecasting, but you can easily add your data to aggregate with other users at Weather Underground and pull radar data from just about anywhere.

Me? I just like to know what's happening NOW, but it's also pretty handy to know what the temp is in your home "server room".

Tying all this historical data back into longer range forecasts would be fun. I've found TV forecasting to be pretty stale and inaccurate. How many of them have real meteorological degrees anyway?

Old wives tails

[rf0]

Red sky at night shepards delight. Red sky in the morning shepards warning.

Lot of truth in that saying

Rus

Where's the raw data?

[Quixote]

I work in machine learning, and weather forecasting has always seemed like an interesting problem to explore. However, the lack of accurate raw data (temperature, dewpoints, pressure, humidity, precipitation, etc.) going back 30-40 years is a handicap. Where can one get such raw data, in a machine-readable format?

Also, I have been toying with the idea of writing a script to automatically grade the predictions put out by Wunderground and Weather.com, to see how accurate they are. It would be nice to see if it is really worth it to rely on their 5-day forecast.

Not sure they've gotten more accurate....

[analog_line]

I'm in the Beverly/Salem area and at least the numbers I've been seeing for accumulation are all way too low. They're saying 7-14 inches total for the weekend on NECN for our area, and I personally walked through at least 7 inches of new undrifted snow that has fallen between 2:30 AM and 12:00PM. There's two foot of fall out there if there's an inch, and the snow plows STILL can't keep up with it. 4 foot plus drifts. There was 3 feet of snow on two of the three doors, and the other only a foot and a half.

7-14 inches overnight, I can believe. For the whole storm is utterly ridiculous. Don't know where these people are getting their figures, but someone around here isn't looking out the window, all I have to say.

On this note, but something completely different..

[cshuttle]

Since I was real little, I'd always been interested in the weather and forecasting and all that stuff. Now that toasters and other such standalone devices are becoming so prevalent, I'm looking for the simplest thing that apparently doesn't exist:

How about an 802.11 weather station?

I'm just looking for something that sits outside my house, collects weather data and other such simple stuff, and relays that data back to a server to build a web page with or whatever.

True, there are some devices like this available, but they all require a dedicated machine to log the data, and some really hard work to make them operate properly over a network (why would I possibly want a 1-wire data transmission solution, or even phone-line communication when I've got 802.11 right here?)

Have I simply missed the magic google search that has the toaster I'm looking for?

Lorenz Equations

[Stranger4U]

I wonder what relationship (if any) exists between current weather models and the ones created by Lorenz back in the '60s. Those simple equations can produce some very chaotic behavior, and were the influence for the infamous "butterfly effect."

This is silly.

[Anonymous Coward]

"It's a great job... a magnificent physics puzzle, solvable with high level math and some acquired skill."

I hate to cast a damp towel on this, but personally, I find this is to be silly self-promotional drivel. What Weather forecasters won't tell you is that anything beyond a 3-5 day forecast is just a guess. Their accuracy rates beyond this period go down below 50%; which means that flipping a coin is more accurate. What's more, there have been little changes in improving this accuracy over the past 30-40 years. Most of the improvments have been in the under 5 day forecast. This is despite a great addition in technology (like satellites and computers).

Here's one study which shows this: http://www.nwas.org/ej/rose/verify.htm

There are others. This has been known for decades, but is generally kept quiet.

So the next time you see a Weather report on the news telling how it's going to be 3+ days out, mostly it's just a guess.

You are better off saving your CPU cycles for something more valuable, like Primenet (www.mersenne.org), IMHO.

Re:That's great for the US

[Anonymous Coward]

In Canada the only organisation doing weather prediction is Environment Canada (www.ec.gc.ca). Everyone else, your favourite country station raving about their ACCU-WEATHER and SUPERFORCAST are... just reading the forecasts from Environment canada. Nobody has their own radar, it's all EC's.

Re:There are only 10 types of people

[thedogcow]

I do!

Time : 1709Z
Winds: 21 knots from 030 gusts to 32knots
Visibilty: 6 Statute miels
Weather: Rain showers
Clouds: overcast at 1200 feet
And so on.

Re:If Only it was right!

[gurustu]

Seattle (and West Coast) accuracy is always going to be lower than you'd like (and lower than the East Coast's, for example) simply as a result of fewer observations.

The rest of the nation gets to see weather data from thousands of weather stations as patterns move from West to East, while Seattle gets its (relatively) skimpy data from satellites and scattered ocean stations.

Very True

[waldoj]

When hiking the Appalachian Trail, knowing what the weather would be like for at least the next 12 hours or so was as easy and unconscious as knowing how I felt and how I would be feeling in the near future.

Now that I'm off the trail, that skill is all but gone, unfortunately. It just takes a few days out of doors, though, and I start to pick it up again.

-Waldo Jaquith

Re:A great job...

[Anonymous Coward]

IANAWM, but my understanding is that accurate forecasting is all about the quality of your data, the quality of your models, and how many times and how far out you can run them, accounting for typical local variations.

The guys who report the weather forecast on TV and radio don't. They're basing their forecasts on the basic atmospheric data and doppler radar. Which is not bad in and of itself, but it's only part of the picture. Forecasting like that is by nature limited in its ability to be accurate. Especially as you get more than a few hours out.

There are models and technology capable of pretty amazing forecasting. But it's too expensive for everyday weather. That's what NOAA uses to track and forecast hurricanes--the stuff that can wreak catastrophic damage. But the side effect is that the hydrologists who work with these models in coastal areas and river basins also have access to the most accurate and distant weather predictions. But it's an NOAA policy to limit its information dispersal so as not to provide competition to business. Their mandate is to provide services which protect the U.S. interests and populace.

It's far beyond what the media has time or interest in providing. Besides, the most important part of weather forecasting is to know in advance of potentially dangerous weather, and that message gets out when it exists.

Incidentally, here outside Boston I've just cleared ~26" of snow from my--thankfully short--driveway. More where it drifted, predominantly around the cars, of course.

APRS: Also a Mobile Weather Data network

[ivi]

I didn't notice any posts on APRS, the Amateur Radio based position reporting system, which has been used for years to connect & exchange data between mobile, fixed & Internet based sites, including weather data.

The advantage of APRS, even for unlicensed receive-only users, is reception range that by far exceeds that of Wireless Networks (other than, say, 1-way satellite-based Internet connections).

APRS data flows both over VHF/UHF repeaters and from one (RF-based & licensed, ie transmit-capable) APRS station to another.

An APRS shareware (from the UK) that
handles weather data - as well as messages
& position data - is here:

http://www.UI-View.com

There is an excellent introductory site & White Paper on APRS here:

http://vk6.aprs.net.au

You can grab the white paper from here:

http://vk6.aprs.net.au/ukaprswp.pdf

The APRS creator's intro, et al. is here:

http://web.usna.navy.mil/~bruninga/aprs.html

Now that an Amateur Radio license is easier
than ever to get, we should all have one (&
some RF-based voice comms going as we surf,
ie the way that doesn't eat into our 'net
bandwidth, ie via Ham Radio...

with an APRS-connected weather station on
the side, and - when we're mobile (on land,
in the air or on sea - a GPS connected,
so we can be tracked/contacted & even found
in the unlikely even that we get lost. ;-)

Oh, and the speech-enabled Linux-based
GPS-driven, "moving-map" program from Austria - GpsDrive - will help preclude our getting lost
in the first place. (AFAIK, it's not APRS-
enabled... yet, but it lets [WiFi-connected]
near neighbors see each other's positions,
I understand).

Re:There are only 10 types of people

[AlphaOne]

BZZT. Try again.

On the 17th, at 09:14 GMT, winds were from 030 at 21 knots, gusting to 32 knots. Visibility was six statute miles with light rain and mist. The sky was overcast at 1200 feet.

Temporarily, for less than two hours, between 0900 to 1300 GMT, visibility will drop to 3 statue miles with moderate rain and mist and the ceiling will drop to 800 feet overcast.

From 14:00 to 21:00 GMT, winds will shift to be from 020 at 13 knots, gusting to 20 knots. Visibility will rise to 5 statute miles with light rain and mist. Sky will be overcast at 1200 feet. ... etc. They're actually easy to read if you do it often enough. I can usually decode METAR faster than I can read an ordinary plain-english forecast.

Re:A great job...

[GuidoDEV]

I've got an AthlonXP 2100+ and can run an 84-hour forecast using the MM5 in 2 1/2 hours. The domain is the continental US (plus some buffering around the CONUS, such as southern canada, parts of the atlantic/pacific, etc.), with a 40km gridpoint spacing and 30 vertical levels.

By contrast, the Eta model run by NCEP currently runs at 12km grid spacing with around 100 vertical levels (I believe), but the key is that the forecast from a 40km model run and a 12km model run usually differ very little, though the 12km run will have a little better resolution and may be a little more useful as a result. If the model runs have a resolution much better than 12km, however, you start running into problems where events like individual thunderstorms are explicitly resolved in the models' physics, as opposed to simply being "parameterized" (i.e., triggered in the model when certain larger-scale, favorable features are present). This sounds all well and good, but unfortunately the models are notoriously bad at explicitly resolving small-scale features like thunderstorms--much work needs to be done in this area.

In short, the average person can run their own weather model on their PC with no problems whatsoever, it no longer requires a supercomputer to do so. Good luck getting very skillful results beyond days 5-7 in your forecast, though.

Weather in Australia

[Foddrick]

For those of us in Australia, the Bureau Of Meteorology have an excellent site, with radar, rainfall, river conditions, forecasts and whole lot of useful weather information.

Re:OK, here's a question for mister weatherman...

[Anonymous Coward]

Good question. Since NWP models are deterministic in nature, for any gridpoint and for a period of time, it will either rain or not, i.e., the models forecast either 0 or 100% probability of precipitation. So where does "30% of rain" come from? The answer is MOS -- Model Output Statistics. By comparing historical observations (e.g. METARs [and yes I can read those and TAFs and TWEBs, too!]) with what the model actually predicted during those times, scientists have formulated regression equations that correct NWP model biases, e.g., model forecast too much light rain beyond 48hrs; max temperatures too high with thin snow cover, etc, etc. In correcting model biases the scientists have come up with regression
equations that look at various model output values and crank out PoP values for thousands of sites around the U.S as often as the model runs per day. Currently there are three flavors of MOS in the U.S. for the forecasters to choose from: Eta, NGM, and GFS.

Hope that answers your question . . .

Close, but no cigar

[Jerry]

As Dr Lorenz demonstrated in 1963, long range weather forecasting to any degree of accuracy is doomed to failure, even using several different mathematical models. Even accurate predictions no farther out than 5 days are limited to wind directions, barometric pressures, and the high and low temperatures. Precipitation, like thunderstorms, is 'predicted' in terms of percent probability that your area will get rained on, but even 0% or 100% predictions have often failed. Five years ago, on Oct 14th, none of the 14 inches of heavy wet snow we received was predicted. It downed over 50,000 trees in the city. Our award winning meteorologist spends about half of each broadcast, following a weather event, apologizing for mis-forecasting the previous day's weather. As Dr. Lorenz pointed out, the various runs of the model 'look similar' but that's not the same as a prediction, nor proof that the weather will even obey the model, regardless of what the model says.

Weather prediction is a standing joke. You are use to it. I am used to it. That's about best that can be done, despite all the high powered computers, mathematical models and and their theories. That's the nature of Chaos. Even when a thunderstorm is raging in the next county and heading in your direction there is no model that will predict if and when it will arrive. The 'meteorologists' at most TV stations use composite radar to 'predict' where storms are heading and when they will get there, and they make their predictions only minutes before hand, not hours or days, weeks or months ahead. I find that I can do exactly the same for the Lincoln area, with exactly the same accuracy, using the Omaha composite radar at www.crh.noaa.gov/radar/loop/DS.p37cr/si.koax.shtml

The best predictor for bad weather on the NOAA website I gave is the one-hour rain loop. But even when it shows a steadily advancing area of wetness, the "Great Wall of Lincoln" has unpredictable effects in diverting or suppressing rainfall. Ditto for snow and tornadoes.

The really arrogant folks are those who use models to predict global weather 50 years from now, even when they limit their 'predictions' to general high temperature 'averages' for regions like North America or Africa. Such dire 'Global Warming' predictions are fueled not by valid math models, because none exist, but by their political agenda. Those kinds of 'predictions' can only be classed as flagrant propaganda, and people willing to fabricate 'scientific' evidence for their political agendas scare me, just as much as folks who pass laws destroying my Constitutional Rights, while claiming to protect those freedoms from the actions of terrorists. They are from the same mold.

Here is a nice java applet demonstrating the Lorenz Attractor.
http://www.exploratorium.edu/complexit y/java/loren z.html