Perfect Weather on the Net

ctwxman writes "Here in the East, we're having a heck of a weekend. The snow has been flying, accompanied by wind, accompanied by people on TV saying, 'stay home.' I'm one of those people. I forecast the weather for a living. It's a great job... a magnificent physics puzzle, solvable with high level math and some acquired skill. And, there's a new puzzle every day, tenuously linked to the one you solved yesterday. When I started doing this over 20 years ago, the data to accomplish the task was tough to come by. I remember how excited I got when I first was able to get data at 300 baud on a request/reply basis (I guess we call that interactive now). Now, nearly everything you need to forecast the weather is here, on the net. Yes, there are a zillion sites that already have the finished forecast, but this is Slashdot. We don't need no stinkin' forecasters!" Read on for ctwxman's notes on do-it-yourself weather forecasting.

ctwxman continues:

I always like to start at NCEP (National Centers for Environmental Prediction) for a look at the dynamic models. Each is run using somewhat different equations, making them often come up with different, quite contrary solutions. Some of these models, like the GFS are worldwide in their coverage and forecast out an amazing 16 days (note: the word accurately was not used in the last sentence). Once the dynamic models are through, we can massage them against past performance under similar circumstances at specific places. These are the statistical models, referred to as MOS (Model Output Statistic) models. Again, there are somewhat different solutions from different models. If none of these work for you, run your own. There are programs available to allow you to run your own model, specifying the domain, grid spacing, time interval, etc. The most commonly used research model of this type is the MM5, produced at Penn State University. Run it on your PC! Of course, it's freely available and supported. Sometimes, the data you want already exists, but not in the form you'd like to see it. That's where software like GrADS comes in. Put out by the Institute of Global Atmosphere and Society's Center for Ocean-Land-Atmosphere Studies, GrADS claims to be an 'interactive desktop tool that is used for easy access, manipulation, and visualization of earth science data.' I agree with all except 'easy.' I run a version of GrADS on my server in order to produce localized forecast graphics like this that wouldn't otherwise be available. Yes, looking at satellite imagery and radar is a lot of fun... but the real fun is knowing what will be there before you look. And, astoundingly enough, we are significantly more accurate (and I get assaulted significantly less often at the grocery store) than even a few years ago.

Comments

Run your OWN weather station

[caferace]

I bet I'm not alone doing this. Decent weather stations are cheap, and it's simple enough to hook it into your box and keep statistical models on a running basis. Using wx200d and an Oregon Scientific WM-918 (sold under other names as well) is a great solution for less than USD $200.

Granted, it's not true forecasting, but you can easily add your data to aggregate with other users at Weather Underground and pull radar data from just about anywhere.

Me? I just like to know what's happening NOW, but it's also pretty handy to know what the temp is in your home "server room".

Tying all this historical data back into longer range forecasts would be fun. I've found TV forecasting to be pretty stale and inaccurate. How many of them have real meteorological degrees anyway?

Re:Run your OWN weather station

[The Dobber]

How many of them have real meteorological degrees anyway?

I ain't watchin the Weather Bunnies cause of thier metrological (sp) ability.

Anyone ever note the frequency of pregnancies on the Weather Channel. Place must be a god damn rabbit warren.

Re:Run your OWN weather station

[barzok]

How many of them have real meteorological degrees anyway?

Ever notice how some of the TV weather people are called "weathermen" while others are called "meteorologists"? Many meteoroglogists are also members of a professional organization (the name escapes me) and will display that organization's logo next to their name when they're on-screen.

Re:Run your OWN weather station

[gnu-generation-one]

Even if you don't want to run a climate model, try putting XPlanet on your background, and get the updated cloud images every 3 hours.

It's a great way to stay informed about what's happening weather-wise around you, and you'll be able to watch the weather-forecasts with an "I guessed as much", from being constantly aware of the cloud pictures.

Re:Run your OWN weather station

[534154414E]

Wow indeed, I routinely worry about the weather at home when traveling beyond Earth's atmosphere. ;)

Tom to Ground Control, come in. Over.

Ground Control here, what's problem, Tom? Over.

I was just looking at my home weather status and my A/C seems broken, again. Could you get a tech over there to check it out? Over.

Sorry, Tom, You've started chsshhhh chshhhshshhh -king up cshshhhhhhhhh...

Re:Run your OWN weather station

[AlexisMachine]

If you have a degree in Meteorology or Atmospheric Science (basically you took all the hard classes with Calculus) then you're a Meteorologist.

These days, you can get a broadcast Meteorolgy degree from Mississippi State or SUNY Albany, and become an weatherman without all the tough classes.

The logo you're talking about is the
AMS Seal of Approval.

Old wives tails

[rf0]

Red sky at night shepards delight. Red sky in the morning shepards warning.

Lot of truth in that saying

Rus

Re:Old wives tails

[Anonymous Coward]

I thought it was sailor, not shepard.

I guess you're closer to the sheep than the sea.

Re:Old wives tails

[Professor_Quail]

It has to do with the way fronts and pressure systems move...if there's a red sky at night, it means a high pressure front has moved in (means clear weather); a red sky at morning means just the opposite, an area of low pressure has moved in.

Here's a site with more on weather proverbs:
Weather Proverbs

Re:Old wives tails

[Anonymous Coward]

Didn't know wives had tails!!? I guess we must descend from the monkeys after all, then.

Re:Old wives tails

[Anonymous Coward]

The origin can be traced to the Bible (possibly before, but don't know of any reference):

Matthew 16:3:

He answered and said unto them, "When it is evening, ye say, 'It will be fair
weather: for the sky is red.'
And in the morning, 'It will be foul weather today: for the sky is red and
lowring.'"

Not only can the colour of the sky be affected by pressure fronts as pointed out in other places, but it can also indicate the movement of cold fronts. I suppose the 'sailor version' is more well known in the US because almost the entire population imigrated by sea... but the 'shepherd version' which is more well known in Europe, will pervade agricultural commuities more.

I got it...

[hookedup]

Look out the window.

A great job...

[wrinkledshirt]

It's a great job... a magnificent physics puzzle, solvable with high level math and some acquired skill.

Coupled with the fact that nobody expects weathermen to be right anyway, must be a great living.

Imagine a nuclear physicist saying, "You know, it's funny, but yesterday all indications were that today was going to be a smooth day for our reactor. How about that, eh?"

Re:A great job...

[thorpie]

"Trust me, i'm a weather forecaster" is my standard "you can trust me line". I still use it, but if forecasts get better I may have to change back to "I'm a used car salesman" or "I work for SCO" or the like.

Re:A great job...

[glrotate]

Alternatively tell them your an economist.

Re:A great job...

[GuidoDEV]

I've got an AthlonXP 2100+ and can run an 84-hour forecast using the MM5 in 2 1/2 hours. The domain is the continental US (plus some buffering around the CONUS, such as southern canada, parts of the atlantic/pacific, etc.), with a 40km gridpoint spacing and 30 vertical levels.

By contrast, the Eta model run by NCEP currently runs at 12km grid spacing with around 100 vertical levels (I believe), but the key is that the forecast from a 40km model run and a 12km model run usually differ very little, though the 12km run will have a little better resolution and may be a little more useful as a result. If the model runs have a resolution much better than 12km, however, you start running into problems where events like individual thunderstorms are explicitly resolved in the models' physics, as opposed to simply being "parameterized" (i.e., triggered in the model when certain larger-scale, favorable features are present). This sounds all well and good, but unfortunately the models are notoriously bad at explicitly resolving small-scale features like thunderstorms--much work needs to be done in this area.

In short, the average person can run their own weather model on their PC with no problems whatsoever, it no longer requires a supercomputer to do so. Good luck getting very skillful results beyond days 5-7 in your forecast, though.

Weather Stations?

[Xenothaulus]

Spend time outside. The more you do that, the more attuned you become to the weather and what it's going to do. Yes, I said outside. It's that place on the other side of the window by your desk.

Where's the raw data?

[Quixote]

I work in machine learning, and weather forecasting has always seemed like an interesting problem to explore. However, the lack of accurate raw data (temperature, dewpoints, pressure, humidity, precipitation, etc.) going back 30-40 years is a handicap. Where can one get such raw data, in a machine-readable format?

Also, I have been toying with the idea of writing a script to automatically grade the predictions put out by Wunderground and Weather.com, to see how accurate they are. It would be nice to see if it is really worth it to rely on their 5-day forecast.

Re:Where's the raw data?

[aemain]

NOAA's National Climatic Data Center has a lot of the raw data available, most of it for free (especially if you're coming from a .edu domain). Both US and international data.

In a related story...

[cr@ckwhore]

Meanwhile, weather forecasters around the world were puzzled Sunday as access to weather data over the internet crawled to a halt. A Whitehouse representative denied rumors of a terrorist attack on the weather infrastructure.

The mysterious cause of the delays in weather data are under investigation.

There are only 10 types of people

[Anonymous Coward]

those who understand METAR and TAF and those who don't...

e.g.:
(Terminal Airport Forecast of J-F Kennedy Airport)
2002/11/17 09:14
KJFK 170914Z 170913 03021G32KT 6SM -RA BR OVC012
TEMPO 0913 3SM RA BR OVC008
FM1400 02013G20KT 5SM -RA BR OVC012
FM2100 36011KT 5SM -RA BR OVC020
FM0400 34009KT 5SM -RA BR OVC015

Re:There are only 10 types of people

[nicklott]

Sorry, have to plug my own site here... :)

I know a website that does it all for you

Re:There are only 10 types of people

[AlphaOne]

BZZT. Try again.

On the 17th, at 09:14 GMT, winds were from 030 at 21 knots, gusting to 32 knots. Visibility was six statute miles with light rain and mist. The sky was overcast at 1200 feet.

Temporarily, for less than two hours, between 0900 to 1300 GMT, visibility will drop to 3 statue miles with moderate rain and mist and the ceiling will drop to 800 feet overcast.

From 14:00 to 21:00 GMT, winds will shift to be from 020 at 13 knots, gusting to 20 knots. Visibility will rise to 5 statute miles with light rain and mist. Sky will be overcast at 1200 feet. ... etc. They're actually easy to read if you do it often enough. I can usually decode METAR faster than I can read an ordinary plain-english forecast.

Programing for dummies

[KillerCow]

Yes, there are a zillion sites that already have the finished forecast, but this is Slashdot. We don't need no stinkin' forecasters!

Find a problem that has already been solved, and re-solve it.

On this note, but something completely different..

[cshuttle]

Since I was real little, I'd always been interested in the weather and forecasting and all that stuff. Now that toasters and other such standalone devices are becoming so prevalent, I'm looking for the simplest thing that apparently doesn't exist:

How about an 802.11 weather station?

I'm just looking for something that sits outside my house, collects weather data and other such simple stuff, and relays that data back to a server to build a web page with or whatever.

True, there are some devices like this available, but they all require a dedicated machine to log the data, and some really hard work to make them operate properly over a network (why would I possibly want a 1-wire data transmission solution, or even phone-line communication when I've got 802.11 right here?)

Have I simply missed the magic google search that has the toaster I'm looking for?

Re:If Only it was right!

[no reason to be here]

Here in Seattle the weather forcasts are wrong more than they are right.

How the Hell can you screw up a weather forecast for Seattle?

"Tomorrow's weather forecast: 'Rain'"

how about ... Environment Canada?

[Anonymous Coward]

Oh, I don't know... how about Environment Canada? You know, the place the weather network and every other forecasting service in the country gets their data...

weatheroffice.ec.gc.ca

Re:That's great for the US

[girouette]

The Meteorological Service of Canada has a web page at:
http://weatheroffice.ec.gc.ca/canada_e.html . The going was a little rough there for a while (we don't have the resources of the big media outlets to make it look pretty) but it is getting better all the time. It is also the place where the information is guaranteed to be up to date.

Watch out about the images under "weather charts". They are still the large, old-style monochromatic images. (That will change, eventually, to smaller colour images).

Lookit

[UtilityFog]

One of the best pattern-detection and analysis systems out there is still the human visual cortex -- watch the radar for your area consistently and you'll soon start getting a feeling for what is going to happen next. Wunderground's regional is about the right size.

Re:This is silly.

[azaris]

"It's a great job... a magnificent physics puzzle, solvable with high level math and some acquired skill."

I hate to cast a damp towel on this, but personally, I find this is to be silly self-promotional drivel. What Weather forecasters won't tell you is that anything beyond a 3-5 day forecast is just a guess.

So what? There are a lot of fields in science where most of our knowledge is about making accurate guesses and then seeing how well they fit the reality. It's not just about weather, the same tools can then be used on other complicated dynamic systems.

Their accuracy rates beyond this period go down below 50%; which means that flipping a coin is more accurate.

Really? I'd hardly call predicting the weather a simple case of true-or-false. Otherwise they could simply always give the exact opposite result that the complicated computer simulation gives and arrive at over 50% accuracy.

Though there are some places (like Ireland, I've heard tell) where simply predicting rain every day will be correct 80% of the time.

You are better off saving your CPU cycles for something more valuable, like Primenet (www.mersenne.org), IMHO.

Computer scientists and their everlasting silly infatuation with primes... There are other important research areas, you know.

Re:On this note, but something completely differen

[RevMike]

How about an 802.11 weather station?

I'm just looking for something that sits outside my house, collects weather data and other such simple stuff, and relays that data back to a server to build a web page with or whatever.

I run a Davis VantagePro Weather station. It relays data from a sensor cluster up on my roof to a console in my computer room via 900MHz radio. Then Davis' WeatherLink software submists it to weatherunderground and to my own website.

What about orange sky at 11:00 AM?

[Skim123]

Here's a picture from my street at 11:00 AM from back in late October. Orangish-brownish, although I think this had more to do with the San Diego wildfires burning 10 miles away moreso than weather fronts. ;-)

Re:Lorenz Equations

[Stranger4U]

Actually, Lorenz's first paper on the subject "Deterministic Nonperiodic Flow" was motivated by computer simulations he ran using a set of three highly coupled, non-linear differential equations that represented a simple weather model. The solutions to those equations will yield the very standard butterfly-wing graph, as well as exhibit sensitive dependence. It was on the basis of this model that Lorenz several years later coined the famous "butterfly effect" term. So, it was initially a weather model, and that's what my post referred to.

You know what they say about...

[Anonymous Coward]

weathermen and economists. They can have long successful careers without ever being right.

Re:If Only it was right!

[gurustu]

Seattle (and West Coast) accuracy is always going to be lower than you'd like (and lower than the East Coast's, for example) simply as a result of fewer observations.

The rest of the nation gets to see weather data from thousands of weather stations as patterns move from West to East, while Seattle gets its (relatively) skimpy data from satellites and scattered ocean stations.

Here a recipe for a cheap weather station.

[smartin]

1. One Wire Weather station from AAG.
   
2. Excellent free software to run it.
   
3. gnuplot to plot the results.
   
4. Post the information to The Weather Underground
   

Re:Here a recipe for a cheap weather station.

[citking]

Weather Underground is an amazing site, full of historical information, severe weather reports, and scientific explanations of the forecast. If you are a regular visitor to the site, you'll notice that the people behind the site are always updating and tweaking things to work better and to provide more features. For $5/year you get the ad-free version (or just grab Firebird if you are stingy) along with unlimited radar access and other goodies.

I've begun using their site over weather.com because the radars on The Weather Channel's site are not updated as often as Weather Underground's, and their site seems to be completely built around advertising, making it hard to find severe weather information and damage reports.

One last thing I like about Weather Underground is its speed. Radars load quickly, my favorites are stored easily, and (at least for subscribers) the layout is impeccable, utilizing a Google-like whitespace rather than cramming sensory overload everywhere.

Kudos to the team at Weather Underground!

The low tech wind gauge, very reliable.

[Bob Bitchen]

wind sock

Long Term Forecasting

[sjbrown]

Here are some links for the Earth Simulator -- an attempt to model the entirety of the planet's weather systems.

Official Site

Short Blurb from Time Magazine


Descriptive Article (with pictures)

Details Regarding its Supercomputer Status

Re:The Fraud is finally exposed...

[Shadwell]

You could have at least credited Lewis Black with this one.

"So what's the weather like today John?"

"Uh...nice. Back to you."

Close, but no cigar

[Jerry]

As Dr Lorenz demonstrated in 1963, long range weather forecasting to any degree of accuracy is doomed to failure, even using several different mathematical models. Even accurate predictions no farther out than 5 days are limited to wind directions, barometric pressures, and the high and low temperatures. Precipitation, like thunderstorms, is 'predicted' in terms of percent probability that your area will get rained on, but even 0% or 100% predictions have often failed. Five years ago, on Oct 14th, none of the 14 inches of heavy wet snow we received was predicted. It downed over 50,000 trees in the city. Our award winning meteorologist spends about half of each broadcast, following a weather event, apologizing for mis-forecasting the previous day's weather. As Dr. Lorenz pointed out, the various runs of the model 'look similar' but that's not the same as a prediction, nor proof that the weather will even obey the model, regardless of what the model says.

Weather prediction is a standing joke. You are use to it. I am used to it. That's about best that can be done, despite all the high powered computers, mathematical models and and their theories. That's the nature of Chaos. Even when a thunderstorm is raging in the next county and heading in your direction there is no model that will predict if and when it will arrive. The 'meteorologists' at most TV stations use composite radar to 'predict' where storms are heading and when they will get there, and they make their predictions only minutes before hand, not hours or days, weeks or months ahead. I find that I can do exactly the same for the Lincoln area, with exactly the same accuracy, using the Omaha composite radar at www.crh.noaa.gov/radar/loop/DS.p37cr/si.koax.shtml

The best predictor for bad weather on the NOAA website I gave is the one-hour rain loop. But even when it shows a steadily advancing area of wetness, the "Great Wall of Lincoln" has unpredictable effects in diverting or suppressing rainfall. Ditto for snow and tornadoes.

The really arrogant folks are those who use models to predict global weather 50 years from now, even when they limit their 'predictions' to general high temperature 'averages' for regions like North America or Africa. Such dire 'Global Warming' predictions are fueled not by valid math models, because none exist, but by their political agenda. Those kinds of 'predictions' can only be classed as flagrant propaganda, and people willing to fabricate 'scientific' evidence for their political agendas scare me, just as much as folks who pass laws destroying my Constitutional Rights, while claiming to protect those freedoms from the actions of terrorists. They are from the same mold.

Here is a nice java applet demonstrating the Lorenz Attractor.
http://www.exploratorium.edu/complexit y/java/loren z.html