NOAA Goes Live With New Forecasting Supercomputers

dcblogs writes "The National Oceanic and Atmospheric Administration (NOAA) Thursday switched on two new supercomputers that are expected to improve weather forecasting. The supercomputers are each 213 teraflops systems, running a Linux operating system on Intel processors. The U.S. is paying about $20 million a year to operate the leased systems. The NWS has a new hurricane model, Hurricane Weather Research and Forecasting (HWRF), which is 15% more accurate in day five of a forecast both for forecast track and intensity. That model is now operational and running on the new systems. In nine month, NWS expects to improve the resolution of the system from 27 kilometers to 13 kilometers. The European system, credited with doing a better job at predicting Sandy's path, is at 16 kilometers resolution. In June, the European forecasting agency said it had a deal to buy Cray systems capable of petascale performance."

Comments

How do you shitbirds like the ACA now?

[Anonymous Coward]

Cause we know most of you are young statists who loves them some Obama and ain't got jobs.

Well, no more full time jobs for you. Go get yourself two part time jobs and fuck off then.

http://www.huffingtonpost.com/2013/07/25/white-castle-obamacare-part-time-workers_n_3651751.html?ncid=edlinkusaolp00000003

Stupid libtards.

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

Oh and by the way, this is EXACTLY what we all predicted will start to happen.

Logic (do you shitbirds actualy know what LOGIC is?) tells us that the rest of what is poredicted will also be coming down the road.

Death panels.

Rationing.

Single payer.

Substandard care and shortages.

Fuck you libtard fuckity fucks.

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

Hmmm, isn't this what companies like WalMart have been doing for decades? Never enough hours to earn benefits? Yeah, blame the government.

Businesses are greedy and care not for their employees. This hasn't changed much since the first rulers who used slave labor (of course back then who else could you blame but the government?).

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

"Businesses are greedy"

Of course they are, no one is debating this. But here's the thing. All people are greedy; governments are greedy as well. Governments however are the only ones allowed to shoot you.

And your plan is to give more power - and money - to the government.

Yea, you are genius fucking material aren't you.

You stupid fuckity fuck.

Re:How do you shitbirds like the ACA now?

[K. S. Kyosuke]

All people are greedy

I'm not. But that might have something to do with the fact that I'm certified. Perhaps we've finally reached the stage that what's natural for the individual isn't healthy for the society, and vice versa.

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

Of course a generalization like 'all people are greedy' does not address the specific; indeed there are men who are not greedy for one reason or another.

But these are the exception, not the rule; and you cannot deny this.

As to the rest of your post, I honestly haven't got a fucking clue what you are talking about. Certified? Healthy for society? Sorry dude, you aren't making any actual point. Try again.

Re:How do you shitbirds like the ACA now?

[__aaltlg1547]

That's a serious defect in the design of US employment laws. Employee benefits should be pro-rated with hours worked.

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

People, I am happy to discuss this all day long, but you cannot defeat my arguments.

You say 'such and such should be blah blah' does not make it true. You must back up your assertions with facts and logic, you have done neither of these things, you have simply made a request, and a whiny GREEDY one at that.

Why should benefits be directly tied to hours worked by law? Are you not a free man able to agree or disagree to an employment arragement on your own? WHY - and by what right - can the state intervene in this relationship? They cannot, it is not their right if we are to remain free men.

Which I am.

You it would seem are not.

Re:How do you shitbirds like the ACA now?

[khallow]

I'm not.

That's why I don't use the word, greedy. It's just a negative connotation word for a class of interest. You might not be overly concerned with furthering your own interests, but that doesn't mean that you don't have interests which can cause just as much harm as greed can.

Perhaps we've finally reached the stage that what's natural for the individual isn't healthy for the society, and vice versa.

For example, this sentence indicates the possibility of unhealthy interests. What has changed so that individual interests are more unhealthy for society now than they've been in the past? I don't see it at all.

Instead, I see the opposite. In prehistoric times, tribes needed a great deal of cooperation from their members. Someone not pulling their load could kill other members of the tribe. That's no longer true. Now, we can pursue radically different beliefs or goals without causing a great deal of harm (unless, of course, the beliefs or goals are inherently harmful, like killing members of ethnic groups you dislike).

What I see nowadays is a society-level tendency to create public goods with the resulting tragedy of commons exploitation by the public, and then trying to stamp out this manifestation of individual self-interest rather than fixing or removing the public good that led to the problem in the first place.

Those people seem to have all sorts of problems with natural individual behavior. But those who anticipate human nature and account for that in their schemes don't have that kind of problem.

Re:How do you shitbirds like the ACA now?

[Applekid]

All people are greedy

I'm not. But that might have something to do with the fact that I'm certified. Perhaps we've finally reached the stage that what's natural for the individual isn't healthy for the society, and vice versa.

Sure, everyone says they aren't greedy and it's the other guy that's greedy, and that it's the other guy's greed that's the problem. That's right up there with "I'm not rich, the rich are only those with more money than me."

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

Oh and by the way, this is EXACTLY what we all predicted will start to happen.

Logic (do you shitbirds actualy know what LOGIC is?) tells us that the rest of what is poredicted will also be coming down the road.

Death panels.

Rationing.

Single payer.

Substandard care and shortages.

Fuck you libtard fuckity fucks.

nigger nigger nigger nigger nigger nigger nigger nigger

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

This is why Republicans will win 2014/2016! ^^^^^^^^ /sarcasm

Re:How do you shitbirds like the ACA now?

[Anonymous Coward]

[...] Death panels, rationing, single payer, substandard care and shortages [...]

I fail to see how any of this is worrisome, considering that it just increases the value of my stocks in Cigna, Aetna, etc.

We should buy them some windows

[Overzeetop]

I suspect if they removed the computers and installed windows in the offices of our local TV meteorologists we would get better short term forecasts. I've also decided that any precipitation forecast more than about 3-4 days out that doesn't involve a system as large as a hurricane is just a wild-ass guess*. Heck, even real time they're often wrong, the local guys are fond of reporting sunny all day while I'm actually looking outside at it raining.

*Well, unless you're in SoCal Mar-Dec, in which case "Sunny" is always the statistically correct answer, or Orlando/Daytona, where "It will rain at 3:45pm for 5 minutes" is always the statistically correct answer.

Re:We should buy them some windows

[Bud Light Lime]

Larger atmospheric features such as air masses and mid-latitude cyclones are more predictable than smaller features. Thunderstorms are much smaller and less predictable. Also, thunderstorms are driven by instability in the atmosphere. That is, if air is nudged upward, it will accelerate upwards. This occurs when warm (or hot) moist air is beneath cold air aloft. If there's a lot of cloud cover left over from thunderstorms the previous day, for example, that makes predicting thunderstorm chances the next day much more difficult. Predicting the behavior of large air masses is done with much more skill than smaller features such as thunderstorms.

Re:We should buy them some windows

[Anonymous Coward]

in other words, it seems straightforward to track a feature that already exists, and while tropical storms can last days to weeks, many thunderstorms can appear and dissipate within hours. Figuring out if the conditions are favorable for storm formation is a lot different than figuring out exactly where it will form.

Re:We should buy them some windows

[93 Escort Wagon]

In the US anyway, TV weather persons don't do their own forecasts - they rely on their local National Weather Service reports. And those are based on actual meteorologists looking at what the US's GFS says, as well as the European ECMWF, and combining it with their own experience regarding what the models tend to get right/wrong locally.

UW's professor Cliff Mass has written, many times, about the problems with US weather prediction. The computers they rely on are old and less powerful than the European ones - plus it's exacerbated by the US NWS having a broader mandate, so the computers they do have are used to run several other types of simulations in addition to the standard weather models.

This purchase is definitely good news.

Re:We should buy them some windows

[dkleinsc]

Why not just hire Al Sleet?
"Tonight's forecast: Dark. Continued dark throughout most of the evening, with some widely scattered light towards morning."

Re:We should buy them some windows

[PPH]

Or Chrome. There's a good weather app for Chrome.

Re:We should buy them some windows

[Anonymous Coward]

Many US TV meteorologists do their own forecasting. Both meteorologists I've met and friends who have taken jobs as broadcast meteorologists have said everyone they work with, including themselves, make their own forecasts. They all have degrees just the same as the NWS forecasters do.

I've heard this isn't the case in other parts of the country, and I have no knowledge to verify or deny it, but in the midwest, many of the broadcast meteorologists are doing their own forecasting.

Re:We should buy them some windows

[Fnord666]

Do you know what the difference is between climate and weather? Climate is what you expect, weather is what you get.

Re:We should buy them some windows

[wooferhound]

The weather guys around here are 60% right 90% of the time

Re:We should buy them some windows

[davester666]

Now they just need to lob a couple of satellites up in the sky to gather the data for this computer, and we'll be set.

Re:We should buy them some windows

[Anonymous Coward]

The county I grew up in 30 some years ago had a population of barely 100k, but had three different TV stations that have their own Doppler radar systems and their own meteorologists. They were fully trained as meteorologists and did their own forecasting, and as far as I can tell, that hasn't changed in the area. Their radar systems have been upgraded to have some new buzzword compliance, but in one case they still have the same guy that I had talked to decades ago about the work he did. Despite the availability of NEXRAD radar and NWS predictions, there still seem to be plenty of TV stations that have their own radar and make their own forecasts. YMMV, especially if you live in a more or less populated area.

Huh? Awful article summary...

[Bud Light Lime]

HWRF runs at a much finer grid spacing than 27 or 13 kilometers. As I recall, the grid spacing is around 3 km in the inner nest. This is done to explicitly simulate the convection at the inner core of a tropical cyclone. This nest moves with the storm, and is embedded within a much larger domain. The upgrade from 27 to 13 kilometers actually refers to the GFS model. It's a spectral model that has a global domain. Other models that are regional (including the outer domain of the HWRF) need to know the conditions at their lateral boundaries, so they know what's moving into the domain. In the US, they typically use the GFS for their boundary conditions. I'm actually very skeptical of the need for upgrading the resolution of the GFS. That may have a role in improving GFS forecasts, but there have been studies showing that the initial conditions of the GFS are the real problem. The atmosphere is a chaotic system; that is, two similar initial states will diverge over time to produce two very different outcomes. In a study where the GFS was initialized with ECMWF initial conditions, the performance of the GFS improved. Hurricanes are typically steered by large scale features, which aren't necessarily going to be simulated better by using a finer resolution. It also doesn't address the initial conditions problem. I'm in favor of throwing more computing power at meteorology, but I'm not convinced it will solve the problems with the GFS.

Re:Huh? Awful article summary...

[hwrfboy]

I'm actually an HWRF developer and you are correct that the summary was wrong. Our innermost domain is 3km, at a size of around 600x600km, intended to resolve the storm's inner core region (the area with the dangerous winds and, typically, largest rainfall). It is within a larger 1100x1100km 9km resolution domain, for resolving the nearby environment, and there is a gigantic 7500x7500 km, 27km resolution domain to resolve large-scale systems that drive the track. Also, the 3km resolution is not just needed to resolve convection: you need it to resolve some of the processes involved in intensity change, and in concentration of the wind maximum, such as double eyewalls, mesovortices, hot towers, and vorticity sheets. The GFS is our boundary condition, and part of our initial condition. We tried using ECMWF instead as an experiment, but that causes mixed results on track, and worse intensity. The intensity issues are likely due to their model's lack of skill at intensity prediction and primitive ocean model. (GFS has better hurricane intensity than ECMWF, despite having lower resolution!) ECMWF also has completely different physics and dynamics that ours, which results in larger shocks at the boundary.

You can see a better description of our model on our website:

http://www.emc.ncep.noaa.gov/index.php?branch=HWRF

and if you're interested in running HWRF yourself, you can do that too, though it will be another week or two before the new 2013 version is publically available. HWRF is an open source model, put out by the NOAA Developmental Testbed Center (DTC), which handles the public distribution and community support. (Support of HWRF installations in other countries' forecast centers is generally handled through the NOAA Environmental Modeling Center (EMC).) Here is the webpage for user support and downloads:

http://www.dtcenter.org/HurrWRF/users/overview/hwrf_overview.php

As for your point about improved resolution not helping the GFS, that's not true, especially in the case of hurricanes. The resolution of the GFS (~27km) is so low that it cannot even resolve the structure of most storms, let alone see the complex features involved in predicting intensity, rainfall or the finer points of track. When it can resolve the storm, such as with Superstorm Sandy, it has intensity skill competitive with regional models. The upcoming GFS upgrades to 13km and later 9km resolution (~2-4 years away) will allow the model to get a good idea of the basic structure of the storm, and start having real skill at predicting intensity, even for smaller storms. That, in turn, will help the HWRF and GFDL regional hurricane models improve their track and intensity prediction since they both rely on GFS for initial and boundary conditions.

Re:Huh? Awful article summary...

[Bud Light Lime]

Your point about the GFS is well-taken, but at present, I'd never use the GFS or ECMWF to forecast hurricane intensity. Yes, there's value in increasing the resolution of the model. But there's also a need to improve the data assimilation to produce better initial conditions (see this NOAA white paper). The conclusion was that there are gains to be made by bumping up the resolution, but that's only one of the recommended approaches to improving the GFS. Others included better data assimilation and improving parameterizations. Much of what the public hears has been focused on the resolution of the model. Yes, it does matter, but there are other considerations that are at least equally important.

Much of the criticism of the GFS with respect to Sandy has focused on the track forecast several days out. While increasing the resolution of the model could provide some improvement to the track forecasts, I would expect better initialization to have a larger role, especially at that forecast range. I'd believe bumping up the resolution would provide much better gains in the area of forecasting intensity.

Re: Huh? Awful article summary...

[Anonymous Coward]

Thanks for your comment which is far more accurate than the article. I am currently comparing WRF run with ERA-Interim boundary conditions to previous runs with NCEP and indeed the results are much improved even at coarse (100km) resolutions.

Re:Huh? Awful article summary...

[hwrfboy]

Your point about the GFS is well-taken, but at present, I'd never use the GFS or ECMWF to forecast hurricane intensity. Yes, there's value in increasing the resolution of the model. But there's also a need to improve the data assimilation to produce better initial conditions (see this NOAA white paper). The conclusion was that there are gains to be made by bumping up the resolution, but that's only one of the recommended approaches to improving the GFS. Others included better data assimilation and improving parameterizations. Much of what the public hears has been focused on the resolution of the model. Yes, it does matter, but there are other considerations that are at least equally important.

Much of the criticism of the GFS with respect to Sandy has focused on the track forecast several days out. While increasing the resolution of the model could provide some improvement to the track forecasts, I would expect better initialization to have a larger role, especially at that forecast range. I'd believe bumping up the resolution would provide much better gains in the area of forecasting intensity.

Actually, I agree that the main problem is the initial state, and the experiences with HWRF upgrades back that hypothesis. Contrary to what the (completely wrong) article summary states, there was absolutely no HWRF resolution upgrade this year. The upgrades were all data assimilation, model physics and model dynamics. (Everything except resolution and ocean.) Our upgrade to 3km resolution was last year, in May 2012. While last year's resolution upgrade did help, it had nowhere near the impact of this year's upgrade. The lesson we learned from that is that, although resolution is needed to resolve the storm, resolution alone has no meaningful impact unless the rest of the model can handle that high resolution.

I hope the GFS developers learn the same lesson, though with our manpower shortages, it may be another year before the 13km upgraded physics makes it into the model. There are also some major data assimilation upgrades (improved initial state) planned for the GFS, but those are 2-3 years away. Partly that is due to the manpower shortages, and partly due to the small size of the new WCOSS computer- we have to wait until 2016-2018 for the later phases of the WCOSS upgrade.

Forget Skynet...

[Anonymous Coward]

When the machines take over, they'll be led by a core of weather-predicting super computers.

Hurricanes, a strange game...

[Tyr07]

The only winning move is not to play.

Beaches

[Impy the Impiuos Imp]

Is their raw data available? I'd like a crack at it with my desktop computer.

Re:Beaches

[hwrfboy]

The summary is confusing two different models: HWRF and GFS. The HWRF model is a public model you can download and run, as long as you have ~20 GB of RAM free on your computer:

http://www.dtcenter.org/HurrWRF/users/overview/hwrf_overview.php

There is a public version of the GFS, but I'm not sure where. I'm mainly an HWRF developer.

Also, you can download GFS and HWRF forecasts in real-time (ie.: files less than 10 minutes after they're created by the operational NCEP WCOSS supercomputer) here:

GFS: ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.*/

You want the files named gfs.t??z.pgrb2f* - those are the forecast files every 1-6 hours at 0.5 degree resolution.

The HWRF real-time data is here:

HWRF: ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/hur/prod/hwrf.*/

The *.hwrfprs_* files contain model fields. The *prs_n* are the 3km domain, prs_m are combined 9&3km, prs_p are 27km, prs_i is 9km and prs_c are combined 27:9:3km. The track files are *.atcfunix for six-hourly, *.3hourly for three-hourly and *.htcf for experimental per-timestep (5 second) information.

You can also get archived track files from a three season retrospective test of the GFS and various HWRF configurations here:

http://www.emc.ncep.noaa.gov/HWRF/tracks/

Formats of the track files contained within are described well on JTWC's website (the equivalent of the NHC for everything not near mainland US):

http://www.usno.navy.mil/NOOC/nmfc-ph/RSS/jtwc/best_tracks/

Re:Beaches

[TheLoneGundam]

Yes, I believe you _can_ get a lot of the raw data via NOAA's site, it is I believe free for US citizens since we already paid for it through funding NOAA. Maybe you can use Google's new algorithm for detecting 100,000 features in an image on a single computer to make the model run on you desktop in usable time - remember, as one of the NOAA sites quotes a scientist: if your model doesn't run in faster than real time, you might as well just swivel your chair to look out the window (assuming I'm not being an insensitive clod who doesn't pay attention to the fact that you don't have a window).

power

[flok]

"The system uses so much power that its emissions directly influence the weather on all continents and mars."

Re:power

[rwise2112]

"The system uses so much power that its emissions directly influence the weather on all continents and mars."

The best way to predict the weather is to control it, obviously.

Re:power

[hwrfboy]

The best way to predict the weather is to control it, obviously.

Actually, that was attempted, and aborted due to diplomatic reasons. NOAA tried cloud seeding experiments in the 1960s-1970s attempting to weaken or destroy a tropical cyclone when it is out to sea. Unfortunately, the experiment usually failed, and occasionally the surviving hurricanes made landfall and did significant damage. When that happened, some countries suspected that the US was doing this secretly to develop weather weapons, so the project was shut down in the early 1980s to avoid the resulting public outcry and diplomatic incidents. Why should congress keep funding a failed experiment that causes diplomatic problems? You can read about this here:

http://www.aoml.noaa.gov/hrd/hrd_sub/stormfury_era.html

and Wikipedia has a good page with a lot more information:

http://en.wikipedia.org/wiki/Project_Stormfury

On a positive note, the project contributed to the formation of the present-day AOML Hurricane Research Division, which now has the invaluable Hurricane Hunter aircraft, as well as some hurricane modeling experts. They contribued a lot in the past few years to callibrating the HWRF model physics and dynamics to observations.

Re:power

[Anonymous Coward]

To make it clear, while diplomatic and political issues killed the project (more than once...), it ultimately stayed dead because we're pretty sure it was completely ineffective. Initially promising results ended up just being a fundamental misunderstanding of the eyewall replacement cycle of hurricanes. The amount of energy involved in a hurricane is just too large to be directly affected by human action (e.g. a nuclear bomb) and for things like raid seeding, there is just not enough supercooled water to force to rain.

Re: power

[Anonymous Coward]

japan has been doing successful cloud seeing for years ... i believe the usa also does it some, too

Re:power

[Anonymous Coward]

If only the postal service was as reliable as the weather service!

Re: power

[Anonymous Coward]

There is a big difference between "successful cloud seeding" in the sense of making a cloud rain, and using that to affect the path or intensity of a hurricane. In the latter case, there is a limit in how much you can make it rain and how little that will do to the storm.

Teraflop?

[Anonymous Coward]

Teraflop? GTFO.
Your geek card will be confiscated on your way out.

I miss the good old days of weather forecasting

[Anonymous Coward]

http://ed-thelen.org/comp-hist/mak-ILLIAC-IV.html

Anybody else know the actual ratio of weather vs military applications run on this machine?

This is America . . .

[Anonymous Coward]

Please use American measurements, like rods and furlongs for distance.

Re:This is America . . .

[hwrfboy]

Please use American measurements, like rods and furlongs for distance.

You will be happy to hear that we do. The official track forecast products for the US, including HWRF and GFS, put out intensity in knots and speed in nautical miles. The only SI units are the pressure, for which we use millibars. This format predates most of our developers, and originates back in the days of punch card machines. In fact, a set of hurricane track files is still known as a "deck file" (referring to decks of punch cards), with the numerical guidance being the "A Deck" and the best track (best known actual storm locations) being the "B deck". The result consists of a series of lines that look like this:

AL, 04, 2013072612, 03, HWRF, 048, 193N, 591W, 42, 1012, XX, 34, NEQ, 0073, 0000, 0000, 0074, 0, 0, 64

That's from the HWRF 48 hour forecast of Tropical Storm Dorian. The "42" is the wind intensity in knots, the 0073...0074 are the radius of 34 knot winds in nautical miles in four quadrants, and the "64" is the radius of maximum wind in nautical miles. There are a number of input products, such as the tcvitals, which are in a mix of SI units and knots or nautical miles. Nobody uses miles, rods, furlongs or hogsheads for anything related to tropical cyclones though (sorry).

The HWRF model is actually run by several other countries' forecast centers (Taiwan, India, Vietnam, and Oman last time I checked) and many researchers worldwide, and I'm sure you can imagine how annoying it is to keep converting its input and output between obscure American units and SI units.

Breaking: IRS wants out of ACA

[Anonymous Coward]

How about this for irony, and which further proves my point.

http://washingtonexaminer.com/irs-employee-union-we-dont-want-obamacare/article/2533520?custom_click=rss

The IRS, something like 95% of which are Democrats and vote statist, want nothing at all to do with Obamacare. I won't paste the details, you morons can barely read anyway.

Again, I am proven right.

We are fucked thanks to you shitbird greedy bastards who vote commie. Fuck you all with Carlos Dangers's stinky weenie.

Weather satellites needed for data

[cold fjord]

Supercomputers need data. The forecast for US weather satellites is partly cloudy.

Turbulence Ahead for Weather Satellites

JPSS

This won't last

[ulatekh]

These weather-predicting supercomputers will be shut down by the politicians as soon as they calculate that climate change has nothing to do with human activity, and everything to do with that massive hydrogen-fusion reactor only 93 million miles away.

Re: This won't last

[Anonymous Coward]

You are confusing weather and climate. We covered this on the first day of class. Please see me in office hours at your earliest convenience so we can identify and improve your misunderstanding of the course material.

Re:This won't last

[Anonymous Coward]

You mean just like how a program developed to track the path of a baseball seen on camera can be used to predict the winner of the World Series a season from now?

Got Data?

[Anonymous Coward]

Great, now all that NOAA needs now are some satellites to provide data. (Main US Weather Satellite Fails As Hurricane Season Looms)

$20M/year?

[markhahn]

someone is making a killing, I think. the purchase cost of these computers should be under $30M total, and less than $3M/year to run.

Re:$20M/year?

[hwrfboy]

Actually, the high cost per year is because there are several stages of planned upgrades, intended to support the steady increase in resolution and data assimilation capacity of the various models. (Including a massive GFS upgrade next year.) The project, from the NCEP side at least, was completed five weeks early and under budget. The estimated savings, from shutting down the old overpriced Power6/AIX CCS cluster early, is about $1 million, and the switch to Intel/Linux will save taxpayer dollars in the long term. I know that's small compared to the national debt, but it isn't the usual government waste that you hear about, and I'm proud to say we're doing our part (even if most of the government isn't).

As for who is getting the money, have you ever heard the old adage "nobody ever got fired for buying IBM?" While this is somewhat of a Stockholm syndrome situation, I'm told IBM did manage to underbid everyone else this time, and the cluster mostly working, five weeks early. (Completely working would have bene nice, but you get what you pay for.) We've used creativity to work around the problems and get everything working with the cluster they gave us.