NWS Announces Big Computer Upgrade

riverat1 writes "After being embarrassed when the Europeans did a better job forecasting Sandy than the National Weather Service Congress allocated $25 million ($23.7 after sequestration) in the Sandy relief bill for upgrades to forecasting and supercomputer resources. The NWS announced that their main forecasting computer will be upgraded from the current 213 TeraFlops to 2,600 TFlops by fiscal year 2015, over a twelve-fold increase. The upgrade is expected to increase the horizontal grid scale by a factor of 3 allowing more precise forecasting of local features of weather. The some of the allocated funds will also be used to hire some contract scientists to improve the forecast model physics and enhance the collection and assimilation of data."

Comments

so..

[msauve]

Why not just pay attention to the European forecasts, which would cost nothing?

No, Europe had 50 TFLOPS, 1/5th the USA

[Anonymous Coward]

http://www.ecmwf.int/services/computing/overview/supercomputer_history.html

Europe: 70 TFLOPS by and upgrade to be finished by early 2013 (Sandy was in Oct 2012), which they say will make it about 3 times the power of the computer it replaces. i.e. 23 TFLOPS, they did a part upgrade during Sandy, to about 50 TFLOPS

USA: 213 TFLOPS, to be upgrade to 2,600 TFlops

So no, the Europeans did the prediction with 10%-20% of the supercomputing power, 2% of the proposed supercomputing power. This is just a subsidy to the Supercomputer industry (and indirectly USA chip makers), at a time when the PC market is tanking. It has nothing to do with the garbage the US produced, they just used a bad model.

"Replacement of the second cluster will be completed in early 2013. Each cluster has 768 POWER7-775 servers connected by the IBM Host Fabric Interface (HFI) interconnect."

"For the first time the processor clock frequency actually decreased, going from 4.7GHz to 3.83GHz, despite this each processor core has a theoretical peak performance 60% greater than that of the POWER6. For ECMWF's applications the system is about three times as powerful as the system it replaced.The first operational forecasts using this system were produced on 24 October 2012."

Re:No, Europe had 50 TFLOPS, 1/5th the USA

[Anonymous Coward]

So much ignorance, I'm not sure where to start. First, I work in a public school. I've worked in private schools, my mother runs a (non-religious) private school. My wife has taught in other private and public schools. My daughter has an IEP.

Teacher's unions generally do not want more money thrown at the schools, depending on the state. There will be significant differences in the political games in public education depending on the state. In most "at-will employment" states, the teacher's union is mostly there for show. The district mostly controls the teacher's union. In states like this, the district administration pushes for more money to be thrown at "schools." This is because they control how the money is actually spent, and as a result, most of the money doesn't get to the school level.

There is a huge misdirection that most of the general public has fallen for in public education. The perception among the general public is that the "schools" are at fault. In reality, the district administration controls and dictates everything. A school principal has much less authority and autonomy than most people realize. This works out great for the district administration, because the school staff regularly become the scapegoat for failed district policies. In many states, counties, districts, cities, a school can do very little other than what district administration tells them to do. In effect, a school has all the accountability with none of the authority. Meanwhile, the district administration continues to make decisions in a vacuum while collecting paychecks that would make a seasoned IT Director blush.

I've been in countless IEP meetings, both as a parent and as a school administrator. Most IEP meetings are educators and parents. 4-5 school staff, 1-2 parents. The school staff are usually the various specialists (speech, OT, learning specialist, etc) and general ed teacher. I will usually be involved if there's some behavior concerns related to the IEP. I have never had a lawyer in an IEP meeting, other than a child advocate when there's social services involvement with a student. If a district needs lawyers at every IEP meeting, they're doing something very wrong. That would suggest the district is the problem, not the system itself.

One last note, socio-economic status has a larger impact on student success than most people want to admit. Districts don't want to talk about that because then they might lose the federal programs thanks to No Child Left Behind.

tl;dr It's not the individual public schools that are a problem, it's district administrations and school boards that have created huge bureaucratic structures and keep huge portions of money at the district level. This is really why private and charter schools generally do better with less money. They don't have the huge bureaucracy sucking up the money.

I would start looking at the algorithms

[Chrisq]

It appears that the computers that Europe was using for the "better forecast" were not as powerful as the old system being replaced. Upgrading because Europe's forecast better would be like taking a slow route to a holiday destination then buying a Porsche because your neighbours got there sooner when all you need is a new roadmap.

Now, can we please upgrade their NEXRAD backhaul?

[Miamicanes]

Supercomputing improvements are nice, but I personally want to see them get the cash to profoundly increase their NEXRAD backhaul (the data lines connecting their radar sites to the outside world).

Right now, they're HORRIBLY backhaul-constrained. I believe most/all NEXRAD sites only have 256kbps frame relay to upload raw data to NOAA's datacenter for further processing & distribution to end users. As a result, they're forced to throw away data at the radar site to trim it down to size, and send it via UDP with little/no modern forward error correction. That's a major reason why glitches are common. In theory, the full-resolution data is archived to tape on site and CAN be mailed in if some major weather event happens that might merit future study, but the majority of collected data gets archived to tape, then unceremoniously overwritten a few days later. And most of the tapes that DO get sent in sit in storage for weeks or months before finally getting added to their near-line data archive.

The low backhaul bandwidth is made worse by the fact that the secondary radar products (level 3 radar, plus the derived products like TVS) get derived on site, and wedged into the SAME bandwidth-constrained data stream. That's part of the reason why level 3 data lags by 6-15 minutes... they send the raw level 2 data, and interleave the previous scan's level 3 data into the bandwidth that's left over. I believe the situation with TDWR sites is even worse... I think THEY actually have a single ISDN line, which is why level 2 data from them isn't available to the public at all.

As I understand it, they can't use lossless compression for two reasons -- since they have no error correction for the UDP stream, a glitch would take out a MUCH bigger chunk of data (possibly ruining the remainder of the tilt's data), and the error correction would defeat the size savings from the compression. Apparently, the processors at the site are pretty slow (by modern computer standards), so it would also add significant delay to getting the data out. When you're tracking a tornado running across the countryside at 50-60mph, 30 seconds matters.

If NWS had funding to increase their backhaul to at least T-1 speeds, they could also tweak their scan strategies a bit to make them more useful to others. For example, they could do more frequent tilt-1 scans (the lowest level, which is the one that usually affects people the most directly), and almost immediately upgrade all current NEXRAD sites to have 1-minute updates for tilt 1 (adding about a minute to the time it takes to do a full volume scan, but putting data more immediately useful to end users out much more frequently).

Going a step further, more bandwidth would open the door to a fairly cheap upgrade to the radar arrays themselves... they could mount a second antenna back-to-back with the current one with fixed tilt (ideally at 10cm, like the main one, but possibly 5cm like TWDR if 10cm spectrum isn't available, or a second dish of the proper size for 10cm wouldn't fit), and do some moderate hardware and software tweaks that would effectively increase their tilt-1 scanrate to one every 6-10 seconds (because every full rotation of the main antenna would give them a full tilt-1 rotation off the back). This means they could send out raw tilt-1 data with 6-10 second frequency. It's not quite realtime, but it would be a HUGE improvement over what we have now.

Unfortunately, NWS has lots of bureaucracy, and a slow funding pipeline. I think it's safe to say that the explosion in popularity of personal radar apps, combined with mobile broadband, almost totally caught them by surprise. Ten years ago, very few people outside NWS were calling for large-scale NEXRAD upgrades. Now, with abundant Android and IOS apps & 5mbps+ mobile data the norm, demand is surging.

That said, I hope they DON'T squander a chunk of cash on public datafeed bandwidth instead of upgrading their backhaul. I'd rather see them do the back-end upgrades that only THEY can do, and tell people who want reliable & frequent upgrades to get their data feed through a private mirror service (like allisonhouse or caprockweather) who can upgrade their own backhaul as needed, instead of having to put in funding requests years in advance.