Slashdot Mirror
NWS Announces Big Computer Upgrade
riverat1 writes "After being embarrassed when the Europeans did a better job forecasting Sandy than the National Weather Service Congress allocated $25 million ($23.7 after sequestration) in the Sandy relief bill for upgrades to forecasting and supercomputer resources. The NWS announced that their main forecasting computer will be upgraded from the current 213 TeraFlops to 2,600 TFlops by fiscal year 2015, over a twelve-fold increase. The upgrade is expected to increase the horizontal grid scale by a factor of 3 allowing more precise forecasting of local features of weather. The some of the allocated funds will also be used to hire some contract scientists to improve the forecast model physics and enhance the collection and assimilation of data."
maybe it would pay for the upgrade.
Well unless we're in a butterfly wing effect situation, you'll generate 3 times the amount of garbage.
I think this is more a subsidy to the troubled supercomputing market disguised as a technical improvement.
Why not just pay attention to the European forecasts, which would cost nothing?
"National Security is the chief cause of national insecurity." - Celine's First Law
Looks like they'll have to shut down most essential weather services. The People have to be taught a lesson about telling government to spend less money.
Because Europe was better? Why not:because we want to increase our quality regardless of what others are doing. Think about it: if the Europeans would not have been better, what you had would have been good enough. Or "We could predict the storms better and save potential lives, but who really cares? We are already the best in doing it. USA! USA! USA!"
Don't fight for your country, if your country does not fight for you.
Not that the 5 people in that field have trouble getting jobs anyway. But if you like math and the ocean, it's a good field to go into.
Your fantasies contain the seeds of important concepts.
... the Europeans did a better job forecasting Hurricane Sandy. Oh. Didn't know that. But hey when they make a movie of it, I'm sure they will present as fact that the American system was the most awesome thing and NWS was right on the money with typical awesome American ingenuity .... sorry, 'Argo' flashback.
"Consensus" in science is _always_ a political construct.
Hopefully this new system will be able to calculate in Celsius instead of Fahrenheit...
http://www.ecmwf.int/services/computing/overview/supercomputer_history.html
Europe: 70 TFLOPS by and upgrade to be finished by early 2013 (Sandy was in Oct 2012), which they say will make it about 3 times the power of the computer it replaces. i.e. 23 TFLOPS, they did a part upgrade during Sandy, to about 50 TFLOPS
USA: 213 TFLOPS, to be upgrade to 2,600 TFlops
So no, the Europeans did the prediction with 10%-20% of the supercomputing power, 2% of the proposed supercomputing power. This is just a subsidy to the Supercomputer industry (and indirectly USA chip makers), at a time when the PC market is tanking. It has nothing to do with the garbage the US produced, they just used a bad model.
"Replacement of the second cluster will be completed in early 2013. Each cluster has 768 POWER7-775 servers connected by the IBM Host Fabric Interface (HFI) interconnect."
"For the first time the processor clock frequency actually decreased, going from 4.7GHz to 3.83GHz, despite this each processor core has a theoretical peak performance 60% greater than that of the POWER6. For ECMWF's applications the system is about three times as powerful as the system it replaced.The first operational forecasts using this system were produced on 24 October 2012."
It appears that the computers that Europe was using for the "better forecast" were not as powerful as the old system being replaced. Upgrading because Europe's forecast better would be like taking a slow route to a holiday destination then buying a Porsche because your neighbours got there sooner when all you need is a new roadmap.
It's kind of astonishing how little we (by which I mean the U.S.A.) spend on weather forecasting relative to the economic effects. The economic costs of weather are in the hundreds of billions of dollars annually. You can't change the weather, but more accurate predictions will save more lives and property.
I try not to plan my life around the weather, but a few million to possibly offset billions in damage from an incorrect hurricane path prediction is a no-brainer.
I'm a bit confused ... why is so much money being spent if the technology already exists elsewhere? What about remote computing? Why can't we share resources? A 2.6TFlop super computer had better last us a long time. I can't imagine what the "1.21 Gigawatt" power bill will look like.
Here's a great chance to jump in on another multi-billion dollar government tech boondoggle. Why let SAIC and the other Beltway Bandits scarf up all the big bucks? A bunch of us ought to slap a shell company together and bid like there's no tomorrow. Get on board that gravy train while we can!
If this goes anything like recent FAA, USPS, and VA projects to name but a few, a successful contractor can bill for years while never delivering a finished, operational product.
Surely we can spec a 2.6K TFlop monster, with ancilliary systems, and market it to the GSA purse-holders. Easy math. Calculate the probable actual cost (fair bid price), triple it (IBM, Kray, or SAIC's price), and multiply by .9 = winning bid (never bid too low on a government contact; they automatically chuck out the highest and lowest).
What could possibly go wrong?
Scruting the inscrutable for over 50 years.
To detect butterfly effects they bracket the scenarios with a small delta and see if it swings off chaotically. If that happens, then they know they can't make a realistic prediction, because the sensors they have don't permit it. Adding more computing power doesn't fix anything then.
No, it was simply a little better algorithm run on a computer a tiny fraction of the processing power of the *current* US supercomputer. I notice there's a lot of government money going into supercomputers as the PC market dries up. I look at the use case and they're a joke. Similar to this one.
I think its just a case of pumping money into the US semiconductor industry and IBM at a time when they're struggling.
You don't need a weather man to know which way the wind blows.
Just a 2.6 pflop computer.
We need the focus back on public schools, affordable college education, respect for science, and good learning for EVERYONE who wants a good education.
This is how Government funding works. I was at a workshop on the then-new field of space weather forecasting in the mid 1990s where the keynote address was given by Dr. Joe Friday, at the time the head of the NWS. He pointed out that we would see no serious funding from Congress until there was the space-weather equivalent of a train wreck that kills many voters, or costs the monied interests lots of dinero. (Joe later lost his job when a non-forecastable flood in the mid-west that exceeded the 100-year flood levels wasn't correctly forecast. In this case, the solution was of the Shoot the Messenger variety since the real cause of the bad flooding was lousy planning by the Corps of Engineers.) The local government version of this is not putting a stoplight at a bad intersection until someone, preferably a cute child or pregnant mother, is killed there.
Posting anon to avoid burning bridges. NCAR has tried to develop better forecast models but they've layed off experienced US staff to hire foreign H1B grad students to write their software. I lost my 18+ yr position as a software engineer at NCAR, while helping to replace the 1980's crap they use to verify the accuracy their models with modern software, using modern techniques . They have great hardware but very amateur software. I got a "we've lost funding for you" while they were hiring H1B's. I was often the only US born software engineer in many of the projects I worked on at NCAR. The US could have much better forecasts, but the public wants everything on the cheap. The Europeans are doing better because they hire professionals to do development and charge for the output. IMO, American weather science is quickly becoming a joke.
Somehow it was necessary to mention that the budget was affected by sequestration?
I want to delete my account but Slashdot doesn't allow it.
You're talking as a meteorologist or computer programmer.
He's talking as 90% of the US public.
I think you need to be a little less a pompous ass and a little more understanding.
Supercomputing improvements are nice, but I personally want to see them get the cash to profoundly increase their NEXRAD backhaul (the data lines connecting their radar sites to the outside world).
Right now, they're HORRIBLY backhaul-constrained. I believe most/all NEXRAD sites only have 256kbps frame relay to upload raw data to NOAA's datacenter for further processing & distribution to end users. As a result, they're forced to throw away data at the radar site to trim it down to size, and send it via UDP with little/no modern forward error correction. That's a major reason why glitches are common. In theory, the full-resolution data is archived to tape on site and CAN be mailed in if some major weather event happens that might merit future study, but the majority of collected data gets archived to tape, then unceremoniously overwritten a few days later. And most of the tapes that DO get sent in sit in storage for weeks or months before finally getting added to their near-line data archive.
The low backhaul bandwidth is made worse by the fact that the secondary radar products (level 3 radar, plus the derived products like TVS) get derived on site, and wedged into the SAME bandwidth-constrained data stream. That's part of the reason why level 3 data lags by 6-15 minutes... they send the raw level 2 data, and interleave the previous scan's level 3 data into the bandwidth that's left over. I believe the situation with TDWR sites is even worse... I think THEY actually have a single ISDN line, which is why level 2 data from them isn't available to the public at all.
As I understand it, they can't use lossless compression for two reasons -- since they have no error correction for the UDP stream, a glitch would take out a MUCH bigger chunk of data (possibly ruining the remainder of the tilt's data), and the error correction would defeat the size savings from the compression. Apparently, the processors at the site are pretty slow (by modern computer standards), so it would also add significant delay to getting the data out. When you're tracking a tornado running across the countryside at 50-60mph, 30 seconds matters.
If NWS had funding to increase their backhaul to at least T-1 speeds, they could also tweak their scan strategies a bit to make them more useful to others. For example, they could do more frequent tilt-1 scans (the lowest level, which is the one that usually affects people the most directly), and almost immediately upgrade all current NEXRAD sites to have 1-minute updates for tilt 1 (adding about a minute to the time it takes to do a full volume scan, but putting data more immediately useful to end users out much more frequently).
Going a step further, more bandwidth would open the door to a fairly cheap upgrade to the radar arrays themselves... they could mount a second antenna back-to-back with the current one with fixed tilt (ideally at 10cm, like the main one, but possibly 5cm like TWDR if 10cm spectrum isn't available, or a second dish of the proper size for 10cm wouldn't fit), and do some moderate hardware and software tweaks that would effectively increase their tilt-1 scanrate to one every 6-10 seconds (because every full rotation of the main antenna would give them a full tilt-1 rotation off the back). This means they could send out raw tilt-1 data with 6-10 second frequency. It's not quite realtime, but it would be a HUGE improvement over what we have now.
Unfortunately, NWS has lots of bureaucracy, and a slow funding pipeline. I think it's safe to say that the explosion in popularity of personal radar apps, combined with mobile broadband, almost totally caught them by surprise. Ten years ago, very few people outside NWS were calling for large-scale NEXRAD upgrades. Now, with abundant Android and IOS apps & 5mbps+ mobile data the norm, demand is surging.
That said, I hope they DON'T squander a chunk of cash on public datafeed bandwidth instead of upgrading their backhaul. I'd rather see them do the back-end upgrades that only THEY can do, and tell people who want reliable & frequent upgrades to get their data feed through a private mirror service (like allisonhouse or caprockweather) who can upgrade their own backhaul as needed, instead of having to put in funding requests years in advance.
Cliff Mass, University of Washington Atmospheric Sciences Professor, has been arguing for an upgrade for a long time. He sees great potential for this new system if used right. The reasons for the upgrade boil down to having "huge economic and safety benefits" with better forecasting, and he says these benefits are within our reach.
On the other hand, zero celcius is the boundary line between dealing with frozen water (as ice or snow) and dealing with liquid water (as flooding or rain). That's incredibly convenient when travelling.
Not that I'm in favor of fahrenheit (I'm not) but water freezing at 32F isn't any more difficult to deal with than 0C. Both are arbitrarily chosen chosen scales. Celsius has the nice round numbers but from a practical day to day usage standpoint that matters not at all. I know that water freezes below 32F and that doesn't take up any more room in my brain than 0C. The only real problem is that I have to remember two scales instead of one. Since Celsius is the more widely used scale, I wish we would switch to that one.
One key thing you missed. The NWS 88-D Radar system *can* take a scan every minute at the expense of resolution and distancee. A "full" scan across the commonly used tilts takes *six* minutes. You can have a OC-3 to every radar site, but you're only going to get data ever six minutes most of the time.
I was led to this place, a place I can't understand. A place that demands my belief just as strongly as my disbelie
This upgrade in computing power is to move the US from 3DVAR to 4DVAR, however, it does nothing to improve the US weather models. This is interesting, in that 4DVAR can give worse results than 3DVAR, while using additional compute power. There was a nice paper written in this:
http://onlinelibrary.wiley.com/store/10.1256/qj.05.85/asset/200513161304_ftp.pdf
"The first proviso is that on other measures of analysis quality the conclusions are less clear cut. For instance, Fig. 5 shows that using evolved covariances gives worse fits for synoptic 4D-Var than 3D-Var with FGAT. Despite the 4D-Var schemes giving better forecasts overall, their analyses are not consistently better. An explanation may be that the evolved covariances used in 4D-Var (see appendix) increase the background error and hence the impact of observations in modes growing in the time-window, and that this improves the analysis and hence the forecast if these modes continue to grow. This effect was discussed by The paut et al. (1996) who showed that the evolved covariances implicit in 4D-Var are very similar to the singular vectors representing the fastest growing modes. This effect is strongest for longer evolutions; most of The paut et al.’s results are for 24 hours rather than the 3 hours in this paper. For a 6-hour window they see some decaying modes, which they attribute to discrepancies between the 3D- Var covariances and the PF dynamics which is used to evolve them. There are also probably biases in the simple PF model, which may distort the evolved covariances. For these decaying or distorted structures 3D-Var has more freedom to fit the observations, so 4D-Var analyses can be worse."
Which big government contractor needs work now? That seems to drive these projects more than actual need. I'm guessing the NWS/NOAA has plenty of computing resources, just need to fine tune the models a little bit and collaborate techniques with the Europeans...
The accuracy measurements in the article are meaningless by themselves. Does anybody know how those slight differences in accuracy translate into dollars saved? Furthermore, why can't we piggy-back on the European system? They run world-simulations after all and share the data.
I agree. And there's no evidence that it would be different under McCain or Romney or Clinton.
how long it will be before TWC tries to name it?
Wow. Informative as all get out. Thanks.
You're mostly right, but you're overlooking the software limits that exist mainly due to the limited bandwidth. If they upgraded the sites to a full T1 and tweaked the software a bit, they could give us new tilt-1 updates every minute, with about 15-60 seconds of radar-to-end-user latency, without major hardware upgrades besides the T1 interface itself.
Compare that to now, where we get only a single tilt-1 scan every 6 minutes, and that scan might itself be delayed by another 6-10 minutes on top of that. There are ALREADY several VCP programs that sample tilt 1 every minute... they just can't send out that data, and only use it locally for calculating their derived products, because they don't currently have the dedicated bandwidth to send it out.
Remember, WSR88D is kind of like an Atari 2600... it has very few limits that are truly "hard" and insurmountable. Rather, they're software-imposed in recognition of other limiting factors like backhaul bandwidth, or are precautionary limits imposed to guarantee that some specific product can always be fully-derived and delivered within some specific amount of time, or in a way that won't be destroyed by random errors. Many of them could be substantially improved with even minor hardware upgrades in other areas.
There are real limits to resolution imposed by scattering, wavelength, and particle size, but from what I've read, the current level 2 scan data is still throwing away about 30-50% of the nominal max resolution, and enormous amounts of theoretical resolution that could be recovered through oversampling. At this point, NWS doesn't even *know* what they could derive offsite from oversampled level 2 data, because they've never had the backhaul resources to even *fantasize* about streaming it in its full oversampled glory, or even archiving it all on site. 20 years ago, the idea of having 64 terabytes of on-site raid storage for Amazon/Google-like raw indiscriminate archiving would have been unthinkable, and never even entered into the equation.
The current scan rates are a compromise that tries to balance their backhaul against the need to track fast-moving storms like tornadoes. If they mounted a second, fixed-tilt dish back to back with the current dish so that every rotation produced a tilt-1 sample, they could alternate the back-facing samples between slow and fast pulse rates (so every other scan would be alternately optimized for range or resolution), and dedicate the front-facing dish currently in place to sampling the higher tilts (interleaving them to sample lower tilts twice at both PRF rates). Freed of the need to dedicate at least two full sweeps out of each volume scan to tilt 1 (because the back-facing antenna would sample tilt one every time the dish rotated), they could possibly slow down the rotation rate and use it to increase the resolution.
The closest thing I've seen to my idea was a paper someone at NOAA wrote about a year or two ago, proposing a compromise between fixed-tilt back-to-back conventional radar, and full-blown (and likely to be cost-prohibitive) phased-array radar 360-degree fixed radar. Basically, their idea was to build a limited wedge of PAR modules capable of sampling 4 tilts over ~1 degree horizontal, and mount it to the back side of the existing dish assembly, so that it could sample 4 tilts per revolution, and give us the equivalent resolution of 4-tilt level 3 TDWR every 12-15 seconds. The idea is that NOAA would then have a TDWR-resolution rapidly-updating radar source for tracking fast-moving/rapidly-developing storms off the back, and could slow down the overall rotation to get more detailed ultra-hires samples than we have now off the front dish.
The catch, from what I recall, was that they'd HAVE to decrease the RPM, and use 5.8GHz (like TDWR) for the rear array, because there just isn't enough C-band 10cm spectrum available to simultaneously broadcast 5 pulse beams without creating an interference scenario that would make their current range-folding issues look downright tame. They'd
Reliability is a *huge* issue.
Consumer-grade broadband is NOTORIOUSLY vulnerable to regional power outages... something that tends to happen simultaneously with bad storms. Imagine the outrage if South Florida lost its radar every time the outer rain bands of a hurricane started to knock out the local power grid, or if Oklahoma or Kansas lost their radar when an advancing squall line knocked out Comcast's power a half hour before the parade of tornadoes following it arrived?
Even if they had lower-res lower-bandwidth modes to fall back on, the fact remains that they can't have weather radar failing precisely when it's needed the most. They MUST have reliable broadband that doesn't depend upon commercial power, backed up with SLAs that have teeth and real penalties if the telco doesn't keep up its end of the deal.
So what do they use in practice?
AFAIK, 256kbps frame relay at WSR88-D sites, and 128kbps ISDN at TDWR sites. I believe they're now in the process of upgrading the TDWR sites to 256kbps frame-relay, and enabling 1-minute updates for tilt-1 data as they get the backhaul updates completed.
The big, huge, immediate improvement from backhaul upgrades is basically 1-minute updates for the lower tilt. I believe they're doing TDWR now, and hoping to use it as a demonstration of value to gain support for doing the same for the WSR88D sites "really soon". That said, I think the upgrades might have fallen victim to the budget sequester, because they haven't (officially, AFAIK) said a word about them in several months, even though they were all supposed to be coming online right about now. :-(
It would be nice if they'd also do something about the remote sensing infrastructure to get more data to these nice new supercomputers. My current understanding is that the Feds are getting increasingly weak in that department.
Credo sim. - I think I am.