I see your point and agree with most of it, but...
GRIB and METAR are not "obscure". They are well-documented international industry standards that serve their intended purpose quite well. Is a tar file "obscure"?
GRIB data was designed as a means to store and exchange weather model output data in its (almost) rawest form. Because it is in binary form, it is only natural that it needs to be post-processed to be visualized or used in other applications.
For those who might be interested : a description of the GRIB format.
I've used cooker for almost all of this development cycle, because I wanted to try out kernel 2.6. Mdk 9.2 did ship with a pre-release 2.6 kernel, but I wanted fully upgradable kernel packages.
For the most part the experience has been positive, and there was no severe breakage. But you have to be willing to put up with occasional annoyances. The KDE desktop was a little broken for a while (no drive icons), but I think it's ok now. Also, it's kind of hard to find reliable cooker mirrors. I've had to use European sites even though I am in North America.
My only persisting complaint is that Konqueror gets random segmentation faults (about once in 25 tries) when I click on a link on any given site. Could be my hardware, although I don't get the same trouble with Mozilla.
The computing context is similar where I work (Meteorological Service of Canada). There is a big difference between the forecaster's (technical) desktop and the clerical desktop. Our technical desktop never belonged to MS WIndows. Cost remains a concern, obviously, which is why we are moving away from proprietary Unix.
For what it's worth, our next-generation workstation is going to be java-based. (Joint effort with a handful of European countries led by Germany.) http://www.dwd.de/de/Technik/Projekte/NinJo/ (German language)
Dunno about the NWS, but for our clerical desktop to move away from Microsoft would be nothing short of miraculous.
There have been efforts toward using XML in meteorological codes, but so far they have been somewhat scattered and isolated. The US Navy gave it a fairly extensive shot at http://zowie.metnet.navy.mil/~spawar/JMV-TNG/XML/O MF.html
The World Meteorological Organization is ultimately the standards body for international meteorological codes; you may want to have a look at sections 4.2 through 4.6 of this document to get a feel for the way XML is perceived by the WMO (not negatively, but as a complement to already well-established codes): http://www.wmo.ch/web/www/WDM/reports/FWI S-2001.ht ml
That may be too much information if you only intend to build a quick demo... but the overall background may still interest you.
I encourage you to use the comment form on the site. I can vouch for the fact that user comments are read and taken into account. The site's maintainers actually have a sincere concern for accessibility.
The Meteorological Service of Canada has a web page at: http://weatheroffice.ec.gc.ca/canada_e.html . The going was a little rough there for a while (we don't have the resources of the big media outlets to make it look pretty) but it is getting better all the time. It is also the place where the information is guaranteed to be up to date.
Watch out about the images under "weather charts". They are still the large, old-style monochromatic images. (That will change, eventually, to smaller colour images).
With the intense competition and peer review in the field of weather and climate forecasting, model validation and comparison is a constant concern. I also happen to know researchers in the field who would go to great lengths to validate their results anyway, because they take pride in doing their work well and they want to get to the truth.
In the case of a model used for the daily forecast, there are archives and canned cases that can be used for comparison.
In the case of a climate model, you are interested in getting the long term trends right. Storms that last from a couple of days to a week are a high-frequency signal in terms of climate; you are not trying to reproduce every shortwave peak. So the materials you use for validation come either from historical records, if they are available, or from geological records and ice from ancient icecaps. If you can reproduce known historical trends, then you may have a model with some predictive value. Your comment about cheating has value for short term forecast model development, but misses the point of a climate model.
Yes, it is a difficult set of problems, with scales going from the molecular to the planetary. But all of it is based on sound physical science done by smart folks. There aren't many mysterious areas left; the challenge lies in the implementation and validation.
The point of using larger computers is so that fewer aspects have to be neglected, and to have more sophisticated representations for the phenomena we do handle.
You are right, the forecasting of individual hurricanes or storms is completely besides the point of a climate model.
The application here is in the area of climate forecasting, attempting to forecast trends in upcoming decades. It's not even important whether the model gets individual storms right, as long as the averages are realistic.
The advance is in becoming able to incorporate hurricanes in the simulation. This should help improve the realism of those trends and averages.
Each run of the model only offers one solution (called a deterministic forecast).
There is a technique called ensemble forecasting, whereby you run multiple instances of the model with slightly disturbed initial conditions and/or slightly tweaked model parameters. You can then examine the statistics of the ensemble to try and obtain information a deterministic forecast might not be able to give you.
Note that the goal in this particular case is not hurricane forecasting as such. The newsworthy information is that this is the first time that a climate model can be run at a resolution high enough that hurricanes become possible within the simulation. Short term models used for the daily weather forecast do this reasonably well already.
a more accurate method would be to extrapolate from the amount of water actually present in a cloud
Actually we don't know the method the meteorologist in the article used. I would have liked to know. But I think it's pretty clear she was not estimating all of the moisture in the cloud, just condensed water in suspension.
Even direct measurement methods, like taking a volumetric scan with a radar, would have a fairly large margin of error. But I think the point of the exercise, from the meteorologist's point of view, was to go through a fun "back of the envelope" calculation. The mass of clouds, as such, is never a direct consideration in day-to-day meteorology. Making estimates of precipitable water is the closest real-world calculation in this respect.
clouds... they're just concentrations of moisture that happen to refract and reflect visible light
The word 'cloud' applies to a concentration of water droplets or ice crystals, not to "any concentration of moisture". Atmospheric water vapour is invisible, and while it is an important parameter to meteorologists, vapour does not equal cloud.
So while we're at it, why not just weigh the air?
Weighing the air is not such a useless or purely theoretical activity. Measuring the air pressure with a barometer does exactly that. You need that kind of measurement, among other things, to calibrate aircraft altimeters and to analyse weather systems.
For weather modelling, I can perhaps envision DSPs as co-processors, but much of the modelling work would still have to be performed by general-purpose CPUs.
Pure atmospheric dynamics can be done in the spectral domain, but other physics (convection, cloud physics, radiative processes, surface/atmosphere interactions) are handled by algorithms that usually require more than just solving a transform or a differential equation.
Re:Fallout
on
Nuke-Lobbing
·
· Score: 2, Informative
Fallout can of course be quite deadly. But these are very large areas you are talking about. When something is spread, it is diluted, and also the most radioactive isotopes will be gone in a matter of days anyway.
This is a fallacy. Allow me to quote from Sakharov's Memoirs (Knopf, 1990), pages 201-202, on the long-term effects of nuclear detonations:
"Bearing in mind that the average humand lifetime is 20,000 days, each roengten of global radiation will reduce this average lifetime by one week! My overall estimate of the number of human victims of a one-megaton detonation was 10,000. Two-thirds of this huge figure was attributed to the radioactive isotope carbon-14, which is formed during both "clean" and "normal" thermonuclear explosions. Carbon-14 has a half life of 5,000 years; its damaging effects thus continue over thousands of years. [...] By 1957, the total power of the nuclear bombs that had been tested around the world added up to nearly fifty megatons. According to my estimates, this would mean 500,000 casualties."
Footnote: "Frank von Hippel of Princeton University has used recent UN surveys of population exposures to atmosperic fallout and the health effects of ionizing radiation to obtain an estimate of 1,000 to 25,000 cancers and genetic disorders per megaton, which is consistent with Sakharov's earlier estimate"
Sakarov's article on the long term effects of atmospheric testing was instrumental in motivating the USSR and the US toward the landmark Atmospheric Testing Ban Treaty.
You make a valid point, of course, but if we stay on the Star Wars analogy, the philosophy in there was pretty weak too, if not weaker. (Speaking as someone from the Star Wars generation).
Bottom line: it's entertainment, but if it gets you thinking, all the better.
Don't reduce science to one experimental protocol. Anyone who is testing a theory by using it to make a prediction, then devising a reproducible experiment to test that prediction, is doing science.
Control groups can be one means to that end, but they are not the only one.
Einstein predicted gravity would bend light. The effect was accurately observed during a solar eclipse. No control group required.
Yes, they are a fun-loving bunch. They did a fair amount of proselitizing at my junior college, as far back as 1979. That's when I was told that Rael once went skinny-dipping on an Ocean-world with Jesus and Mary-Magdalen. (Name-dropper!)
I'd sign up for that.
For such feats of imagination, you could almost forgive them the small matter of self-glorifying, pseudo-scientific, harmful experiments on human beings.
Your wish sounds almost like Cygwin. If people can get to appreciate a Unix-like environment (regardless of the underlying engine), perhaps that would motivate them to make the full transition later on. But there still has to be the willingness to try something different.
FWIW: 10-12 years ago, I cut my Unix teeth on something called MiNT, which was essentially a Cygwin precursor for the Atari ST/TT/Falcon line. The move to Linux was natural after that.
Nice post, but I will offer one small nitpick. The Hobbits that went on the Quest weren't regular guys: they were carefully handpicked.
I agree that the hobbits as a whole represent regular people unfettered by nobility, but they do have their own pecking order and even a "noble" family, the Tooks.
Frodo Baggins, himself, is referred to as the finest hobbit in the Shire, and of course, is independently wealthy. His pursuits, like studying the elvish language, are unusual for "regular" people. Come to think of it, Sam is probably the only ordinary hobbit of the four Companions.
Gandalf, among other preoccupations, has an agenda of raising a new generation of hobbit leaders for the new age, a role that will be taken by Sam, Pippin, and Merry upon the liberation of the Shire.
The Earth Simulator is a 640-node computer, housed in a building the size of a stadium. Each node is a 64 GFlop Nec SX-6 supercomputer. (5,120 CPUs total).
It has its own dedicated, custom-built power station. 'nuff said.
> I don't get it. www.canada.gc.ca runs solaris with apache, Why doesn't www.gol.ged.gc.ca?
Because procurement isn't done on a uniform government wide basis. And that's a good thing, because different depts, and different agencies, have different requirements. My dept is MS-based for office work, but runs various Unices and Linux for science and web applications.
The significant analogy, used in the article, is that we know we are walking (hurtling semi-blindly?) toward a cliff. The uncertainty is with how fast we'll get there.
What bothers me is that they seem to be unable to quantify that distance, even a little. The fact that it happened once, only 12500 year ago, should indicate that we are not that far from the "cliff".
At first glance, it seems to me like the type of problem that numerical modelling should be able to handle reasonably well, at least well enough for what-if studies. Real-world prediction would require field data on ice melt. But I guess these things are never easy.
Finally, I agree with posters in other threads that this should not be misconstrued as "the end of global warming". Complex problems have complex ramifications.
I encourage everyone read the article, if you haven't alreday. It's good to see accessible writing from one of the authorities in the field.
Open file formats is one thing, but not the only thing.
Regarless of its technical prowess; if the software comes with over-controlling licensing terms, and the vendor has a bad track record in dealing with customers and other business, should you still use it?
And before you tell me that the GPL is over-controlling, take a good look at Microsoft EULAs.
This has nothing to do with global warming, and everything to do with measuring the local effect of aircraft-induced cirrus cloud. There was a three-day window when there was no such 'artificial' cirrus being produced. If you consider the space and time scale of the physical processes being looked at (a few hours to a day), I think this is not as bad as you are trying to make it sound.
Funny, I haven't seen any articles about chemtrails in Nature, though.
This is fair and square experimental science. It doesn't break theoretical ground, but measures the magnitude of a predictable effect. And as a trained gubmint meteorologist, I find the reported magnitude of the effect both surprising and interesting.
I couldn't care less what the chemtrail wackos will make of this, and we can't start modulating the spread of valid scientific information with concerns about the effect it will have on people with bad memes on their brain.
If I had mod points, the parent post would earn -1, troll.
Slashdot Mirror
Voyager scientists attach a plaque on the outbound trip - aliens attach a plague on the return trip.
I see your point and agree with most of it, but...
GRIB and METAR are not "obscure". They are well-documented international industry standards that serve their intended purpose quite well. Is a tar file "obscure"?
GRIB data was designed as a means to store and exchange weather model output data in its (almost) rawest form. Because it is in binary form, it is only natural that it needs to be post-processed to be visualized or used in other applications.
For those who might be interested : a description of the GRIB format.
I've used cooker for almost all of this development cycle, because I wanted to try out kernel 2.6. Mdk 9.2 did ship with a pre-release 2.6 kernel, but I wanted fully upgradable kernel packages.
For the most part the experience has been positive, and there was no severe breakage. But you have to be willing to put up with occasional annoyances. The KDE desktop was a little broken for a while (no drive icons), but I think it's ok now. Also, it's kind of hard to find reliable cooker mirrors. I've had to use European sites even though I am in North America.
My only persisting complaint is that Konqueror gets random segmentation faults (about once in 25 tries) when I click on a link on any given site. Could be my hardware, although I don't get the same trouble with Mozilla.
The computing context is similar where I work (Meteorological Service of Canada). There is a big difference between the forecaster's (technical) desktop and the clerical desktop. Our technical desktop never belonged to MS WIndows. Cost remains a concern, obviously, which is why we are moving away from proprietary Unix.
For what it's worth, our next-generation workstation is going to be java-based. (Joint effort with a handful of European countries led by Germany.) http://www.dwd.de/de/Technik/Projekte/NinJo/ (German language)
Dunno about the NWS, but for our clerical desktop to move away from Microsoft would be nothing short of miraculous.
There have been efforts toward using XML in meteorological codes, but so far they have been somewhat scattered and isolated. The US Navy gave it a fairly extensive shot at http://zowie.metnet.navy.mil/~spawar/JMV-TNG/XML/O MF.html
I S-2001.ht ml
The World Meteorological Organization is ultimately the standards body for international meteorological codes; you may want to have a look at sections 4.2 through 4.6 of this document to get a feel for the way XML is perceived by the WMO (not negatively, but as a complement to already well-established codes):
http://www.wmo.ch/web/www/WDM/reports/FW
That may be too much information if you only intend to build a quick demo... but the overall background may still interest you.
I encourage you to use the comment form on the site.
I can vouch for the fact that user comments are read and taken into account. The site's maintainers actually have a sincere concern for accessibility.
The Meteorological Service of Canada has a web page at:
http://weatheroffice.ec.gc.ca/canada_e.html . The going was a little rough there for a while (we don't have the resources of the big media outlets to make it look pretty) but it is getting better all the time. It is also the place where the information is guaranteed to be up to date.
Watch out about the images under "weather charts". They are still the large, old-style monochromatic images. (That will change, eventually, to smaller colour images).
With the intense competition and peer review in the field of weather and climate forecasting, model validation and comparison is a constant concern. I also happen to know researchers in the field who would go to great lengths to validate their results anyway, because they take pride in doing their work well and they want to get to the truth.
In the case of a model used for the daily forecast, there are archives and canned cases that can be used for comparison.
In the case of a climate model, you are interested in getting the long term trends right. Storms that last from a couple of days to a week are a high-frequency signal in terms of climate; you are not trying to reproduce every shortwave peak. So the materials you use for validation come either from historical records, if they are available, or from geological records and ice from ancient icecaps. If you can reproduce known historical trends, then you may have a model with some predictive value. Your comment about cheating has value for short term forecast model development, but misses the point of a climate model.
Yes, it is a difficult set of problems, with scales going from the molecular to the planetary. But all of it is based on sound physical science done by smart folks. There aren't many mysterious areas left; the challenge lies in the implementation and validation.
The point of using larger computers is so that fewer aspects have to be neglected, and to have more sophisticated representations for the phenomena we do handle.
You are right, the forecasting of individual hurricanes or storms is completely besides the point of a climate model.
The application here is in the area of climate forecasting, attempting to forecast trends in upcoming decades. It's not even important whether the model gets individual storms right, as long as the averages are realistic.
The advance is in becoming able to incorporate hurricanes in the simulation. This should help improve the realism of those trends and averages.
Each run of the model only offers one solution (called a deterministic forecast).
There is a technique called ensemble forecasting, whereby you run multiple instances of the model with slightly disturbed initial conditions and/or slightly tweaked model parameters. You can then examine the statistics of the ensemble to try and obtain information a deterministic forecast might not be able to give you.
Note that the goal in this particular case is not hurricane forecasting as such. The newsworthy information is that this is the first time that a climate model can be run at a resolution high enough that hurricanes become possible within the simulation. Short term models used for the daily weather forecast do this reasonably well already.
a more accurate method would be to extrapolate from the amount of water actually present in a cloud
Actually we don't know the method the meteorologist in the article used. I would have liked to know. But I think it's pretty clear she was not estimating all of the moisture in the cloud, just condensed water in suspension.
Even direct measurement methods, like taking a volumetric scan with a radar, would have a fairly large margin of error. But I think the point of the exercise, from the meteorologist's point of view, was to go through a fun "back of the envelope" calculation. The mass of clouds, as such, is never a direct consideration in day-to-day meteorology. Making estimates of precipitable water is the closest real-world calculation in this respect.
clouds... they're just concentrations of moisture that happen to refract and reflect visible light
The word 'cloud' applies to a concentration of water droplets or ice crystals, not to "any concentration of moisture". Atmospheric water vapour is invisible, and while it is an important parameter to meteorologists, vapour does not equal cloud.
So while we're at it, why not just weigh the air?
Weighing the air is not such a useless or purely theoretical activity. Measuring the air pressure with a barometer does exactly that. You need that kind of measurement, among other things, to calibrate aircraft altimeters and to analyse weather systems.
For weather modelling, I can perhaps envision DSPs as co-processors, but much of the modelling work would still have to be performed by general-purpose CPUs.
Pure atmospheric dynamics can be done in the spectral domain, but other physics (convection, cloud physics, radiative processes, surface/atmosphere interactions) are handled by algorithms that usually require more than just solving a transform or a differential equation.
Fallout can of course be quite deadly. But these are very large areas you are talking about. When something is spread, it is diluted, and also the most radioactive isotopes will be gone in a matter of days anyway.
This is a fallacy. Allow me to quote from Sakharov's Memoirs (Knopf, 1990), pages 201-202, on the long-term effects of nuclear detonations:
"Bearing in mind that the average humand lifetime is 20,000 days, each roengten of global radiation will reduce this average lifetime by one week! My overall estimate of the number of human victims of a one-megaton detonation was 10,000. Two-thirds of this huge figure was attributed to the radioactive isotope carbon-14, which is formed during both "clean" and "normal" thermonuclear explosions. Carbon-14 has a half life of 5,000 years; its damaging effects thus continue over thousands of years. [...] By 1957, the total power of the nuclear bombs that had been tested around the world added up to nearly fifty megatons. According to my estimates, this would mean 500,000 casualties."
Footnote: "Frank von Hippel of Princeton University has used recent UN surveys of population exposures to atmosperic fallout and the health effects of ionizing radiation to obtain an estimate of 1,000 to 25,000 cancers and genetic disorders per megaton, which is consistent with Sakharov's earlier estimate"
Sakarov's article on the long term effects of atmospheric testing was instrumental in motivating the USSR and the US toward the landmark Atmospheric Testing Ban Treaty.
You make a valid point, of course, but if we stay on the Star Wars analogy, the philosophy in there was pretty weak too, if not weaker. (Speaking as someone from the Star Wars generation).
Bottom line: it's entertainment, but if it gets you thinking, all the better.
Don't reduce science to one experimental protocol. Anyone who is testing a theory by using it to make a prediction, then devising a reproducible experiment to test that prediction, is doing science.
Control groups can be one means to that end, but they are not the only one.
Einstein predicted gravity would bend light. The effect was accurately observed during a solar eclipse. No control group required.
Yes, they are a fun-loving bunch. They did a fair amount of proselitizing at my junior college, as far back as 1979. That's when I was told that Rael once went skinny-dipping on an Ocean-world with Jesus and Mary-Magdalen. (Name-dropper!)
I'd sign up for that.
For such feats of imagination, you could almost forgive them the small matter of self-glorifying, pseudo-scientific, harmful experiments on human beings.
Your wish sounds almost like Cygwin. If people can get to appreciate a Unix-like environment (regardless of the underlying engine), perhaps that would motivate them to make the full transition later on. But there still has to be the willingness to try something different.
FWIW: 10-12 years ago, I cut my Unix teeth on something called MiNT, which was essentially a Cygwin precursor for the Atari ST/TT/Falcon line. The move to Linux was natural after that.
Nice post, but I will offer one small nitpick. The Hobbits that went on the Quest weren't regular guys: they were carefully handpicked.
I agree that the hobbits as a whole represent regular people unfettered by nobility, but they do have their own pecking order and even a "noble" family, the Tooks.
Frodo Baggins, himself, is referred to as the finest hobbit in the Shire, and of course, is independently wealthy. His pursuits, like studying the elvish language, are unusual for "regular" people. Come to think of it, Sam is probably the only ordinary hobbit of the four Companions.
Gandalf, among other preoccupations, has an agenda of raising a new generation of hobbit leaders for the new age, a role that will be taken by Sam, Pippin, and Merry upon the liberation of the Shire.
The Earth Simulator is a 640-node computer, housed in a building the size of a stadium. Each node is
a 64 GFlop Nec SX-6 supercomputer. (5,120 CPUs total).
It has its own dedicated, custom-built power station. 'nuff said.
Google is your friend, but for starters:
http://www.sw.nec.co.jp/hpc/sx-e/sx6/index.html
http://www.nec.co.jp/press/en/0203/0801.html
> I don't get it. www.canada.gc.ca runs solaris with apache, Why doesn't www.gol.ged.gc.ca?
Because procurement isn't done on a uniform government wide basis. And that's a good thing, because different depts, and different agencies, have different requirements. My dept is MS-based for office work, but runs various Unices and Linux for science and web applications.
The significant analogy, used in the article, is that we know we are walking (hurtling semi-blindly?) toward a cliff. The uncertainty is with how fast we'll get there.
What bothers me is that they seem to be unable to quantify that distance, even a little. The fact that it happened once, only 12500 year ago, should indicate that we are not that far from the "cliff".
At first glance, it seems to me like the type of problem that numerical modelling should be able to handle reasonably well, at least well enough for what-if studies. Real-world prediction would require field data on ice melt. But I guess these things are never easy.
Finally, I agree with posters in other threads that this should not be misconstrued as "the end of global warming". Complex problems have complex ramifications.
I encourage everyone read the article, if you haven't alreday. It's good to see accessible writing from one of the authorities in the field.
Open file formats is one thing, but not the only thing.
Regarless of its technical prowess; if the software comes with over-controlling licensing terms, and the vendor has a bad track record in dealing with customers and other business, should you still use it?
And before you tell me that the GPL is over-controlling, take a good look at Microsoft EULAs.
This has nothing to do with global warming, and everything to do with measuring the local effect of aircraft-induced cirrus cloud. There was a three-day window when there was no such 'artificial' cirrus being produced. If you consider the space and time scale of the physical processes being looked at (a few hours to a day), I think this is not as bad as you are trying to make it sound.
Funny, I haven't seen any articles about chemtrails in Nature, though.
This is fair and square experimental science. It doesn't break theoretical ground, but measures the magnitude of a predictable effect. And as a trained gubmint meteorologist, I find the reported magnitude of the effect both surprising and interesting.
I couldn't care less what the chemtrail wackos will make of this, and we can't start modulating the spread of valid scientific information with concerns about the effect it will have on people with bad memes on their brain.
If I had mod points, the parent post would earn -1, troll.
Here's a guy caught with a pick-axe in a submarine:
"Boats have ALWAYS sprung up leaks -- massively!"