And, yes, pilot privacy is a concern because certain well-known air crashes have involved the airline and/or even government falsifying data to put the blame on the pilots (cue fingers wagged at France).
One of the, if not THE, most common causes of aircraft crashes issued by the National Transportation Safety Board is "pilot error". But, there's a reason for that. There's a lot that can go wrong in an airplane, and we're trained to do things about almost all of them (having a piece of FOD penetrate your delta-wing fuel tank on takeoff and essentially render your plane a molotov cocktail looking for a place to die excluded). When a private pilot ignores worsening weather and meets cumulo-granite, that's pilot error. Continued flight into known icing conditions, ditto. Running out of fuel, yep, same thing. Now, two out out of three of those are little-airplane-related, and the third often is, but running out ouf fuel has happened to the big-iron drivers, too, and they didn't admit it to get priority or emergency handling from air traffic control. By the same token, sometimes, pilots are required by COMPANY regulations to do things a particular way, and that comes out as "pilot error" too. And that is something that should be exposed to scrutiny. But, by the letter of the law, anything that happens on a flight is the responsibility of the senior pilot on the aircraft. There's a lot more that goes into Pilot In Command structure, too, but that's for another post.
In the end, it really is all about organization, and a few basic rules. 1. Store your data first. Before you do anything to modify it, transmogrify it, or distort it. 2. Assimilate like datasets. This obviously doesn't apply to the parent's datasets, but it does to mine. 3. Normalize for rapid retrieval (CSV, ISAM, flat file, or RDMBS as appropriate; flash cards are fine with me, if that's what you like). Note that you've not done anything to the original data. 4. Process/analyze. SQL queries, process chains, whatever it takes. That's what the normalization stage allows you to do readily.
What you've described is an inredibly straightforward approach to scientific data management, and worth wading through the other comments to find.
My personal preference is to store data in NetCDF (or another self-describing common data format) and also encode it into a database. I can do manipulations better in the database, but NetCDF has a staying power derived from its NASA and UCAR origins.
Science Informatics. You just told me what I do, when I'm not doing my own research... which seems to be happening less and less these days. A lot of the kids we train now can write something in Matlab or IDV, but couldn't perform a first normalization on a database if their lives depended on it. A lot of them have learned a little perl... or.Net, but know nothing of C or Fortran, and can't spell MPICH even when the models they use depend on it.
And seriously, corporate and university IT staff are not suited for this purpose, just as I'm poorly suited to help you with your next Windows installation. My expertise lies elsewhere. Getting the senior management to understand these differences, however, is a problem.
Fusion splicing the glass underwater is a bitch. Getting a good cleave at those depths with the diving gloves on is the hardest part, assuming the batteries haven't collapsed (nor your frail body) at those depths.
Nonetheless, since 9/11 the explosive growth has cause a marked increase in delays in clearance investigations. I've heard this from all sides of the situation, those waiting for their clearances (or even for their investigations!), and those responsible for doing same. The number of required personnel clearances has gone 'way up. We may not have that many working on classified projects, or more to the point, with classified information, but we have a lot of folks who are required to have clearances now who weren't before the explosive growth following the 2001 attack.
In weather forecasting, we find ourselves starting, stopping and waiting. Some of the tiles on which we compute will be trivially simple to complete, while others will not run to something approaching a numerically complete solution for some larger number of iterations. Ttherefore, we have to wait for the slowest computation/solution before we advect results to the surrounding tiles and begin the process all over again.
The nature of parallel problems isn't so simple that you generalize about how synchronization isn't important. I've also got examples of two parallel codes which have to stop, synchronize data and restart, which is very inefficient. The ability to scale the combination of shallow water equations and non-hydrostatic weather codes into a large, embarassingly parallel system would be a Good Thing for earth-system modeling in general, but is practically difficult, both in its core nature and because message passing isn't trivial at some point in a large MPI system. It simply becomes unwieldy and breaks down.
When you're parallizing a Monte Carlo problem, you can achieve significant speedup with little concern for asynchrony.When you submit a problem where each computation requires waiting for all of its surrounding neighbors, asynchrony, using conventional approaches becomes problemmatical.
I suspect that, while this project might not demonstrate 1megacore high performance, that they're on the right path.
If you look at today's multicore processor HPC, you see some (relatively) small number of cores on a single board, communicating at higher speeds than inter-board links, but the interboard links tend to be sorta fast, and tied to an interconnect fabric of some sort. This can be faster or slower depending on the base technology, and little things like bisectional bandwidth, and subscription. If one's not running some form of OpenMP (essentially threaded) environment, which also often equates to SM but doesn't mandate it, one can take significant advantage of the on-board higher speed interconnects. What's off the board doesn't affect your outcome.
On the other hand, if you have to subscribe to DM-parallelism, today's offerings are similar with minor tweaks in performance: MPICH, MPICH2, OpenMPI, etc., all striving to give the best multi-NODE intercommunication one can broadcast, and then gather back in. One of the key drawbacks, however, is that most times, I/O is scatter-gather, or, essentially serial. One, or some small subset of intercommunicating nodes gathers all the results, decides how to parse them, and/or commits them to results.In general, during a big I/O operation, the nodes wait.
In the original author's description, there's a bunch of relatively slow interconnects to allow a new topology of interconnect. A bunch of gigabit doesn't equal a QDR InfiniBand connection, but by not tying the interconnect to a fixed, store and forward interconnect, you just MIGHT gain a bit of efficiency.Somehow, I think 6 GBE interconnects is light, but I'll let better minds than mine optimize that over time.
This may well be a "plain old message passing computer" but it's novel in its reduced dependence on scatter-gather and store-forward technology.
Old news. This story was making the science rounds early last week. It is a devastating spill, and it is depleting subsurface oxygen, but the scare tactics are not helping anyone clean it up.
Maybe it's because there are few enough dinosaurs like me around who learned assembly and are still teaching. I've learned to let the kids learn in C, but I do tend to think the bedrock resides with assembly and machine code. Or, at least, if you really want to know what's happening.
And, well, that may be the case. However, if you look, there's been an ebb and flow to the quality of CompSci around the US over the last several years. I still do what I can to keep quality up. However, we do have folks who're more interested in making sure the kids are Microsoft compliant than in determining if they understand the Science part of Computer Science.
When I hire a graduate student or undergrad worker, I have them do a couple of things for assessment... 1. Write a relatively simple program in the language of their choice for our *nix boxes. Yes, it's got to run. 2. Demonstrate their virtuosity with SQL in PostgreSQL and PostGIS. 3. Show their web programming skills in perl/php/ajax, and javascript on a real application. 4. Prove they can maneuver around a linux system from a command line.
By this time, I've already hired them, and unless they do something like piss in the boss's potted plants, I'll likely keep them, but it tells me how much I can expect, and what sort of remedial work I have to do with them.
1. Yes, I'm now installing Mono to support the kids who only know.net framework. It's gotten that bad. 2. There's a lot of remedial SQL training going on. Lots of kids are "learning database" using Access as their sole tool. That's just wrong. 3. A bright spot. Seems everyone knows how to do some web programming now. 4. Command line? Linux? Those are evil...
Sorry, I spend a lot of money supporting students with work, and a lot of time and money teaching them things in a practical environment, that they should have at least been exposed to in some class, somewhere.
Retaining VB and VB.net suggest a hefty Microsoft contribution. At least to me.
I've been watching the dilution of programming skills in our CompSci department over the last several years. The question I've been asking is, "Who's going to write the next OS, or even the next decent compiler, if we stop teaching languages that get us closer to the hardware?" From our administration, the answer's been a bit quiet, but seems to be either, "who cares?", or "why?"
In an OS class I taught recently, 3 of 5 students had not been exposed to any assembly language over the course of 4 years of CompSci, two had not met a full C programming class, either in high school or college. Most had experience in java. All had taken.net classes. None had any class requirement to be familiar with Linux or Unix. In the comments at the end of the class, several were very unhappy that all my class examples required a command line and didn't show them a flashy GUI to look at registers or other output.
On the positive side, although the senior professor in OS liked what I'd done, and was nonplussed by the evaluations, the department head has said I wasn't likely to teach again, because I tended to focus on technologies that weren't relevant to our students, like web programming and SAAS. Upon reminding her I'd been assigned an OS class, she said I could have integrated more web programming into the content instead of, say, the lectures on HPC, threading, and message-passing, which is what I do a lot of these days, and an area I feel is underrepresented. That I "wasted" two lectures, and still covered all the (approved) syllabus material, and that the kids all did manage to pass with decent grades, wasn't as important as the fact that I'd not focused on what she (and the students) thought was important. (Strange, when did my syllabus become a topic for debate with the class?)
And while I'm ranting, there's an awful lot of real science still written in Fortran, but it's not taught much anymore. I'm thinking of reviving a free-university class in fortran to support scientific newbies who need to learn it for their course-work and graduate degrees.
I taught a summer OS class a couple of years ago, with 5 graduating seniors. I asked what everyone's favorite language was, and then, what other languages they knew well enough to program in.
Three claimed they did all their current work in Python. One, Java. And one, Visual Basic. Yeah, really. Yes, the class was that small, we made the class to accommodate graduation at two universities.
None had any exposure to Fortran. Now, before you all get wrapped up in this, I generally write and run numerical analysis codes and weather models, and Fortran is well suited to this application. In the case of three of the students, I think I know why: Fortran hasn't been offered here for 6 years.
One of the kids claimed proficiency with C, and another said she had experience with C++. The other three claimed C# as an equivalent, and what they had learned.
Assembly? Oh, really. Give me a break. The first (and last) day I showed an assembly language routine, it was obvious they couldn't follow what I was saying: The logic just wasn't there. And, none had been required to take a class in assembly language for any processor.
ALL had been exposed to Visual Studio, and had been assured that this was the superior tool for programming at all levels.
On the positive side, all were comfortable with threads and grasped OpenMP. None, however, had any concept of why we might use MPI for distributed computation.
Actually, it interfered with a lot more than Amateur Radio spectrum: There were issues with DoD, DHS, public safety, SCADA operations, marine and petroleum. The multiple carrier aspect of it was interesting to examine on the spectrum analyzer, as it indicated a seriously broad-spectrum threat to RF services.
Addressing Manassas, BPL was never well-conceived, and Manassas was destined to fail. I'm sorry, but you transmit an RF signal along an unshielded random wire length without radiation and susceptibility problems. The BPL folks wanted regulation to prevent interference from all the existing users out there, and then lied to their potential customers about the impacts. Good engineering practice, and adherence to solid engineering won out here. It's not like BPL was going to do great things: It's expensive, complicated and requires regeneration at each transformer, and a variety of other points along the way. It's bad engineering done poorly.
It's a bit problematical to be a politician. If you don't promise things, but rather say, "I'll try to get this past Congress", folks don't think you'll do anything. If you do promise something, it's subject to the whim of the folks in Congress, and who they're trying to curry favor with at the time.
If you have a party bent on denying you any successes, using any of the arcane methods at their disposal through Roberts' Rules, or the Byzantine Senate and House rules, you suddenly look unsuccessful. If, un the other hand, you hold a solid majority and have leadership that whips you into line if you step out, you can pass whatever you want.
If you're the current Democratic leadership, you can manage to swap a supermajority for something less without getting legislation passed. If, on the other hand, you're Tom Delay or Newt Gingrich, you can keep your party in line and push through what your Party wants.
No one ever said politics at the National level was pretty, or subject to the rules of logic. In fact, while I'd like to have seen some of Obama's "promises" (I'm a bit more realistic in my expectations) passed and enacted, I'm pretty glad we're getting toward deadlock. That way, the Congress-critters can't break too many more things.
Slashdot Mirror
And, yes, pilot privacy is a concern because certain well-known air crashes have involved the airline and/or even government falsifying data to put the blame on the pilots (cue fingers wagged at France).
One of the, if not THE, most common causes of aircraft crashes issued by the National Transportation Safety Board is "pilot error". But, there's a reason for that. There's a lot that can go wrong in an airplane, and we're trained to do things about almost all of them (having a piece of FOD penetrate your delta-wing fuel tank on takeoff and essentially render your plane a molotov cocktail looking for a place to die excluded). When a private pilot ignores worsening weather and meets cumulo-granite, that's pilot error. Continued flight into known icing conditions, ditto. Running out of fuel, yep, same thing. Now, two out out of three of those are little-airplane-related, and the third often is, but running out ouf fuel has happened to the big-iron drivers, too, and they didn't admit it to get priority or emergency handling from air traffic control. By the same token, sometimes, pilots are required by COMPANY regulations to do things a particular way, and that comes out as "pilot error" too. And that is something that should be exposed to scrutiny. But, by the letter of the law, anything that happens on a flight is the responsibility of the senior pilot on the aircraft. There's a lot more that goes into Pilot In Command structure, too, but that's for another post.
In the end, it really is all about organization, and a few basic rules.
1. Store your data first. Before you do anything to modify it, transmogrify it, or distort it.
2. Assimilate like datasets. This obviously doesn't apply to the parent's datasets, but it does to mine.
3. Normalize for rapid retrieval (CSV, ISAM, flat file, or RDMBS as appropriate; flash cards are fine with me, if that's what you like). Note that you've not done anything to the original data.
4. Process/analyze. SQL queries, process chains, whatever it takes. That's what the normalization stage allows you to do readily.
What you've described is an inredibly straightforward approach to scientific data management, and worth wading through the other comments to find.
Mod up +3. You're right on-target.
My personal preference is to store data in NetCDF (or another self-describing common data format) and also encode it into a database. I can do manipulations better in the database, but NetCDF has a staying power derived from its NASA and UCAR origins.
Science Informatics. You just told me what I do, when I'm not doing my own research... which seems to be happening less and less these days. A lot of the kids we train now can write something in Matlab or IDV, but couldn't perform a first normalization on a database if their lives depended on it. A lot of them have learned a little perl... or .Net, but know nothing of C or Fortran, and can't spell MPICH even when the models they use depend on it.
And seriously, corporate and university IT staff are not suited for this purpose, just as I'm poorly suited to help you with your next Windows installation. My expertise lies elsewhere. Getting the senior management to understand these differences, however, is a problem.
Where do I find the open source COBOL projects?
Oh, there are, but the US Navy, and a number of other naval forces, tend to only share amongst themselves.
No, that would have implied too much accurate information for our Bored of Education folks.
Fusion splicing the glass underwater is a bitch. Getting a good cleave at those depths with the diving gloves on is the hardest part, assuming the batteries haven't collapsed (nor your frail body) at those depths.
A GT-550? Late model rig. I used a Galaxy V Mk II...
No commercial aircraft have ever flown on solar- or wind-derived energy. Think about that.
Nonetheless, since 9/11 the explosive growth has cause a marked increase in delays in clearance investigations. I've heard this from all sides of the situation, those waiting for their clearances (or even for their investigations!), and those responsible for doing same. The number of required personnel clearances has gone 'way up. We may not have that many working on classified projects, or more to the point, with classified information, but we have a lot of folks who are required to have clearances now who weren't before the explosive growth following the 2001 attack.
Got any spare tools to automagically parallelize that serial weather physics code?
In weather forecasting, we find ourselves starting, stopping and waiting. Some of the tiles on which we compute will be trivially simple to complete, while others will not run to something approaching a numerically complete solution for some larger number of iterations. Ttherefore, we have to wait for the slowest computation/solution before we advect results to the surrounding tiles and begin the process all over again.
The nature of parallel problems isn't so simple that you generalize about how synchronization isn't important. I've also got examples of two parallel codes which have to stop, synchronize data and restart, which is very inefficient. The ability to scale the combination of shallow water equations and non-hydrostatic weather codes into a large, embarassingly parallel system would be a Good Thing for earth-system modeling in general, but is practically difficult, both in its core nature and because message passing isn't trivial at some point in a large MPI system. It simply becomes unwieldy and breaks down.
When you're parallizing a Monte Carlo problem, you can achieve significant speedup with little concern for asynchrony.When you submit a problem where each computation requires waiting for all of its surrounding neighbors, asynchrony, using conventional approaches becomes problemmatical.
I suspect that, while this project might not demonstrate 1megacore high performance, that they're on the right path.
If you look at today's multicore processor HPC, you see some (relatively) small number of cores on a single board, communicating at higher speeds than inter-board links, but the interboard links tend to be sorta fast, and tied to an interconnect fabric of some sort. This can be faster or slower depending on the base technology, and little things like bisectional bandwidth, and subscription. If one's not running some form of OpenMP (essentially threaded) environment, which also often equates to SM but doesn't mandate it, one can take significant advantage of the on-board higher speed interconnects. What's off the board doesn't affect your outcome.
On the other hand, if you have to subscribe to DM-parallelism, today's offerings are similar with minor tweaks in performance: MPICH, MPICH2, OpenMPI, etc., all striving to give the best multi-NODE intercommunication one can broadcast, and then gather back in. One of the key drawbacks, however, is that most times, I/O is scatter-gather, or, essentially serial. One, or some small subset of intercommunicating nodes gathers all the results, decides how to parse them, and/or commits them to results.In general, during a big I/O operation, the nodes wait.
In the original author's description, there's a bunch of relatively slow interconnects to allow a new topology of interconnect. A bunch of gigabit doesn't equal a QDR InfiniBand connection, but by not tying the interconnect to a fixed, store and forward interconnect, you just MIGHT gain a bit of efficiency.Somehow, I think 6 GBE interconnects is light, but I'll let better minds than mine optimize that over time.
This may well be a "plain old message passing computer" but it's novel in its reduced dependence on scatter-gather and store-forward technology.
Old news. This story was making the science rounds early last week. It is a devastating spill, and it is depleting subsurface oxygen, but the scare tactics are not helping anyone clean it up.
Maybe it's because there are few enough dinosaurs like me around who learned assembly and are still teaching. I've learned to let the kids learn in C, but I do tend to think the bedrock resides with assembly and machine code. Or, at least, if you really want to know what's happening.
C is where I'd like 'em to show proficiency at first. They can learn other languages later.
And, well, that may be the case. However, if you look, there's been an ebb and flow to the quality of CompSci around the US over the last several years. I still do what I can to keep quality up. However, we do have folks who're more interested in making sure the kids are Microsoft compliant than in determining if they understand the Science part of Computer Science.
When I hire a graduate student or undergrad worker, I have them do a couple of things for assessment...
1. Write a relatively simple program in the language of their choice for our *nix boxes. Yes, it's got to run.
2. Demonstrate their virtuosity with SQL in PostgreSQL and PostGIS.
3. Show their web programming skills in perl/php/ajax, and javascript on a real application.
4. Prove they can maneuver around a linux system from a command line.
By this time, I've already hired them, and unless they do something like piss in the boss's potted plants, I'll likely keep them, but it tells me how much I can expect, and what sort of remedial work I have to do with them.
1. Yes, I'm now installing Mono to support the kids who only know .net framework. It's gotten that bad.
2. There's a lot of remedial SQL training going on. Lots of kids are "learning database" using Access as their sole tool. That's just wrong.
3. A bright spot. Seems everyone knows how to do some web programming now.
4. Command line? Linux? Those are evil...
Sorry, I spend a lot of money supporting students with work, and a lot of time and money teaching them things in a practical environment, that they should have at least been exposed to in some class, somewhere.
Retaining VB and VB.net suggest a hefty Microsoft contribution. At least to me.
I've been watching the dilution of programming skills in our CompSci department over the last several years. The question I've been asking is, "Who's going to write the next OS, or even the next decent compiler, if we stop teaching languages that get us closer to the hardware?" From our administration, the answer's been a bit quiet, but seems to be either, "who cares?", or "why?"
In an OS class I taught recently, 3 of 5 students had not been exposed to any assembly language over the course of 4 years of CompSci, two had not met a full C programming class, either in high school or college. Most had experience in java. All had taken .net classes. None had any class requirement to be familiar with Linux or Unix. In the comments at the end of the class, several were very unhappy that all my class examples required a command line and didn't show them a flashy GUI to look at registers or other output.
On the positive side, although the senior professor in OS liked what I'd done, and was nonplussed by the evaluations, the department head has said I wasn't likely to teach again, because I tended to focus on technologies that weren't relevant to our students, like web programming and SAAS. Upon reminding her I'd been assigned an OS class, she said I could have integrated more web programming into the content instead of, say, the lectures on HPC, threading, and message-passing, which is what I do a lot of these days, and an area I feel is underrepresented. That I "wasted" two lectures, and still covered all the (approved) syllabus material, and that the kids all did manage to pass with decent grades, wasn't as important as the fact that I'd not focused on what she (and the students) thought was important. (Strange, when did my syllabus become a topic for debate with the class?)
And while I'm ranting, there's an awful lot of real science still written in Fortran, but it's not taught much anymore. I'm thinking of reviving a free-university class in fortran to support scientific newbies who need to learn it for their course-work and graduate degrees.
What was Cisco thinking? Well, they weren't thinking about cooling.... probably about who to acquire next and call it "innovation".
I taught a summer OS class a couple of years ago, with 5 graduating seniors. I asked what everyone's favorite language was, and then, what other languages they knew well enough to program in.
Three claimed they did all their current work in Python. One, Java. And one, Visual Basic. Yeah, really. Yes, the class was that small, we made the class to accommodate graduation at two universities.
None had any exposure to Fortran. Now, before you all get wrapped up in this, I generally write and run numerical analysis codes and weather models, and Fortran is well suited to this application. In the case of three of the students, I think I know why: Fortran hasn't been offered here for 6 years.
One of the kids claimed proficiency with C, and another said she had experience with C++. The other three claimed C# as an equivalent, and what they had learned.
Assembly? Oh, really. Give me a break. The first (and last) day I showed an assembly language routine, it was obvious they couldn't follow what I was saying: The logic just wasn't there. And, none had been required to take a class in assembly language for any processor.
ALL had been exposed to Visual Studio, and had been assured that this was the superior tool for programming at all levels.
On the positive side, all were comfortable with threads and grasped OpenMP. None, however, had any concept of why we might use MPI for distributed computation.
Actually, it interfered with a lot more than Amateur Radio spectrum: There were issues with DoD, DHS, public safety, SCADA operations, marine and petroleum. The multiple carrier aspect of it was interesting to examine on the spectrum analyzer, as it indicated a seriously broad-spectrum threat to RF services.
Addressing Manassas, BPL was never well-conceived, and Manassas was destined to fail. I'm sorry, but you transmit an RF signal along an unshielded random wire length without radiation and susceptibility problems. The BPL folks wanted regulation to prevent interference from all the existing users out there, and then lied to their potential customers about the impacts. Good engineering practice, and adherence to solid engineering won out here. It's not like BPL was going to do great things: It's expensive, complicated and requires regeneration at each transformer, and a variety of other points along the way. It's bad engineering done poorly.
It's a bit problematical to be a politician. If you don't promise things, but rather say, "I'll try to get this past Congress", folks don't think you'll do anything. If you do promise something, it's subject to the whim of the folks in Congress, and who they're trying to curry favor with at the time.
If you have a party bent on denying you any successes, using any of the arcane methods at their disposal through Roberts' Rules, or the Byzantine Senate and House rules, you suddenly look unsuccessful. If, un the other hand, you hold a solid majority and have leadership that whips you into line if you step out, you can pass whatever you want.
If you're the current Democratic leadership, you can manage to swap a supermajority for something less without getting legislation passed. If, on the other hand, you're Tom Delay or Newt Gingrich, you can keep your party in line and push through what your Party wants.
No one ever said politics at the National level was pretty, or subject to the rules of logic. In fact, while I'd like to have seen some of Obama's "promises" (I'm a bit more realistic in my expectations) passed and enacted, I'm pretty glad we're getting toward deadlock. That way, the Congress-critters can't break too many more things.