Slashdot Mirror
Patching Software on Another Planet
An anonymous reader writes "Sixteen years ago, the Mars Pathfinder lander touched down on Mars and began collecting about the atmosphere and geology of the Red Planet. Its original mission was planned to last somewhere between a week and a month, but it only took a few days for software problems to crop up. The engineers responsible for the system were forced to diagnose the problem and issue a patch for a device that was millions of miles away. From the article: 'The Pathfinder's applications were scheduled by the VxWorks RTOS. Since VxWorks provides pre-emptive priority scheduling of threads, tasks were executed as threads with priorities determined by their relative urgency. The meteorological data gathering task ran as an infrequent, low priority thread, and used the information bus synchronized with mutual exclusion locks (mutexes). Other higher priority threads took precedence when necessary, including a very high priority bus management task, which also accessed the bus with mutexes. Unfortunately in this case, a long-running communications task, having higher priority than the meteorological task, but lower than the bus management task, prevented it from running. Soon, a watchdog timer noticed that the bus management task had not been executed for some time, concluded that something had gone wrong, and ordered a total system reset.'"
I didn't even read the full summary. But hasn't the occurrence of this priority inversion issue been reported about ... many years ago?
From TFA: "Engineers later confessed that system resets had occurred during pre-flight tests. They put these down to a hardware glitch and returned to focusing on the mission-critical landing software"
... even if a hardware glitch, wouldn't you want to track that down before launch? Especially since in the harsh space environment (bit flops even with hardened RAM/CPU), you want your hardware to be as reliable as possible.
Very surprised by this
Hulk SMASH Celiac Disease
Patching software systems is how you get humans to other star systems.
You send a ship, and if it takes 1000 years to get to the destination, that is ok because once it is there you can upload it a software upgrade that allows it to start growing humans from scratch on the remote planet.
We can patch hardware on another planet...
Yet we can't get commercial stuff we paid money for updated and working correctly down here on earth half the time.
Fixing code written by someone from a different planet.
Who the hell writes a modern, threaded RTOS and doesn't account for priority inversion?
The communications task priority was expressed as in decimal, while the data gathering priority was coded in hex.
This problem is known as priority inversion. Its a common concern in schedulers when critical functions run in their own threads. Its something that they should have known about and tested against. Or they could have used more traditional IO approaches and let the VxWorks IO system, which already has protection against priority inversion by design, do its job.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
I seem to recall reading that the return on investment for the US economy from the 1960s space program was something like 100-1. Today government investment in a space program acts as investment for private companies to develop new technologies - and I would be unsurprised to discover that the return is still not above 10-1 from an economic perspective.
If you really want to attack waste of money spending there are FAR better targets.
Do people still call this the red planet? lol
I remain convinced there should be a mod option of +1 Troll.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Moderation isn't near enough. That one was worthy of the Nobel Prize for Trolling.
Our tax dollars are going to these projects because private enterprise is unwilling to take up projects that will produce a guaranteed return for their investors. It is notably unwilling to take on risky projects, or, projects that do not have that clear return. Only an organization that has no profit motive (I.E. The federal government) is willing to invest the large sums in a project that might blow up during the boost phase of a launch. The fact is that the space program is quite profitable - The early years returned upwards of $14 for every $1 invested...yet, in spite of that, SpaceX is the ONLY private company that is interested in taking up the task. Even they are focusing on being truckers - providing transportation for other things into space..they have no interest in putting an exploration robot onto another planet, or sending a probe out into deep space to see what we can see.
As for the current debt...this article does a good analysis of it: http://useconomy.about.com/od/monetarypolicy/f/Who-Owns-US-National-Debt.htm and, it shows that in terms of the overall debt, China holds about 10% of the total. less than half the debt is held by foreign countries and investors. Not that it means it is good that the country owes that much cash, by any means...but, it is not like China could come in and put a lien on the country....
YAB - http://blog.beemandave.com/
"Scissors cuts paper, paper covers rock, rock crushes lizard, lizard poisons Spock, Spock smashes scissors, scissors decapitates lizard, lizard eats paper, paper disproves Spock, Spock vaporizes rock, and - as it always has - rock crushes scissors."
Seriously? This reads like morality tale for beginner programmers. "Remember kids, always check the settings of your mutexes!"
Will we also have articles about NASA engineers mistyping == for = ? Everyone makes mistakes, just because it happened in a rover doesn't make it interesting.
Your ignorance is showing. Interest rates are very low and the US has no trouble paying its interest. Take a look at the breakdown of federal spending. NASA is just noise in the federal budget. If you want to "fix the problem" you need to go after the big items, social security, Medicare/Medicaid, and the military. After that is unemployment and interest as you pointed out. Anyone who whines constantly about the discretionary part of the budget isn't thinking straight.
This was documented in great detail by a number of people years ago.
Doug Jensen
Is this (another) failed sarcastic statement, or did I really just read that. I'm taking away the point that the world needs less true believers, and people need to stop writing sarcasm online.
-Ultimate Stickman Game Developer Infinite World Puzzler
At the USENIX "Hot Topics in System Dependability 2012" conference Gerard Holzmann of JPL labs gave a fantastic talk about how they developed the software for the Curiosity rover. (spoiler: Having to display a Bieber poster in your cubical if break the nightly build, is a great motivator.)
I too would prioritise the landing software above a rare and recoverable reset after it's standing save and sound on the surface of Mars.
"(Engineers later confessed that system resets had occurred during pre-flight tests. They put these down to a hardware glitch and returned to focusing on the mission-critical landing software.)"
Engineering malpractice, but unfortunately common. It comes from shifting the burden of proof from proving that it will work, to proving that it will fail. It also cost a couple space shuttles.
Not to mention that the developments and data that is made available to the public and private industries by NASA and their space exploration and technology developments are responsible for a not insignificant chunk of the GNP. If it were private industry that were doing that, and they won't due to risk and unquantifiable short term return estimates, they would charge through the nose or hoard all that good stuff for themselves. The net result to the economy and human life would be negligible at best, and compared to our current status quo, a definite negative.
No, it's far better to have this stuff done by the government.
Can you describe how "exploration" of a vacuum would motivate private interests?
Are they certain it wasn't just the person on the tech support line who suggested rebooting it?
Have gnu, will travel.
1000 years? The fastest spacecraft we've ever launched won't reach Alpha Centauri for 40000 years, and that's only because it's not stopping.
This is a rambling bit of history. Move on if that's not your thing. I love reading about problems like the the Pathfinder problems. Trust me - such things often happen on Earth-bound systems, too.
Back in '79, I was working on a multiprocessing router for the ancient ARPANET. At the time the net had over sixty routers distributed across the continent. Actually we called them "imps" - well, "IMPS" but I'll use the modern term "router." We had a lot of the same problems as Pathfinder without ever leaving the atmosphere.
By then all ARPANET routers were remotely maintained. They all ran continuously and we did all software maintenance in Cambridge, MA. By then the basic software was really reliable. They rarely crashed on their own, and we mostly sent updates to tweak performance or to add new protocol features. Once in a while we'd have to use a "magic modem" message to restart a dead machine and to reload things. The software rarely broke so badly that we'd have to have someone on-site load up a paper tape. So remote maintenance was well established by then.
The multiprocessor didn't run "threads" it ran "strips." Each was a non-preemptive task designed to execute quickly enough not to monopolize the processor. If you wrote software for a Mac before OS-X, you know how this works. A multi-step process might involve a sequence of strips executed one after the other.
Debugging the multiprocessor code was a bit of a challenge because we could lock out multi-step processes in several different ways. While we could put our test router on the network for live testing, this didn't guarantee that we'd get the same traffic the software would get at other sites. For example, we had software to connect computer terminals directly to hosts through the router (the original "terminal access controllers"). This software ran at a lower priority than router-to-router packet handling. It was possible for a busy router to give all the bandwidth to the packets and essentially lock out the host traffic. Such problems might not show up until updated software was loaded into a busy site.
Uploading a patch involved assembly language. We'd generally add new code virus style. First you load the new code into some spare RAM. Once the code is loaded, we patch the working program so that it jumps to the patch the next time it executes. The patch jumps back to an appropriate spot in the program once the new code has executed. We sent the patches in a series of data packets with special addressing to talk to a "packet core" program that loaded them.
The bottom line: it's the sort of challenge that kept a lot of us working as programmers for a long time. And they pop up again every time someone starts another system from scratch.
More like "olds," am I right? Huh? Ahhh.
systemd is Roko's Basilisk.
This is the goto example of recent memory that I've always trotted out when asked about possible issues with task priority in embedded systems. I always note that while a more elegant solution might be to raise the holding task's priority to that of the highest priority task pending on the lock, the watchdog performed it's job exactly as it was supposed to do, and returned the system to a known recoverable state.
Even in this challenging scenario, there are still levels of 'good enough' we are prepared to accept.
Leaving now isn't the fastest way to get there. You'd get there faster by waiting back on Earth for more efficient propulsion technology to be developed. So when is the right time to leave in order to get there as soon as possible? That is a question which can only be answered in retrospect. One day people can look and say, hey we could have been there already if only we had left in the year x.
In reality there are other goals as well, so getting humans there as soon as possible isn't desirable as that would fail at some other goals of such a mission. The first mission to reach another star is going to be an unmanned probe. Serious research into such a possibility is actually already happening. The distance unmanned probes have travelled from the Earth is several orders of magnitudes beyond the furthest any living being has ever travelled from the Earth. I don't imagine manned missions will catch up with that in a thousand years.
But on a longer time scale I actually do imagine manned missions will travel beyond autonomous probes. That of course depends on which happens first, mankind successfully colonizing another star system or mankind extinguishing itself. If the successful colonization happens first, I don't see what would be stopping this from continuing throughout the galaxy. Reaching the other side of the galaxy take such a long time that evolution will be at play, and whoever reach the other side first will have evolved into a mindset where it seems entirely natural that once you have colonized a planet you breed for long enough to produce enough humans for the mission to the next star system. At that time the fatality rate of such missions would be high, but evolution would reward sending out a thousand missions even if only two survives rather than safely staying put until the first mission can be done safely.
If we do reach the other side of the galaxy, it will be with a manned mission intending to colonize yet another star system. Antonymous probes from the Earth will never get that far. What will mankind do, once there are no more habitable star systems left in this galaxy? I guess some crazy attempts at reaching other galaxies. But none of this will happen in my lifetime anyway, so we'll never know if my guesses are even remotely correct.
Do you care about the security of your wireless mouse?
Interestingly, I've been involved in open source development for spaceflight software, and I don't see legions of users coming forward with the defects uncovered by their many eyes, and fixes produced by their busy fingers, notwithstanding that the software is available open source to U.S. Person who asks for it.
There is also the open source operating system RTEMS, which is also at Mars, as well as the closed source VxWorks.
This is why you don't use threads for important stuff...
What will mankind do, once there are no more habitable star systems left in this galaxy? I guess some crazy attempts at reaching other galaxies.
One thing is certain - by that time, it would not be mankind, any more than what we are today can be called fishkind.
But I highly doubt that we'll get there. Evolution does not favour long term strategies unless those picking short term strategies die off.
In terms of time passed it could be a shorter period than the time it took to evolve from monkeys into todays humans. Whether we will use the term human about all descendants of humans is a matter of definition. The changes in culture and technology are likely to be greater than the changes in genome. But all of the changes would be subject to evolutionary selection.
The two are not mutually exclusive. Earth would still be populated by humans favouring strategies suitable for life on Earth. Compared to the total size of the human population it only takes a small number of individuals to seed colonization across the galaxy. And other star systems would be dominated by those who are ready to take the risks involved in interstellar travel.
Do you care about the security of your wireless mouse?
or any other Microsoft related OS thanks God.
Sent as ripples into the electromagnetic field. No single photon has been harmed in the process.
Ugh. I can only infer that the 'strips' you ran, if they needed to do something more complex than a single strip in aggregate, sometimes had to be backed by a state machine. What a headache. Arg. ;-P
In terms of time passed it could be a shorter period than the time it took to evolve from monkeys into todays humans.
Humans did not evolve from monkeys. Monkeys and humans evolved from a common haplorhini ancestor which was neither monkey nor human, around 40 million years ago. Your typical random monkey has undergone as much evolution since then as the typical human has.
Journeys to other solar system would take an enormous amount of years. So much so that it probably won't happen with live crews. Sending DNA records and reconstructing life at the destination might be the best bet. But even if we used the same blueprint for seeding millions of stars, evolution would occur on all of them, and by the time any of the descendants could meet (but why would they?), they would have evolved so much in different directions that meeting a cousin from the stars might be like meeting a trout.
The two are not mutually exclusive. Earth would still be populated by humans favouring strategies suitable for life on Earth. Compared to the total size of the human population it only takes a small number of individuals to seed colonization across the galaxy. And other star systems would be dominated by those who are ready to take the risks involved in interstellar travel.
But what's the short term reward that would prevent the migration-friendly to be selected against, as they undoubtedly will have to sink resources into their long term goals before even leaving? What makes us think they'd be favoured enough to be allowed to leave before going extinct?
If you could reconstruct pictures of what they looked like, I bet the majority of people would classify it as a monkey if you showed them the picture. And chances are nobody would classify it as a fish.
If you could build propulsion capable of delivering 1G of acceleration continuously for the entire duration of the flight, then I think you could do it in a lifetime. As a nice side effect that would provide artificial gravity during the flight. Achieving that would require a much higher specific impulse than any current propulsion technology could deliver. You'd actually need a powerful particle accelerator onboard the craft.
I haven't done all of the math, but it would be interesting to see a comparison of the specific impulse needed with the speeds delivered by current accelerator rings.
Even if you couldn't do the trip in a lifetime, that just means you need sustainable recycling on board capable of supporting life for enough generations to make the trip there. Which is probably easier than growing and raising a child without any humans around.
Do you care about the security of your wireless mouse?
At the time it seemed virtuous to implement state machines. One guy did his phd by building a mechanism that did coroutining - the programmer could write out the whole procedure and stick in the strip breaks after the fact. I suppose someone did something like that for the Mac, tho I stopped writing Mac code before seeing such a thing.