Stress-Testing Software For Deep Space
kenekaplan writes "NASA has used VxWorks for several deep space missions, including Sojourner, Spirit, Opportunity and the Mars Reconnaissance Orbiter. When the space agency's Jet Propulsion Laboratory (JPL) needs to run stress tests or simulations for upgrades and fixes to the OS, Wind River's Mike Deliman gets the call. In a recent interview, Deliman, a senior member of the technical staff at Wind River, which is owned by Intel, gave a peek at the legacy technology under Curiosity's hood and recalled the emergency call he got when an earlier Mars mission hit a software snag after liftoff."
gay babies
While I buy that the landing systems need an RTOS, I doubt Curiosity does. Image processing that happens with "precision"? Do x86 processors not process images precisely enough? I get the idea of being hardened to radiation but it was my understanding we have newer processors that fit the bill on this. The rest of this seems like a rationalization for using old hardware. However, as an engineer for the government it's possible I'm just old and embittered.
''recalled the emergency call he got when an earlier Mars mission hit a software snag after liftoff."
From TFA:
Back when Spirit Rover landed on Mars in 2004, it experienced file systems problems. I got a call on landing day while I was in Southern California. I fired up my laptop and worked with three groups who were dealing with a variety of time zones: California, Japan and Mars. Since I had a RAD 6000 systems on my desk running simulations, by the end of first week we figured it out and were able to fix it.
is the worst off about a project is part of the DOG THAT IT IS. IT for a moment and Duty to be a big Yes! The numbers. The niggerness? and And abroad for it. Do noQt share Codebase became personal rivalries not going to play NetBSD posts on and promotes our In any way related Was after a long Of open-source. also dead, its *BSD is dying It is bleak future. In my bedpost up my
The last thing I would want to do is program mission-critical systems. That G*d my programming mistakes are hidden in the mire of a thousand other programmer's mistakes, and never make it to the front page of /.
Sent from my ENIAC
At least one instrument running VxWorks has been flying on the ISS since 2001. I'd be surprised if it were the only one.
If you can survive eight hours, you can survive *ANYTHING*...
With my long experience with VxWorks this doesn't surprise me. VxWorks is not the most robust RTOS. Think of it as a multi-tasking MS-DOS. The version they used has no memory protection between processes and I have found numerous areas of VxWorks to be badly implemented or downright buggy. Up through version 5.3 the malloc() implementation was absolutely horrid and suffered from severe fragmentation and performance problems. On the platform I was working with I replaced the VxWorks implementation with Doug Lea's implementation (which glibc was based off of) and our startup time dropped from an hour to 3 minutes. I was also able to easily add instrumentation so we could quickly find memory leaks or heap corruption in the field, something not possible with Wind River's implementation. After reading about the problems with the filesystem I looked at the Wind River filesystem code. It was rather ugly. They map FAT on top of flash memory (not the best choice) and the corner cases were not well handled (like a full filesystem).
Similarly, their TCP/IP stack sucked as well. If you can drop to the T-shell through a security exploit you totally own the box (i.e. Huawei's poor security record).
VxWorks is fine for simple applications, but for very complex applications it sucks. At least the 5.x series do not clean up after a task if it crashes because it does not keep track of what resources are used by a task. A task is basically just a thread of execution. All memory is a shared global pool. At the time it did have one feature that was useful that was lacking in Linux, priority inheritance mutexes. These are a requirement for proper real-time performance and I believe are now included in Linux.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
My PVR also runs VxWorks. Given that it still crashes randomly now and again, I hope they have a better version for space probes.
the other big player in space RTOS: RTEMS.
Free, open source, rtems.org.
Has all the same problems as VxWorks.. no process memory isolation (because space flight hardware doesn't have the hardware to support it usually)....
One thing that VxWorks has that RTEMS doesn't, and I wish it did, was dynamic loading and linking of applications. You're basically back in 1960s monolithic image days, not even with overlay loaders.
ALERT! Is your computer secretly functioning as a RADIO?
-=-=-=-
WebSDR: a Tunable Virtual Radio from Holland
(www) iw5edi (com)/ham-radio/58/websdr-a-tunable-virtual-radio-from-holland
"Thatâ(TM)s become a funny game during our last local net on 2 meters.
The screenshot you can see here behind is an online tunable radio working on 80 40 and 20 meters band from Holland.
We did enjoy comparing the delay beetween our receivers and the live feed coming form this web radio, and it was nice to discover that the difference was just half a second or less.
We did also test transmitting and getting heard by the remote station, and was interesting to see the waterfall differences on incrementing the output power.
But the most interesting thing was understanding what is behind this web pageâ¦.
First of all this is not a common remotely tunable radio as you may have had the opportunity to play, there are many remote radios on the net but rather this is a Software Defined Radio. Letâ(TM)s call it a Virtual Radio.
A Software Defined Radio, is a computer program, that emulate a complete radio, where tipical components like filters, mixers, amplifiers, demodulators, detectors etc have been reproduced by software programmed libraries. Input Output is demanded to commom sound cards, and a minimal RF frontend is provieded.
The University of Twente in Holland has published an SDR web interface that allow you to tune the radio, along to other concurrent users, by simulating a virtual receiver. It means that what you tune, is not heard by others, but only from your pc.
Like other virtualization technologies, a shared host platform is supplied (here is a common old-fashion pentium III PC running linux), and from this host you can load your own virtual exclusive radio and tune the waves via the supplied W3DZZ trap-dipole for the 80 and 40 meters
To load this sdr radio, requirements are Java Browser Plugin installed and access through TCP port 8901.
We can just say Bravo! to the hams that promoted this experiment."
http://www.france24.com/en/20121010-restaurant-removes-urinals-shaped-like-womans-mouth
AFP - A sumptuous new French restaurant in Sydney said Wednesday it would remove two urinals designed to resemble a woman's lipsticked mouth, apologising for any offence they have caused.
The Ananas Bar and Brasserie said the bright red-lipped urinals shaped like an open mouth were "a commonly used European design piece from female Dutch artist Meike van Schijndel".
"We sincerely apologise if they have caused offence. They are being removed today," a spokeswoman said in a statement.
The stylish restaurant opened three weeks ago, with the Sydney Morning Herald's food reviewer describing the urinals as "no real surprise here at Ananas, merely adding to the extraordinary collision of statements and intent".
But feminist, former political adviser and writer Anne Summers said the design was offensive. "Misogyny is very widespread, and this is just an example of misogyny," said Summers.
"The concept is pretty challenging and confronting. They're asking men to put their d(expletive) in these mouths as urinals."
Australia is the grip of a fierce political debate about sexism after Prime Minister Julia Gillard, the nation's first woman leader, accused opposition leader Tony Abbott of being a misogynist.
The unmarried Gillard said Tuesday she had been personally offended by many of Abbott's remarks over the years -- from urging her to "make an honest woman of herself", to his cat-calling at her in parliament.
"If he wants to know what misogyny looks like in modern Australia, he doesn't need a motion in the House of Representatives, he needs a mirror," she said in stinging comments.
that is (or was?) in newer Linksys routers, that are much less stable than the older Linux based versions..
http://en.wikipedia.org/wiki/VxWorks#Networking_and_communication_components
They start planning this years, years and years ahead. It is not uncustomary to have decided on a hardware platform five years before launch. Since there's a lot at stake for these bigger missions to succeed, they usually don't take risks and put stuff up there that hasn't proven itself. Maybe some evolution like a higher clock rate or more memory or something like that, but a new processor architecture gets tried on other things that have redundancy, lower cost or less exposure and preferably a combination of those.
I have been discussing some technology that was possibly put in an instrument on a weather/climate sat with the primary investigator of the then current mission and named to be the one of the next mission as well. This was around 2007. They had to choose the technology then, so they could work on plans and get funding around now. Once they get their funding, it will still be three to five years before it goes up there. Back then, due to the reliability demands they had for the sensor and the relative unproven state of using CMOS sensors for photon capture (common used in digital consumer cameras in 2007) they chose to go with the previous solution, that was in the current instrument. That means that they will probably launch a pre-CMOS sensor equipped instrument around 2015, because that was the best option available to them when it was decision time.
Unless we change the way we "go to space" in a radical way, I don't see the latest and greatest tech make it in missions like this. It's up there, sure it is, but only a handful people know it is and they don't want their precious black ops budget exposed or taken away from them. Once the statistics they get from the successes and failures (failing in secret "testing missions" once in a while is allowed) to a rating that makes it commercially viable to sell the tech to civilian usage, plus the state of technology used for espionage and military use is such that there isn't any tactical threat to do so, more modern tech will be used for missions like this.
I was promised a flying car. Where is my flying car?
Up through version 5.3 the malloc() implementation was absolutely horrid and suffered from severe fragmentation and performance problems.
I talked to one of Curiosity's software engineers the day it landed... he mentioned that one of their coding rules was: no malloc() allowed.
I don't care if it's 90,000 hectares. That lake was not my doing.
While I agree with most of the sentiments on the 5.x vxworks version, it has to be a said that vxworks is now at version 6.9 and is a much improved beast with a far better IP stack, support for 'proper' processes, etc. Saying that it comes with the cost of dropping or modifying a lot of API's making upgrades difficulty and to be honest looks so like linux once you've finished with it you wonder why you spent $50000 on a developer seat
As a slightly off thread, I always wondered why Intel bought windriver. One of the issues we have is that finding someone who knows the OS well is difficuilt because there is no way of getting exposure to it unless you have a lot money.
I can't help thinking Intel have missed a trick here. With the rise of the embedded hobbiest with things like Raspberry Pi, a new generation of engineers are learning, however there experience is based around ARM and linux, so further marginalising Intel in the embedded world, which is likely to be the big growth area in the future.
If intel was smart they would create there own hobbiest board based around an embeeded core duo or the like and provide a free version of vxworks to run on it. It doesn't need some of the high end features, but would provide early exposure to the OS as well as raising the profile of Intel in the embedded space.
Just a thought....
Didn't they used to do Linux distros back in the day?
Yes , I know, off topic , but just asking...
No malloc()? Interesting, I worked on a project at NG and we had same policy. Everything was on the stack or global. We had the chance to run with Monta Vista embedded Linux but someone higher up decided to go with "tried and true" VxWorks. I agree with a poster above about re-training costs and all that adding up.. but if embedded linux became standard with big companies I don't think it would take too long to make-up the costs of re-training and all the other stuff that goes with it.
An old Power PC can fly a spaceship to mars, execute a difficult landing and now semi autonomously drive a robot across the surface of a planet 30 million miles away , yet its not up to the job of writing documents using the latest word processors. Whats wrong with this picture?
I find the most revealing part of the interview that he publicly acknowledges his customers working on secret designs for space.
I'm sure those customers will deny any such project exists.
To Terminate, or not to Terminate, that's the question - SCSIROB
That is a good policy if you can do it, but in this case it was impossible. We had to use some 3rd party software which used malloc and realloc extensively. To make matters worse, for a long time we could only get obfuscated code to support the network processor we were using, meaning that it was impossible to make changes to it. We also had to make use of it because of the dynamic nature of the software. In our case it really wasn't feasible to avoid mllox. Replacing Windriver's malloc had some huge advantages. Fragmentation was horrible with the VxWorks malloc to the point where there were many tens of thousands of fragments of memory. VxWorks used a sorted linked list from smallest to largest free block. Due to the extensive dynamic reallocs, this linked list turned into a huge bottleneck.
Replacing the code with Doug Lea's malloc eliminated the fragmentation problem completely. By including the task ID and calling function's program counter in each block allocated it made it trivial to find memory leaks and keep track of how much memory and how many blocks were allocated per task or even by function.
There really was no good reason why VxWorks was chosen since there were no hard real-time requirements. The product was a mess (router and broadband remote access server) since each box had to include a Sun Ultrasparc computer running Solaris (we required big-endian) where most of the software ran. Solaris was an even worse choice. Trying to write streams drivers for Solaris was a nightmare compared to Linux drivers, especially when trying to tie into the TCP/IP stack. Not only that, Solaris was quite slow. Give me Linux any day.
The great thing about writing applications in Linux user space is that you can use tools like Valgrind to catch many of these memory leaks, uninitialized variables, etc.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
You probably shouldn't be using malloc() on an embedded system like that anyway. Statically allocate everything. That way you know exactly how much memory will be consumed at any time and can budget appropriately. It also reduces the chance of having a bug malloc() all your memory or running out of stack space.
VxWorks claims to have memory protection, chances are it is the CPU they are using which lacks an MMU to support it.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
If your objhective requires an RTOS, you're probably not going to malloc(). There are edge cases, but we've successfully banished them. We don't use VxWorks, thank god, but we do use a real memory machine instead of a virtual memory machine. Getting young programmers to understand that is challenging, and getting CS grads, of all fucking people, to program for a real memory machine is just fucking impossible. We make them managers instead.
I wonder what the CPU and memory load graphs would look like for a probe versus some standard desktop applications. Might explain a lot.
policy inheritance can be handled through FUTEX_PI. Issues due to a lock-contention can be handled by the kernel via FUTEX_LOCK_PI.
Get a web developer
Your desktop word processing software also didn't have a licensing cost in the hundreds of thousands of dollars...
Get a web developer
I would imagine that landing a spaceship takes a lot more CPU than reformatting some text and drawing a blinking cursor.
Malloc is non-deterministic. The request for a pointer to return contiguous free bytes will need to search a fragmented memory map to complete the request. The duration of the search depends upon the algorithms and the amount of fragmentation relative to the size of the request. It is worse if it must rearrange memory to accomodate the request. Thus, use of malloc() is typically avoided for time-critical code in a real-time operating system.
Why do you think that? Landing a space ship can be done using analogue electronics as a control system in the 60s. That means it was simple and light enough even when analogue in design that it made it into space. The rate on the feedback loops doesn't have to be more than a few khz and the amount of processing per loop is very low. More intensive than blinking a cursor yes but is it more intensive than reformatting text? Perhaps not so.
A PID controller is say 10 arithmetic operations per evaluation and only has to be evaluated at the rate of the feedback loops. No, it's not very much processing to control a spaceship landing.
"Landing a space ship can be done using analogue electronics as a control system in the 60s"
I don't remember any system in the 60s where a skycrane had to hover in place, lower a lander down, release it then fly off. Or navigate using image recognition. If you know otherwise fill me in.
"more intensive than reformatting text?"
Oh please. Reformatting text algorithms were running on 8 bit home computers in the 70s!
Oh please. Reformatting text algorithms were running on 8 bit home computers in the 70s!
I'm sure that explains why we still don't have hypenation in web browsers and justified text sucks. Hey, browser guys! This one has a clue! You have to use 8 bit home computers!
Or navigate using image recognition.
Well, you can call image recognition anything these days, like what univesity students do in their robotic-fight competitions (based on maybe an 8bit luminosity sensor) or what any laser based mouse does to detect movement across a surface. The devil is in the details, isn't it?
Your desktop word processing software also didn't have a licensing cost in the hundreds of thousands of dollars
It would if you were the only customer and it was only going to run on one computer. Do you have any idea how many programmers MS has and what it costs for salaries and other overhead?
Free Martian Whores!
You were going okay until here. You can't rearrange memory, malloc returns pointers, and there isn't any callback to ask for that pointer back to move it to another location.
Byte compiled languages like Java can rearrange memory but you call new not malloc so I know you weren't talking about them. Garbage collection is a much bigger problem especially if you think about mixing Java and real time operations. C/C++ in realtime means following the best practices, but for Java, get a different Java http://en.wikipedia.org/wiki/Real_time_Java.