Ham Satellite Suffers Failures, Is Silent

Bruce Perens writes "The Phase 3-D satellite, renamed Oscar 40 once it reached orbit, has suffered multiple failures and has stopped transmitting telemetry. It is not yet known if the satellite is responding to commands. The main telemetry beacon on 145.898 MHz was operating poorly after separation and now may be transmitting a weak unmodulated carrier. A fuel valve appears to be sticking closed and appeared to be only 10% open after multiple operation commands. Then, the first flight burn was 3 minutes too long due to another, not-yet-understood problem, boosting the craft into a higher orbit than expected (not yet a problem in itself as the final orbit is to be even higher). A backup flight computer, itself an experiment to see if the CPU would be radiation-hard enough to survive, has its RAM corrupted every 1 or 2 days in orbit as it crosses radiation belts and currently is not set up to reboot automaticaly. The primary flight computer may have crashed. An expected watchdog timer reset did not occur, but this would be the case if the satellite was receiving some commands."

"We may have to wait until after Christmas for another reset. Controllers have not transmitted a hard reset command yet, which would work even with the flight computer crashed, as they wish to explore other options. The satellite would automaticaly cycle through a number of frequencies and antennas if it does not get any commands for 10 orbits, and controllers hope to re-establish control as this happens. A hard reset could delay that option.

"The satellite waited several years for launch due to Arianne 5's early failures and a revised accelleration profile for the booster that required a redisign of the satellite frame to take additional stress. Aging may have effected components such as the fuel valves. Telemetry stopped suddenly during work on the 400 Newton kick motor, leading to speculation that the satellite may have suffered physical damage, but NORAD radar profiles indicate that it has not exploded.

Problems with Phase 3-D are bad news for hams, who raised $1 Million for the satellite, the 40th in the series of Amateur satellites launched since 1962. Phase 3-D has been criticized for pu ting too many eggs in one basket, unlike other small ham satellites built on a low budget and more easily launched as hitch-hikers with other payloads than the multi-ton Phase 3-D.

"The core development team urges patience. The satellite is in a stable orbit in one piece, just where it is expected to be, as confirmed by NORAD. They can take lots of time to debug it where it is, and hope to restore its functionality.

"For bulletins, see the AMSAT web site.

"A number of other ham satellites remain operational, and astronauts are currently using a ham station on the International Space Station to speak with people on the ground."

Comments

This thing really isn't lost yet.

[fwc]

My understanding, being an amateur radio operator (among other things) and following this closely for the last week or so since launch, is that this satellite was designed in such a way that it automatically goes through some error-correcting procedures, such as changing radios and frequencies for telemetry and command transfers, etc. etc. etc. However the complete cycle takes days, not hours, and as such, the people at AMSAT decided to give it a chance to recover before trying more drastic means.

My understanding is that they also have the equivalent of the "big red power button" or "control-alt-delete", which they haven't even started to try yet.

UPDATES

[Bruce Perens]

Someone please moderate this up. Here are the updates to this story.

The RAM corruption on the experimental backup flight computer is a normal feature until the EDAC software "scrubbing" routine is enabled. This constantly scans RAM for errors and corrects them before they grow too large to be correctible. That software has not been loaded yet.

The 145.898 beacon was not damaged, a 440 MHz one was. The 145.898 beacon runs from the backup flight computer while the 440 MHz one runs from the primary one. That's why the backup flight computer could cause a loss of telemetry.

There have been a good deal more than 40 Amateur satellites, only the ones that actually reached orbit and transmitted get numbers, and there are Amateur satellites outside of this numbering series.

Insiders tell me to relax, have patience, and that it is really jumping the gun to think the bird is lost.

Thanks

Bruce

whats this for?

[iAlex]

Just one question. What is the purpose of this satellite? Are people with ham radios supposed to be able to brodacast back and forth to it or something? I guess I am just out of the loop on this one.

*Before* posting!

[maggard]

PLEASE - before asking 100 times the same questions regarding Ham sats remember you're on the web. Spend the 30 seconds BEFORE you post and see if you can find the answer yourself.

Google, Yahoo, Metacrawler, etc. are all more efficient then asking every time someone else to explain it for you.

ps Also check the links in the article itself.

See, this stuff is HARD!

[ceswiedler]

Launching stuff into space is a risky and difficult process. All these problems are why NASA used to build triply-redundant probes (Mariners, Voyagers, etc) and then send two or three of them.

Anyone who's ever programmed on a REAL production system will attest to the fact that it's the last few obscure bugs that are the most difficult to find. The difference between a 99.99% bug-free product and a 100% bug free one is enormous.

Why they are experimenting with CPUs

[Bruce Perens]

Hi again, Flavio,

The rad-hard CPU of choice for spaceborne equipment is the 1802. Remember the RCA COSMAC personal computer of long ago? I think they have this in silicon-on-sapphire. There are a small number of satellite hackers who still practice 1802 assembler at this late date. It should suffice to say that nobody uses this CPU for anything else any longer. So, an experiment with a modern CPU was very desirable. It looks like the problem might not be in the CPU.

Thanks

Bruce

Re:Why they are experimenting with CPUs

[Flavio]

The rad-hard CPU of choice for spaceborne equipment is the 1802. Remember the RCA COSMAC personal computer of long ago? I think they have this in silicon-on-sapphire. There are a small number of satellite hackers who still practice 1802 assembler at this late date. It should suffice to say that nobody uses this CPU for anything else any longer. So, an experiment with a modern CPU was very desirable. It looks like the problem might not be in the CPU.

Well, unfortunately I don't remember the RCA COSMAC PC because I wasn't even alive back then :)

I do remember that this exact same CPU is one exotic baby, starting with its silicon-on-sapphire CMOS fabrication. It's in the heart of the Viking, Voyager and Galileo!

As I couldn't remember anything else, I had to consult the "Great Microprocessors of the past and present" page.

Here's the text for those who actually don't wanna visit the page:

RCA 1802, weirdness at its best (1974)

The RCA 1802 was an odd beast, extremely simple and fabricated in CMOS, which allowed it to run at 6.4 MHz (at 10V, but very fast for 1974) or suspended with the clock stopped. It was an 8 bit processor, with 16 bit addressing, but the major features were it's extreme simplicity, and the flexibility of it's large register set. Simplicity was the primary design goal, and in that sense it was one of the first RISC chips.

It had sixteen 16-bit registers, which could be accessed as thirty-two 8 bit registers, and an accumulator D used for arithmetic and memory access - memory to D, then D to registers, and vice versa, using one 16-bit register as an address. This led to one person describing the 1802 as having 32 bytes of RAM and 65535 I/O ports. A 4-bit control register P selected any one general register as the program counter, while control registers X and N selected registers for I/O Index, and the operand for current instruction. All instructions were 8 bits - a 4-bit op code (total of 16 operations) and 4-bit operand register stored in N.

There was no real conditional branching (there were conditional skips which could implement it, though), no subroutine support, and no actual stack, but clever use of the register set allowed these to be implemented - for example, changing P to another register allowed jump to a subroutine. Similarly, on an interrupt P and X were saved, then R1 and R2 were selected for P and X until an RTI restored them.

A later version, the 1805, was enhanced, adding several Forth language primitives. Forth was commonly used in control applications.

Apart from the COSMAC microcomputer kit, the 1802 saw action in some video games from RCA and Radio Shack, and the chip is the heart of the Voyager, Viking and Galileo (along with some AMD 29000 bit slice processors) probes. One reason for this is that a version of the 1802 used silicon on sapphire (SOS) technology, which leads to radiation and static resistance, ideal for space operation.

Thanks for the info, Bruce!

Flavio

Re:Storage mediums in space.

[IronChef]

I can actually answer some of this. I used to work at JPL on spacecraft computers, back when I was a EE student. I did stuff like writing bootstrap code in assembler that was particular to the custom chips that comprised the computer. I also designed PCBs for the test platform.

Space is a brutal environment. Even out in the deep black between the outer planets, there are a lot of gamma rays and charged particles that would play hell with a conventional computer. Spacecraft computers are built of relatively old-fashioned components, and they are made with a special radiation hardening process. There are no 1GHz CPUs in space. We don't really need them, either; the tasks performed by robotic spacecraft are pretty simple, compared to say running Windows.

Shielding isn't the answer. Imagine a metal box around a computer. Now imagine a proton or some other relatively heavy charged particle is fired into the metal. The particle can hit an atomic nucleus and shatter it, flinging MORE particles into the hapless computer. Bad news. And shielding thick enough to protect against this is heavy, and that's bad for spacecraft too.

So they use simple, rugged components, that can usually resist a proton zipping through them. And for the times when a bit gets flipped by a particle -- this is called a Single Event Upset or SEU -- there is a TON of error correction & detection bits allocated in these computers. The system I worked on, which is the computer in the Cassini craft, used a modified Hamming code. I think that almost 1/2 the bits in every 16-bit word were allocated to EDAC.

Unfortunately I do not remember the details of how you fab a chip to be radiation resistant. Some special substrate, special transistors... I'm sure you can Google for it.