Ham Satellite Suffers Failures, Is Silent

Bruce Perens writes "The Phase 3-D satellite, renamed Oscar 40 once it reached orbit, has suffered multiple failures and has stopped transmitting telemetry. It is not yet known if the satellite is responding to commands. The main telemetry beacon on 145.898 MHz was operating poorly after separation and now may be transmitting a weak unmodulated carrier. A fuel valve appears to be sticking closed and appeared to be only 10% open after multiple operation commands. Then, the first flight burn was 3 minutes too long due to another, not-yet-understood problem, boosting the craft into a higher orbit than expected (not yet a problem in itself as the final orbit is to be even higher). A backup flight computer, itself an experiment to see if the CPU would be radiation-hard enough to survive, has its RAM corrupted every 1 or 2 days in orbit as it crosses radiation belts and currently is not set up to reboot automaticaly. The primary flight computer may have crashed. An expected watchdog timer reset did not occur, but this would be the case if the satellite was receiving some commands."

"We may have to wait until after Christmas for another reset. Controllers have not transmitted a hard reset command yet, which would work even with the flight computer crashed, as they wish to explore other options. The satellite would automaticaly cycle through a number of frequencies and antennas if it does not get any commands for 10 orbits, and controllers hope to re-establish control as this happens. A hard reset could delay that option.

"The satellite waited several years for launch due to Arianne 5's early failures and a revised accelleration profile for the booster that required a redisign of the satellite frame to take additional stress. Aging may have effected components such as the fuel valves. Telemetry stopped suddenly during work on the 400 Newton kick motor, leading to speculation that the satellite may have suffered physical damage, but NORAD radar profiles indicate that it has not exploded.

Problems with Phase 3-D are bad news for hams, who raised $1 Million for the satellite, the 40th in the series of Amateur satellites launched since 1962. Phase 3-D has been criticized for pu ting too many eggs in one basket, unlike other small ham satellites built on a low budget and more easily launched as hitch-hikers with other payloads than the multi-ton Phase 3-D.

"The core development team urges patience. The satellite is in a stable orbit in one piece, just where it is expected to be, as confirmed by NORAD. They can take lots of time to debug it where it is, and hope to restore its functionality.

"For bulletins, see the AMSAT web site.

"A number of other ham satellites remain operational, and astronauts are currently using a ham station on the International Space Station to speak with people on the ground."

Comments

Re:I wonder

[piku]

A hard reset would be physically turning the system off and on again.

ctrl-alt-delete is a soft reset.

This thing really isn't lost yet.

[fwc]

My understanding, being an amateur radio operator (among other things) and following this closely for the last week or so since launch, is that this satellite was designed in such a way that it automatically goes through some error-correcting procedures, such as changing radios and frequencies for telemetry and command transfers, etc. etc. etc. However the complete cycle takes days, not hours, and as such, the people at AMSAT decided to give it a chance to recover before trying more drastic means.

My understanding is that they also have the equivalent of the "big red power button" or "control-alt-delete", which they haven't even started to try yet.

Re:This thing really isn't lost yet.

[Bruce Perens]

One problem is the attitude of the satellite. It's probably got the high-gain antennas pointing in the opposite direction from the earth right now, because of the attitude they got it in for the burn. So, when it resets, it's supposed to use optical sun and earth sensors to orient itself. There are torque wheels and/or gyroscopes to do the physical orientation, it doesn't burn fuel for this. But the earth sensor was being blinded by the sun, so that might not work correctly, at least not in this part of the year.

So, we all cross our fingers.

Thanks

Bruce

Re:Reading this story I was thinking ....

[Bruce Perens]

No offense, but just when did manufacturers start making SSB handhelds?

In 2000. I guess you haven't seen this! 1.8 MHz through 450 MHz, all modes, self-contained AA batteries, and OK it's got a shoulder strap, but it's a handheld.

And no, I haven't missed articles on working satellites via FM. That's what I was talking about.

People can set their own challenge levels, and if they are interested they work from low to high. Having an easy satellite mode would have been a great starter for beginners and especially young people. You know young people, they look like other hams, but aren't bald with a limp :-)

People gave me the same argument about packet in the 80's, which came down to where's the sport in that? Not everybody is into Amateur Radio for the sport.

Thanks

Bruce

UPDATES

[Bruce Perens]

Someone please moderate this up. Here are the updates to this story.

The RAM corruption on the experimental backup flight computer is a normal feature until the EDAC software "scrubbing" routine is enabled. This constantly scans RAM for errors and corrects them before they grow too large to be correctible. That software has not been loaded yet.

The 145.898 beacon was not damaged, a 440 MHz one was. The 145.898 beacon runs from the backup flight computer while the 440 MHz one runs from the primary one. That's why the backup flight computer could cause a loss of telemetry.

There have been a good deal more than 40 Amateur satellites, only the ones that actually reached orbit and transmitted get numbers, and there are Amateur satellites outside of this numbering series.

Insiders tell me to relax, have patience, and that it is really jumping the gun to think the bird is lost.

Thanks

Bruce

Re:TechSat 1B (aka GO-32) Israeli Technion sat

[Bruce Perens]

And PANSAT from the Naval Postgraduate Institute in Monterey, California is similar. I think it was all built by Navy folks, but was promised to be an Amateur satellite using spread spectrum and the software to use it would be made available to Amateurs. That never happened.

Bruce?

Re:Oops

[Sly Mongoose]

Ask NASA, other space agency or people in university labs that research this stuff about proper radiation shielding for processors?

Actually, NASA and most other space agencies utilize data collected on Amateur satellites. IOW, a lot of the research comes from the Amateur community.

A Ham satellite might launch with five different types of RAM, each hardened using five different new techniques. The hardened RAM might be donated by various agencies. They get to test their RAM-hardening techniques cheap. The Amateur community gets free space-hardened RAM in exchange for the risk posed by using something not quite tried-and-true.

I, too, wish them luck. I might even blow the dust off my IC-970H and 22C/40CX!

de Gus 8P6SM (formerly active on AO10, AO13, AO16, FO20, KO23, AO27)

Re:Oops

[liberty!]

Several years ago I worked for an "engineering service contractor" for NASA, and had the opportunity to design the sensor electronics for a space-based sun-staring telescope, called Solar X-ray Imager. The task would not have been excessively difficult except that the radiation levels expected were quite high, and we had to design it to be very radiation resistant. This proved to be very expensive to do.

For certain electronic functions, there are radiation hardened integrated circuits. We used a set of CMOS devices manufactured by Harris (now Intersil) which are manufactured using a silicon-on-sapphire technology which makes them tough indeed. (The manufacturer claimed that they were radiation resistant up to "strategic levels" - to run through massive radiation doses and never glitch...) While they were very good parts, their cost was incredible. They were getting US$ 225 for a single 75hc00 equivalent device which usually costs 25 cents. Processors and memories had prices that were proportionally worse. Nobody but government bodies with the power of taxation can afford these things.

The other method of achieving radiation hardness is shielding. In our case, we had to use this in addition to the radiation hardened integrated circuits, because the telescope sensor itself was not radiation hardened (otherwise it would not have been able to see light either!) The shielding we used was made from tantalum, machined in shapes to cover the sensitive parts. Now tantalum is not inexpensive, it does not machine well, and is heavy. The weight is the limiting factor, since whatever you use you have to launch into space, and the cost per kg is very high. Shielding which is sufficient is also too heavy to launch on a secondary payload.

My thoughts? Amsat did the best they could within the budget, and the budget (raised from donations) did not provide enough money to use the premium radiation hardened parts everywhere.

Too bad...

73 de W4TI

Re:Storage mediums in space

[Bruce Perens]

Tee hee.

How well can the radiation conditions of space be replicated on earth for testing tis sort of thing?

You use an X-ray machine, a van de Graf generator, a vaccumm chamber, and so on. It doesn't get you all of the way there, but it can give you some data.

Thanks

Bruce

Re:So what?

[Andy Dodd]

Where the hell are you getting this ESA stuff from?

All they did was sell space on their rocket - The launch itself went perfectly. No evidence of government screwups there.

The satellite was designed, built, and the launch paid for with private funding. AMSAT is non-profit, but non-profit != government!

Karma and Moderation

[adolf]

As I read this article (at +2, no parent -- seems reasonable in most discussions), I'm distraught by the number of posts being displayed by Bruce Perens. Sure, he's (appearently) a smart guy who knows at least enough about the subject to author the article, but why are 10 out of 22 >+2 posts from him?

Counting the article itself as 1, that's -half- of the displayed posts, which is just silly.

Either the moderators are favoring Bruce's commentary (which is typically useful, informative, and insightful -- though not without context), or Bruce is using his +1 bonus on every post (which is -not- needed on a three-line message, no matter what it contains).

As it is, some of Bruce's replies are without displayed parent posts, and in (at least) one instance he appears to have replied to himself (which means there's at least two <+2 posts which need attention).

Just a random musing...

Re:Karma and Moderation

[Bruce Perens]

I read at -1 and see everyone's posts. If not many non-bruce posts are displayed, it's because you've set your threshold too high. If you are seeing them without parent posts (except for the update post) it's because the answers were moderated higher than the questions.

Until we got a few other hams posting there were few people here who had any information on the topic, and I answered all sensible questions, as I'll usually do on my own articles. I hardly think this is improper.

Thanks

Bruce

whats this for?

[iAlex]

Just one question. What is the purpose of this satellite? Are people with ham radios supposed to be able to brodacast back and forth to it or something? I guess I am just out of the loop on this one.

Re:whats this for?

[MsWillow]

Yup. It picks up signals, usually on one RF band, then retransmits them back again on another. The ones I used to use were the Russian satellites, with an uplink on the 10-Meter band, and a downlink on 2Meters. My station was *way* underequipped to run the 2M / 440MHz birds, and I had no equipment at all any higher than 440MHz.

The antennas were huge, the feedline expensive, the low-noise preamps finiky, and compensating for the doppler shift as the bird orbitted was, well, entertaining - but it *could* give people a chance to try to talk all over the world (well, over a lot of it, depending on what was visible to the satellite at the time).

However, once I got a reliable, steady Net connection (640kbps bi-di DSL), my interest in Ham radio rather waned. I gave most of the equipment to a fellow Ham, who was more into that than I was.

Re:whats this for?

[Bruce Perens]

I have a nice high-speed connection and still find interest. I've a ground station with azimuth-elevation rotator on dual crossed beams for 145 and 440 MHz. And of course I do regular VHF and HF as well. It's fun to be able to communicate without the internet or a phone network, and the technical skills one can acquire this way are substantial. Hamming is what got me started in a technical career.

Thanks

Bruce

*Before* posting!

[maggard]

PLEASE - before asking 100 times the same questions regarding Ham sats remember you're on the web. Spend the 30 seconds BEFORE you post and see if you can find the answer yourself.

Google, Yahoo, Metacrawler, etc. are all more efficient then asking every time someone else to explain it for you.

ps Also check the links in the article itself.

Can we use Iridium?

[LM741N]

I think there's at least 50 satellites up there for grabs. Pick your favorite color.

See, this stuff is HARD!

[ceswiedler]

Launching stuff into space is a risky and difficult process. All these problems are why NASA used to build triply-redundant probes (Mariners, Voyagers, etc) and then send two or three of them.

Anyone who's ever programmed on a REAL production system will attest to the fact that it's the last few obscure bugs that are the most difficult to find. The difference between a 99.99% bug-free product and a 100% bug free one is enormous.

Re:See, this stuff is HARD!

[Bruce Perens]

NASA has launched lots of duds. You learn this by doing that. This is the 40th Amateur satellite to reach orbit, and while it's the most ambitious, these folks know what they are doing.

Thanks

Bruce

Why they are experimenting with CPUs

[Bruce Perens]

Hi again, Flavio,

The rad-hard CPU of choice for spaceborne equipment is the 1802. Remember the RCA COSMAC personal computer of long ago? I think they have this in silicon-on-sapphire. There are a small number of satellite hackers who still practice 1802 assembler at this late date. It should suffice to say that nobody uses this CPU for anything else any longer. So, an experiment with a modern CPU was very desirable. It looks like the problem might not be in the CPU.

Thanks

Bruce

Re:Why they are experimenting with CPUs

[Flavio]

The rad-hard CPU of choice for spaceborne equipment is the 1802. Remember the RCA COSMAC personal computer of long ago? I think they have this in silicon-on-sapphire. There are a small number of satellite hackers who still practice 1802 assembler at this late date. It should suffice to say that nobody uses this CPU for anything else any longer. So, an experiment with a modern CPU was very desirable. It looks like the problem might not be in the CPU.

Well, unfortunately I don't remember the RCA COSMAC PC because I wasn't even alive back then :)

I do remember that this exact same CPU is one exotic baby, starting with its silicon-on-sapphire CMOS fabrication. It's in the heart of the Viking, Voyager and Galileo!

As I couldn't remember anything else, I had to consult the "Great Microprocessors of the past and present" page.

Here's the text for those who actually don't wanna visit the page:

RCA 1802, weirdness at its best (1974)

The RCA 1802 was an odd beast, extremely simple and fabricated in CMOS, which allowed it to run at 6.4 MHz (at 10V, but very fast for 1974) or suspended with the clock stopped. It was an 8 bit processor, with 16 bit addressing, but the major features were it's extreme simplicity, and the flexibility of it's large register set. Simplicity was the primary design goal, and in that sense it was one of the first RISC chips.

It had sixteen 16-bit registers, which could be accessed as thirty-two 8 bit registers, and an accumulator D used for arithmetic and memory access - memory to D, then D to registers, and vice versa, using one 16-bit register as an address. This led to one person describing the 1802 as having 32 bytes of RAM and 65535 I/O ports. A 4-bit control register P selected any one general register as the program counter, while control registers X and N selected registers for I/O Index, and the operand for current instruction. All instructions were 8 bits - a 4-bit op code (total of 16 operations) and 4-bit operand register stored in N.

There was no real conditional branching (there were conditional skips which could implement it, though), no subroutine support, and no actual stack, but clever use of the register set allowed these to be implemented - for example, changing P to another register allowed jump to a subroutine. Similarly, on an interrupt P and X were saved, then R1 and R2 were selected for P and X until an RTI restored them.

A later version, the 1805, was enhanced, adding several Forth language primitives. Forth was commonly used in control applications.

Apart from the COSMAC microcomputer kit, the 1802 saw action in some video games from RCA and Radio Shack, and the chip is the heart of the Voyager, Viking and Galileo (along with some AMD 29000 bit slice processors) probes. One reason for this is that a version of the 1802 used silicon on sapphire (SOS) technology, which leads to radiation and static resistance, ideal for space operation.

Thanks for the info, Bruce!

Flavio

Re:Why they are experimenting with CPUs

[Flavio]

Argh. I probably made a typo and Slashdot ate my link. Here it goes:

Great Microprocessors of the Past and Present (V 12.0.0)

The RCA 1802's in there.

Flavio

Re:Storage mediums in space.

[IronChef]

I can actually answer some of this. I used to work at JPL on spacecraft computers, back when I was a EE student. I did stuff like writing bootstrap code in assembler that was particular to the custom chips that comprised the computer. I also designed PCBs for the test platform.

Space is a brutal environment. Even out in the deep black between the outer planets, there are a lot of gamma rays and charged particles that would play hell with a conventional computer. Spacecraft computers are built of relatively old-fashioned components, and they are made with a special radiation hardening process. There are no 1GHz CPUs in space. We don't really need them, either; the tasks performed by robotic spacecraft are pretty simple, compared to say running Windows.

Shielding isn't the answer. Imagine a metal box around a computer. Now imagine a proton or some other relatively heavy charged particle is fired into the metal. The particle can hit an atomic nucleus and shatter it, flinging MORE particles into the hapless computer. Bad news. And shielding thick enough to protect against this is heavy, and that's bad for spacecraft too.

So they use simple, rugged components, that can usually resist a proton zipping through them. And for the times when a bit gets flipped by a particle -- this is called a Single Event Upset or SEU -- there is a TON of error correction & detection bits allocated in these computers. The system I worked on, which is the computer in the Cassini craft, used a modified Hamming code. I think that almost 1/2 the bits in every 16-bit word were allocated to EDAC.

Unfortunately I do not remember the details of how you fab a chip to be radiation resistant. Some special substrate, special transistors... I'm sure you can Google for it.

Re:What is the normal attrition rate?

[Bruce Perens]

They were really pushing the technology. The thing has an ammonia arc-jet motor for orbit maintainance. Such a thing has never been tried. That's the smaller motor, the kick-motor is more conventional. And of course there are tons of experiments like orbit-to-ground laser communications, TV cameras, and a full compliment of Amateur radio repeaters. You're supposed to be able to talk through this with a walkie-talkie.

But yes, this is a large number of initial failures for a ham satellite. Most of them are much simpler. But all of them have things break and they patch around it and go on using the bird. The same is true for commercial communications satellites.

Thanks

Bruce

AMSAT didn't pay License Fees! Ham HQ Raided!

[tarbabyxxxx]

It has been discovered that AMSAT didn't pay license fees to the RIAA and M$, so the copy protection in the hardware and software turned off the Satellite.
RIAA and M$ have stated, "We can not allow hacker ham radio operators to steal our intellectual property." They will allow AMSAT Corp access again for 26 Million Dollars for a 1 year license fee.
They also want to inspect the satellite HD contents because the Napster server says they have a MP3 of Rocket man and a AVI video of the Moon Launch on the HD

In a related story the FBI has raided the ARRL ham radio HQ looking for antennas and radios that could be used to listen to frequencies, they also confiscated secret PSK31 transmitters that could be used by spies to send signals to spy agencies.

Also discovered was a secret world wide system to track people, cars, boats and planes without the knowledge of the passengers. The secret program, called APRS, violates USC 3, 21 and Janet Reno indicated today that all licensed ham radio operators will be investigated. You may remember that the Branch Davidians used ham radio during the seige at Waco.

Re:More evidence

[joedumb]

actually, read the article and check the link out. the satellite was built by amsat and launched on an ESA rocket. nothing to do with NASA. wake up.

Re:What is the normal attrition rate?

[Detritus]

It is common for subsystems to fail on satellites. That is why they usually have redundant systems for critical functions. On scientific satellites, there may be many experiment packages on the satellite. You normally don't hear about one of them failing, unless you have an interest in that experiment and read the status reports.

Re:Telemetry

[Bruce Perens]

You can't receive it if it's not sending. One of the ground controllers has deep experience in writing software modems and did in fact design the RF link-layer used for much of digital Amateur radio. For the hams in the audience, this is G3RUH. Actually, if this particular beacon was sending properly I could get it, and decode it, at home with no problem.

Thanks

Bruce

Re:Reading this story I was thinking ....

[Bruce Perens]

Sigh. I don't know about you, but this really un-made my day. AMSAT has to be about the bravest amateur experimenter group out there. They envisioned a new day for ham radio powered by a super satellite that would give you worldwide range with a walkie-talkie and a handheld beam. We have that now to some extent, but not as reliably and not for as many people as they were planning. Well, they aren't quitters, and if this bird turns out to be a loss (not necessarily the case yet) we'll see the same capability in a series of smaller, cheaper, easier-to-launch birds.

We did have some good news for hams this week - the space station ham rig is running great, and Germany just lowered the Morse Code requirement to an easy 5 words-per-minute like the U.S. Now, we just have to get rid of that code requirement entirely.

Thanks

Bruce

Re:Reading this story I was thinking ....

[Bruce Perens]

I was wondering if anyone would take me to task for not considering the Wilderness Radio and Norcal stuff to be handhelds.

Oh, I suppose you can classify that 1962 Tuna Tin radio as a handheld, too :-) No doubt there is military stuff from the Korean war era, as well.

Thanks

Bruce

Re:It's still impressive, even if it fails

[Bruce Perens]

Their web site is amsat.org . If you want to learn about what they are doing, go to arrl.org and buy the "Satellite Anthology". I think ARRL or AMSAT have some conference proceedings available, as well.

If you want to talk with an AMSAT developer directly, try Bdale Garbee bdale@debian.org . He did the GPS experiment and possibly some other stuff, and was part of the pre-launch prep team in Ghana.

Thanks

Bruce