Crowdfunded, Solar-powered Spacecraft Goes Silent

Last week saw the successful launch of the Planetary Society's LightSail spacecraft, the solar-powered satellite that runs Linux and was crowdfunded on Kickstarter. The spacecraft worked flawlessly for two days, but then fell silent, and the engineering team has been working hard on a fix ever since. They've pinpointed the problem: a software glitch. "Every 15 seconds, LightSail transmits a telemetry beacon packet. The software controlling the main system board writes corresponding information to a file called beacon.csv. If you're not familiar with CSV files, you can think of them as simplified spreadsheets—in fact, most can be opened with Microsoft Excel. As more beacons are transmitted, the file grows in size. When it reaches 32 megabytes—roughly the size of ten compressed music files—it can crash the flight system." Unfortunately, the only way to clear that CSV file is to reboot LightSail. It can be done remotely, but as anyone who deals with crashing computers understands, remote commands don't always work. The command has been sent a few dozen times already, but LightSail remains silent. The best hope may now be that the system spontaneously reboots on its own.

Comments

Seriously?

[Anonymous Coward]

I’m usually the first to defend others when some bug like this makes it through testing. Hindsight always being 20/20, only takes one bug amongst a million good bits of code, etc. But this just seems like something that even basic testing should have caught.

Did they not run this thing on the ground for a few weeks? That’s just basic testing, especially for something that is going to be inaccessible for a while. Also that some critical bit of processing relies on stuff being written (and then presumably read back from) a csv file is very worrying.

This sounds like some very shoddy work.

Re:Seriously?

[Mr D from 63]

Testing might have found it, but I'd say that regardless of testing they should assume something bad will happen with the software and have a mechanism in place to force reboot & update on a locked up system. Maybe they thought they did. Its a shame if they can't get it fixed.

Re:Seriously?

[mnooning]

As a retired QA guy, I can tell you that checking that no files can grow without bound is standard fare. Same with exercising all code for long periods of time, as you pointed out. That means there was not a single experienced QA guy on the team.

By the way, CSV was the golden standard for many years. Given the tight compactness/memory budget that space projects have, CVS with it's small foot print might well be the logical choice.

Re:Seriously?

[macs4all]

Testing might have found it, but I'd say that regardless of testing they should assume something bad will happen with the software and have a mechanism in place to force reboot & update on a locked up system. Maybe they thought they did. Its a shame if they can't get it fixed.

Speaking as an embedded developer, this is completely inexcusable.

Not having a Watchdog, PLUS not making the limited-filesize log file "roll-over", is clearly Amateur-Hour stuff. Who wrote this code, anyway? An eight year old???

Next we're going to hear that they bricked it with a software update, because they didn't think they needed to checksum the uploads, or provide enough RAM to hold the updated code before they re-flashed the OS, or something similar.

Pathetic. They deserve to lose their spacecraft.

Fortunately, if extraterrestrials discover the floating hulk of this abomination, they will (rightly) conclude that there is no intelligent life worth exploiting on this planet, and will decide not to enslave us...

Re:Seriously?

[amicusNYCL]

Not having a Watchdog, PLUS not making the limited-filesize log file "roll-over", is clearly Amateur-Hour stuff. Who wrote this code, anyway? An eight year old???

It's not even who wrote it, it's who designed it. Reading the summary actually made me angry that there is a group of people out there somewhere with the ability to build, launch, and track a satellite but without the common sense to recognize that they're creating a system that will grow infinitely in size without a mechanism to clear that data out. Does the satellite have unlimited storage space available? No? Then how about designing a way to monitor and clear the data other than saving it in /tmp?

Pathetic. They deserve to lose their spacecraft.

They definitely do. And no amount of descriptions of a CSV file meant for a grade school kid, or saying that 32MB is about the size of 10 songs, is going to minimize the schadenfreude that I'm feeling. Such a basic design error and they never even bothered to run tests for a significant period of time before putting the damn thing in space.

Way to go, LightSail team. I dub thee LightFail.

Re:Seriously?

[edxwelch]

Not to mention why are they using a text file in the first place? If it's going to grow more than a megabyte it should be stored as binary. I suppose they thought it was cool that they could load in MS Excel

CSV

[Anonymous Coward]

I know the average IQ at /. has gone down over the years, but I think the explanation of what a CSV file is is slightly too much dumbing down.

Re:CSV

[ArcadeMan]

I think the "32 megabytes—roughly the size of ten compressed music files" part is even more insulting.

Re:CSV

[gstoddart]

Honestly, I'm surprised they didn't try to define space, Linux, and solar.

This sounds like someone failed to run a bench test where the system was up and running for an extended period of time.

Which strikes me as utterly bizarre.

Comment removed

[account_deleted]

Comment removed based on user account deletion

How embarrasing

[Tyrannosaur]

You'd think that something as small as 32MB would have been tested before they launched the thing... It doesn't sound like it takes very long to fill up 32MB either

Systems Administration 101

[plopez]

Roll your log files. I smell a DevOps debacle.

Re:Systems Administration 101

[prefec2]

Any competent software designer and developer should have known basic rules of embedded systems. One of them: Do not use dynamic memory (and files are just alike). If you need space all the space you need must be determined at compile or design time. BTW why store all this data in the device? This should have been (if at all) implemented as an round robbing database. Yes that overwrites old data, but who cares? If you need all the data you should have calculated the amount for the complete mission and reserved enough memory for that. And why did they use a CSV file? Are their physicists?

Mebibyte is an idiotic term

[Anonymous Coward]

and you are an idiot for using it.

Re:Mebibyte is an idiotic term

[David_Hart]

Just because you don't like the term doesn't make it wrong. Highjacking SI prefixes and changing their meaning is wrong and has led to countless problems.

And historical meanings shouldn't be changed simply so that marketing speak can be used to sell less at the same price.

I love how 1 MB of RAM is 1048576 bytes but 1 MB of storage is now 1000000 bytes of storage, simply because the hard-drive industry decided that they could make more money by using the same term, change the historical meaning in the computing industry from base-2 to base-10 (essentially downsizing the actual storage), and charging the same amount.

Either convert totally to GiB, MiB, etc. for everything computer related or stick with the old convention. It's when you are mixing the two in a particular context (i.e. computers) where you run into problems.

Re:UAT

[itzly]

Well, how do you test it before you're happy ? If the beacon is 40 bytes, and transmitted every 15 seconds, it would take half a year before you fill up 32 MB. That's a long time for testing.

This is the kind of mistake you shouldn't even make in the first place.

Re:What the computer needs is ...

[plopez]

No. They need programmers and sysadmins that knew that they were doing. E.g. roll log files and/or put logs on a non-critical partition. Systems Administration 101 for systems where memory and disk space are at a premium. It was a rookie mistake.