Slashdot Mirror
Hardware Horrors that Firmware Upgrades Would've Fixed?
Anonymous Coward writes "I just started working for a startup that
is developing a new product, which is going to have software bundled
with hardware. Our company outsourced the hardware and firmware
development. I reviewed the hardware product requirements and I
noticed that the hardware will not support firmware upgrades from the
PC. I am concerned that once we ship the product, bugs or
interoperability issues will appear in the field and we won't have
anyway to fix the problem short of a product recall. I have some of
the management team convinced we need to change this requirement but
not the person who has the authority to make the change. I'm looking
for examples of past companies that got bit by a similar mistake and
any other items that will help me convince the decision maker."
Nobody is perfect, so why do we assume that we can design hardware
that is? If it's one thing that our current experiences with software
have shown it's that sometimes, an applications may take more than one
version before it is perfect. Before, our ability to change hardware
coding made getting perfect products out the door important, because
recalls were expensive. Today, we have smarter hardware, which can be
relatively simple to update. The cost of recalls, however,
have not changed. So for what reason would a hardware company balk
at making the need for a recall a thing of the past?
First thing that comes to mind is the Apple iPod Sleep Issue.
Direct TV had this type of problem when their DSS "F" and "H" cards were hacked. They are now on their 3rd generation of smartcards.
Palm pilots (maybe not all, but many and mine) are upgradable. They do this cause they know that software needs to be upgradable. Just about ALL software has bugs.
If your product goes out and has bugs in it and it causes people loss of data, or worse, you will build yourself a reputation. Sort of like the release of Windows 95 did for Microsoft. No matter how they try they now have a reputation for buggy crashing software. Even if your product is the best on the market if it gets a reputation of bugginess, it will be harder to over come if people have to BUY an upgrade to fix it or BUY a whole new device instead of download bug fixes that makes it worse. While many people will do it they do it till something better comes a long.
Features are nice, but FIXES are essential to people staying with a product. I stopped useing Microsoft products whenever possible cause I'd rather use a *nix flavor that is less likely to crash on me while typing. This was after my experiences with Win 3.1/95/98 and NT 4.0. I am not impressed enought and do not trust Win2k, Me, or XP. They just don't have the reputation that Sun, BSD, UNIX and Linux have built. I never used a Windows BOX that could stay up for 275 days, but I have seen and used many Sun, BSD, and Linux boxes that were. In fact many of the IT staff people that I have worked with would not support a windows box if you did not reboot it atleast once a day. Also most people I knew or know who do not reboot about once a day end up rebooting when the system crashes.
So ask your boss, or the person who makes that decision, "Do you want to be a company that works with the consumer to fix the problem and help make their experience with the product better, or do you want to be one that gets a reputation for bad buggy software?"
Only 'flamers' flame!
I once had a Pentium 133 that worked properly in all ways except one: When I installed an IDE CD-ROM, the machine could not see the drive. After much digging, I found that the problem was in the AMI BIOS on the motherboard. I bought a new BIOS from Mr. BIOS, and that fixed the problem completely. If that board had a flashable BIOS, the problem could have been solved without changing an IC.
If tits were wings it'd be flying around.
Voyetra/Turtle Beach's Audiotron home MP3 player has an easily flashable firmware. The system ships as just a player, but the firmware adds functionality such as advanced management and web-based control. Point out to your manager that it's not just for problems - it can be used for features as well!
Not so.
Many of the 12x and 16x units wouldn't read CDRs, which would have made them near worthless in today's world. For most of them, the fix was as simple as slowing the speed and trying another pass at reading before giving up.
For many brands, a flash upgrade was all it took to fix these and give them value again. The upgrade was made available to consumers who suddenly had brand loyalty for what's normally a pretty ambigously branded piece of hardware.
For many other brands, the units became bargain bin fodder and left a lot of consumers pissed off at what they thought was broken hardware.
Back around August 2001, that famous MSTD, the CodeRed worm was swarming across the Internet. One side effect of it's probing behavior was to trigger a bug in certain models of Cisco DSL modems. The result was a crashed modem.
The user could power cycle the modem, but it would die again shortly when their neighbor's infected system probed them. This was a catastrophe for the ISP's involved.
This effected many people, more than a million I believe.
Cisco put out a corrective CD-ROM that reflashed the CPE with fixed firmware. If this had not been possible, Cisco would probably have ended up paying to replace all those modems. Running off some CD-ROMs was a lot cheaper.
The firmware upgrades are what saved this poduct/
Read the readme.txt for a long history of bugs that have been fixed through firmware upgrades - originally I had problems with it, and it was a firmware upgrade that fixed it - the saving grace for this product.
Having firmware upgrades for a product is a very prudent thing to do. Anybody that doesn't think so is arrogant.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
(Saga which I read in Science magazine many years ago.)
The 1975 Viking Mars lander was expected to last only a few months on the Mars surface; battery life was the limiting factor. The battery lasted longer than expected, but eventually the Sun would come between Earth and Mars. With the lander fully powered the battery would be dead by the time Earth came back into view.
NASA (or maybe it was JPL) thought of reprogramming the Viking controller to power down, wait a few months, then power back up. (The power-up had to be automatic; in power-down mode there was no communication with Earth.)
Viking had reprogrammable firmware, but only for pre-flight programming. Reprogramming during the mission hadn't been anticipated, so the diagnostic bus through which the ROM was reprogrammed was removable. There was no record of whether the Viking which NASA had sent to Mars had that bus or not! Nor was there a way to detect bus presence.
On the chance that the bus was installed, new code was tested on an Earth-bound copy of the Viking which had the bus, then uploaded. The Mars lander did have the bus, the code worked, and NASA got several additional months of operation from Viking.
satellites are kindof an extreme case of hard-to-service-in-the field. Especially when the one-way shipping cost = 10x the cost of the satellite (we did small sats). So, getting uploadable code right was a top priority. Radiation could damage most of our memory and upset (reset) our processor. The EPROM held multiple versions of our main software. A small amount of expensive rad-hard PROM code performed checksums on the EPROM and executed an uncorrupted copy. We didn't have FLASH memory, so any new code was stored in RAM, which was even more susceptable to bit hits than the EPROM. Getting the code up to the satellite was something we had to plan well... we had a 9600 baud link that we could use only ~7 minutes out of each 90 minute orbit. At this rate, uploading a new version of code would take quite a while and would have to be performed every time a radiation-induced reset occured. It was going to be a maintenance/reliablity nightmare!! Unfortuantly, our rocket blew up and we never got to try out the satellite.
HIV Crosses Species Barrier... into Muppets
Perhaps that's why they've introduced microcode update functionality (typically done by your BIOS, but there's stuff in the Linux kernel to allow you to do it from the OS).
--
Flash is great, but make sure that it is failsafe.
Example -- I had 2 USR Courier modems in the mid-90s that were 'flash upgradeable'... Once the V90 standard was stable, I flashed one of them.
And killed it.
The modem was supposed to be flashable, and I did everything right, but USR had got the hardware wrong. They replaced both modems at their cost, and both the new modems flashed correctly.
There are also frequently warnings on motherboard flash programs and Palm flash programs to this effect -- if you screw up the flash, you will have a product that you *cannot* use, and must send in/replace to get functioning again.
With that in mind, make sure your product either has a flash loader, or default software image in ROM that can be accessed if the flash image is corrupt, either automatically, or by a jumper. Otherwise you may end up with angry customers who have upgraded paperweights.
Brian.
Dell's recent laptops/desktops have flash bios updates that come out every few weeks. They fix bugs, but also introduce new ones. Be careful with this type of thing! Just because it's easy to make an update doesn't mean that it's okay to skimp on testing because a fix is easy to shove to the customer.
It's fairly obvious that they failed to test the Inspiron 8100 A08 Bios in a system with no floppy drive (e.g. two battery configuration). Once the machine tries to access the floppy drive, the hard drive activity light (shared with the floppy, actually) will not turn off until you suspend or power down. Of course, an antivirus program initializing when you start Windows will access the floppy (that isn't plugged in) and cause the light to turn on forever. While this isn't an issue that causes BSOD (this update actually fixed a few of those bugs), it's definitely an annoyance.
/ \
\ / ASCII ribbon campaign for peace
x
/ \
I don't know much about the details of any of these (other than how easy it was to kill a Pentium with the f00f bug), but when I do cat /proc/cpuinfo, it mentions all of them. Presumably the Linux kernel has workarounds for them all.
The economics of your company are probably quite a bit different from the auto industry; your volume is probably several orders of magnitude less, to name one thing. But you have to consider the loss of goodwill if customers have to pull hardware and ship it to you for firmware fixes, instead of taking 20 minutes to download and install a new patch. You might also consider the benefits of being able to sell firmware functionality upgrades for units already installed; the customers will love you for teaching their old dog a new bunch of tricks.
Scientists restrict study to entire physical universe; creationist