Slashdot Mirror
Hardware Horrors that Firmware Upgrades Would've Fixed?
Anonymous Coward writes "I just started working for a startup that
is developing a new product, which is going to have software bundled
with hardware. Our company outsourced the hardware and firmware
development. I reviewed the hardware product requirements and I
noticed that the hardware will not support firmware upgrades from the
PC. I am concerned that once we ship the product, bugs or
interoperability issues will appear in the field and we won't have
anyway to fix the problem short of a product recall. I have some of
the management team convinced we need to change this requirement but
not the person who has the authority to make the change. I'm looking
for examples of past companies that got bit by a similar mistake and
any other items that will help me convince the decision maker."
Nobody is perfect, so why do we assume that we can design hardware
that is? If it's one thing that our current experiences with software
have shown it's that sometimes, an applications may take more than one
version before it is perfect. Before, our ability to change hardware
coding made getting perfect products out the door important, because
recalls were expensive. Today, we have smarter hardware, which can be
relatively simple to update. The cost of recalls, however,
have not changed. So for what reason would a hardware company balk
at making the need for a recall a thing of the past?
First thing that comes to mind is the Apple iPod Sleep Issue.
Direct TV had this type of problem when their DSS "F" and "H" cards were hacked. They are now on their 3rd generation of smartcards.
Palm pilots (maybe not all, but many and mine) are upgradable. They do this cause they know that software needs to be upgradable. Just about ALL software has bugs.
If your product goes out and has bugs in it and it causes people loss of data, or worse, you will build yourself a reputation. Sort of like the release of Windows 95 did for Microsoft. No matter how they try they now have a reputation for buggy crashing software. Even if your product is the best on the market if it gets a reputation of bugginess, it will be harder to over come if people have to BUY an upgrade to fix it or BUY a whole new device instead of download bug fixes that makes it worse. While many people will do it they do it till something better comes a long.
Features are nice, but FIXES are essential to people staying with a product. I stopped useing Microsoft products whenever possible cause I'd rather use a *nix flavor that is less likely to crash on me while typing. This was after my experiences with Win 3.1/95/98 and NT 4.0. I am not impressed enought and do not trust Win2k, Me, or XP. They just don't have the reputation that Sun, BSD, UNIX and Linux have built. I never used a Windows BOX that could stay up for 275 days, but I have seen and used many Sun, BSD, and Linux boxes that were. In fact many of the IT staff people that I have worked with would not support a windows box if you did not reboot it atleast once a day. Also most people I knew or know who do not reboot about once a day end up rebooting when the system crashes.
So ask your boss, or the person who makes that decision, "Do you want to be a company that works with the consumer to fix the problem and help make their experience with the product better, or do you want to be one that gets a reputation for bad buggy software?"
Only 'flamers' flame!
I once had a Pentium 133 that worked properly in all ways except one: When I installed an IDE CD-ROM, the machine could not see the drive. After much digging, I found that the problem was in the AMI BIOS on the motherboard. I bought a new BIOS from Mr. BIOS, and that fixed the problem completely. If that board had a flashable BIOS, the problem could have been solved without changing an IC.
If tits were wings it'd be flying around.
Voyetra/Turtle Beach's Audiotron home MP3 player has an easily flashable firmware. The system ships as just a player, but the firmware adds functionality such as advanced management and web-based control. Point out to your manager that it's not just for problems - it can be used for features as well!
Not so.
Many of the 12x and 16x units wouldn't read CDRs, which would have made them near worthless in today's world. For most of them, the fix was as simple as slowing the speed and trying another pass at reading before giving up.
For many brands, a flash upgrade was all it took to fix these and give them value again. The upgrade was made available to consumers who suddenly had brand loyalty for what's normally a pretty ambigously branded piece of hardware.
For many other brands, the units became bargain bin fodder and left a lot of consumers pissed off at what they thought was broken hardware.
Word of caution though, and I'm sure you've heard it before. Don't upgrade the firmware unless you are having problems. One time I updated the firmware on one of these routers, and ended up breaking more things than I was trying to fix! I called up Netopia's support line, and apparently this upgrade was a buggy one, it had only been posted on the FTP site for a few hours when they found the bugs. Bad luck for me trying to update during that time though. But hey, those are the breaks.
This was a combination of user error and firmware issues, but we had a disk array lose its mind (and 30 GB of RAIDed data) once. While the tech shouldn't have turned the array off, if there weren't a firware bug, it would have flushed the cache and we wouldn't have lost the data. I would up spending a weekend flipping bits to get back half the data. Not Happy.
that you can update the firmware.. toshiba, sony, ect.. their DVD players all accept firmware updates reason being is that newer movies may or may not play properly as the software gets newer. problems range from lockups to pixelation, garbage on the screen. im an electronics tech, and I see this daily. so, firmware updates are VERY important. on older units, we have to loterally remove the EEPROM and replace it. new ones, an interface and a serial cable is all thats needed. of course, only authorized service centers have this hardware/software. but, your correct... the ability to update the firmware will be VERY helpful. and it makes the life of the tech easier :)
hope this helps you.
Back around August 2001, that famous MSTD, the CodeRed worm was swarming across the Internet. One side effect of it's probing behavior was to trigger a bug in certain models of Cisco DSL modems. The result was a crashed modem.
The user could power cycle the modem, but it would die again shortly when their neighbor's infected system probed them. This was a catastrophe for the ISP's involved.
This effected many people, more than a million I believe.
Cisco put out a corrective CD-ROM that reflashed the CPE with fixed firmware. If this had not been possible, Cisco would probably have ended up paying to replace all those modems. Running off some CD-ROMs was a lot cheaper.
The firmware upgrades are what saved this poduct/
Read the readme.txt for a long history of bugs that have been fixed through firmware upgrades - originally I had problems with it, and it was a firmware upgrade that fixed it - the saving grace for this product.
Having firmware upgrades for a product is a very prudent thing to do. Anybody that doesn't think so is arrogant.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
(Saga which I read in Science magazine many years ago.)
The 1975 Viking Mars lander was expected to last only a few months on the Mars surface; battery life was the limiting factor. The battery lasted longer than expected, but eventually the Sun would come between Earth and Mars. With the lander fully powered the battery would be dead by the time Earth came back into view.
NASA (or maybe it was JPL) thought of reprogramming the Viking controller to power down, wait a few months, then power back up. (The power-up had to be automatic; in power-down mode there was no communication with Earth.)
Viking had reprogrammable firmware, but only for pre-flight programming. Reprogramming during the mission hadn't been anticipated, so the diagnostic bus through which the ROM was reprogrammed was removable. There was no record of whether the Viking which NASA had sent to Mars had that bus or not! Nor was there a way to detect bus presence.
On the chance that the bus was installed, new code was tested on an Earth-bound copy of the Viking which had the bus, then uploaded. The Mars lander did have the bus, the code worked, and NASA got several additional months of operation from Viking.
satellites are kindof an extreme case of hard-to-service-in-the field. Especially when the one-way shipping cost = 10x the cost of the satellite (we did small sats). So, getting uploadable code right was a top priority. Radiation could damage most of our memory and upset (reset) our processor. The EPROM held multiple versions of our main software. A small amount of expensive rad-hard PROM code performed checksums on the EPROM and executed an uncorrupted copy. We didn't have FLASH memory, so any new code was stored in RAM, which was even more susceptable to bit hits than the EPROM. Getting the code up to the satellite was something we had to plan well... we had a 9600 baud link that we could use only ~7 minutes out of each 90 minute orbit. At this rate, uploading a new version of code would take quite a while and would have to be performed every time a radiation-induced reset occured. It was going to be a maintenance/reliablity nightmare!! Unfortuantly, our rocket blew up and we never got to try out the satellite.
HIV Crosses Species Barrier... into Muppets
I have a Garmin eMap. The flash upgrade feature is great, I probably would've had to buy a whole new unit to get the kind of features they've packed into the latest firmware update.
The list of corrections and features they've added to this thing is amazing. The eMap was useful before, now it's downright a necessity when I'm traveling.
Perhaps that's why they've introduced microcode update functionality (typically done by your BIOS, but there's stuff in the Linux kernel to allow you to do it from the OS).
--
The Rio 500 firmware started out at 1.0. The last released firmware was 2.15, and there were plenty of releases inbetween - go figure. Apart from fixing a few battery bugs, they also added support for 64Mb Smart Media cards and ID3 tags, both extremely welcome.
On the other hand, the BIOS for my motherboard requires a chip swap to upgrade. Fortunately, I haven't discovered a serious reason to do this yet, although there are some annoying DMA issues I'd rather fix. But am I gonna mess around ordering and fitting a new chip or will I simply upgrade the board (to another brand)? You got it...
Ade_
/
Big Bubbles (no troubles) - what sucks, who sucks and you suck
Ok, anon coward. Ask me.
I'm not going to name names now, but I used
to work for a company that developed embedded
hardware devices. At least they had a method for
upgrading these devices from the host PC, but
some versions were old or borked because they
were development versions. These units were
essentially not field-upgradeable. a complete
"pita" and PR disaster when these things needed to be fixed. feel free to email for specifics.
At this point, I would not design any embedded
system without a safe way to do field upgrades.
(What the hell is wrong with the lameness filter?)
Greg
galaiama@saunasamaaoakae.org
I can't find my car keys. (no a's in email)
Flash is great, but make sure that it is failsafe.
Example -- I had 2 USR Courier modems in the mid-90s that were 'flash upgradeable'... Once the V90 standard was stable, I flashed one of them.
And killed it.
The modem was supposed to be flashable, and I did everything right, but USR had got the hardware wrong. They replaced both modems at their cost, and both the new modems flashed correctly.
There are also frequently warnings on motherboard flash programs and Palm flash programs to this effect -- if you screw up the flash, you will have a product that you *cannot* use, and must send in/replace to get functioning again.
With that in mind, make sure your product either has a flash loader, or default software image in ROM that can be accessed if the flash image is corrupt, either automatically, or by a jumper. Otherwise you may end up with angry customers who have upgraded paperweights.
Brian.
In both the "MIO" onboard printer network cards and the JetDirect external print servers there was a vulnerability to large ping packets. You could ping them from any win95 box: "ping -l 50000 hplj4_23" and they would print out an error message and then lock up! You had to physically power cycle the printer or print-server before they'd work again. The latest flashes fix this problem, and the HP software let's you flash them remotely.
Dell's recent laptops/desktops have flash bios updates that come out every few weeks. They fix bugs, but also introduce new ones. Be careful with this type of thing! Just because it's easy to make an update doesn't mean that it's okay to skimp on testing because a fix is easy to shove to the customer.
It's fairly obvious that they failed to test the Inspiron 8100 A08 Bios in a system with no floppy drive (e.g. two battery configuration). Once the machine tries to access the floppy drive, the hard drive activity light (shared with the floppy, actually) will not turn off until you suspend or power down. Of course, an antivirus program initializing when you start Windows will access the floppy (that isn't plugged in) and cause the light to turn on forever. While this isn't an issue that causes BSOD (this update actually fixed a few of those bugs), it's definitely an annoyance.
/ \
\ / ASCII ribbon campaign for peace
x
/ \
If you have bosses stupid enough to bet the company on this, their business and that of the unfortunates who invested, but you have no reason to bet your career with them.
Unless you've got a shitload of cash and the company's got really cool technology, in which case your best move is to wait. . . and buy it at fire sale prices, rebrand the product, and build it right this time.
Tech Public Policy stuff
Some of the first PowerPC Performas and Powermacs (circa 95) had issues with the firmware and ROMs forcing apple to replace boards in *alot* of machines. It was no doubt a tremendous pain in the rear, and proably cost them alot of cash and PR standing.
Even people that believe in pre-destiny look both ways before crossing the street.
I don't know much about the details of any of these (other than how easy it was to kill a Pentium with the f00f bug), but when I do cat /proc/cpuinfo, it mentions all of them. Presumably the Linux kernel has workarounds for them all.
It's really pretty tricky to design an embedded system so that re-programming the Flash on board is easy enough for lusers to do, and yet it will never, ever get changed by accident. PC's are probably the easiest cast, since they come with built-in serial comm, a full-featured OS, and user interface through a full-sized display, keyboard, and mouse -- yet I hear of accidents happening in re-flashing and leaving the MB dead until the chip is physically replaced.
Now think of doing this with a CD drive, which has no display other than a few LED's, no keyboard other than a few function keys, and no communications capability...
The economics of your company are probably quite a bit different from the auto industry; your volume is probably several orders of magnitude less, to name one thing. But you have to consider the loss of goodwill if customers have to pull hardware and ship it to you for firmware fixes, instead of taking 20 minutes to download and install a new patch. You might also consider the benefits of being able to sell firmware functionality upgrades for units already installed; the customers will love you for teaching their old dog a new bunch of tricks.
Scientists restrict study to entire physical universe; creationist