Slashdot Mirror
Fatal WeaknessWith High-Capacity MMC/SD Cards?
"I am working on an embedded project where I am using Secure Digital and/or MultiMediaCards to store data. For convenience in developing and updating, I have decided to use a Windows FAT-type file system. This way I can create them, debug them, and update them on Windows development machines using USB card readers.
Since I have to keep around 25,000 files on the card, and since I'd like to minimize disk fragmentation that would result from large cluster size, I would like to use FAT32 with 512-byte clusters. This is no big deal, and certainly supported on windows - "format f: /fs:fat32 /a:512". Done and done.
The interesting thing was that I bought 4 256MB SD cards (Three from SanDisk, one from Lexar Media), and quickly killed 3 of the 4. The SanDisk cards report that track 0 is not readable when I try to format it. Snooping the SD bus shows the card inits OK, and allows writes, but returns an error whenever track 0 is read. The Lexar card's failure is a little more subtle: a format looks like it works, but subsequent chkdsks always fail. The 4th card I'm afraid to repeat this on.
SanDisk (after some weeks of running around) will replace my cards, but hasn't addressed the cause of the failure. I'm also still waiting for a reply from the Lexar 2-day-turnaround support, after 7 business days, including a reminder email.
My theory goes like this: on FAT32, in the first sector (sector 0), there's a field that gives the sector number of the File System Info Sector (FSInfoSec). Every indication I've seen puts this in sector 1, the second physical sector. This sector contains updated counters of used and free clusters on the device. The 256MB cards have about 499,000 512-byte clusters on them. These flash devices have a lifetime of 300,000 write per block, so if I copy 25,000 files to fill the card, the FSInfoSec has been updated either 25,000 times or 499,000 times (depending on when the filesystem updates the counters). If it's the former, I've just eaten up 8% of the lifetime of the card. If the latter, I've killed it before even finishing my write, since a write anywhere also causes a write to sector 1. At best case, once I update this card 12 times, I have to throw it away.
There is some Microsoft documentation that says the FSInfoSec pointer in sector 0 can be set to 0xffff to indicate it's not used. When I used dskprobe.exe from the Microsoft Windows Resource Kit to patch this pointer, Windows 2000 Professional (with a fresh Windows Update applied) blue screens so frequently when I do a dir or chkdsk on the card that I can't do anything useful before I need to cycle power on my PC.
To test my theory, I replaced the dead Lexar card, and repeated the experiment, this time formatting the card FAT16 (no FSInfoSec anymore), and the minimum supported cluster size of 2K. The bad news is of course that I lose about 26MB on the card to fragmentation since the clusters are so large. The good news is that I can write the disk full as many times as I can put up with, and it never fails.
So there are two conclusions: 1) There's a staggeringly high defect rate in the 256 MB cards (SanDisk denies this) and all my ideas about the large cards ever working well with FAT32 are groundless, or 2) even though FAT16 on a 256MB card is hugely wasteful, it's the only way to get the cards to work for very long at all."
PJRC has a nice introduction to the fat32 file system on their website. It's aimed at people writing code for microcontrollers to access fat32 partitions on IDE drives, so it's got the goods.
-Adam
Flash disks tend to have filesystems specifically designed for them because they have very different characteristics from traditional drives (ones can change to zeros, but zeros can't change to ones, unless you erase the entire flash sector, and writing a flash sector doesn't matter, it's the erases that count.)
A good flash filesystem will ensure that sectors are only erased when absolutely necessary, and will spread the allocation table out accross multiple sectors FAT16 and FAT32 are horrible about this, and will lead to extremely early flash death. So, if you are going to use flash, please treat it like it is flash, even though it has an IDE interface, it is very different than a standard disk on the other end.
I believe you are talking about rawrite.exe. At least that's what I used to make boot floppies with RH in the past.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
FAT32 is not really suitable for Flash memory for precisely the too-many-erase/write cycles you've noticed. The usualy solution for this is some sort of leveling algorithm so that blocks are rearranged in physical memory so that they are erased an equal number of times. This can be done with a software translation layer, in the hardware (doubtful on such a small, dumb device, but possible with an ide interface), or with an alternative file system designed for the purpose (such as JFFS).
If possible, use one of the these techniques. If not, can you consolidate all the files into one file (easy if all files are the same size) and just rewrite portions of that file? That way, the FAT wouldn't need to be updated. Lastly, is there some sort of caching algorithm you can enable that would delay the write to the FAT+directory until (at best) the time the card needs to be removed?
p.s. you're not losing space to fragmentation; it's actually the slack space at the end of each file that's doing it (I think you just used the wrong term describing it).
HIV Crosses Species Barrier... into Muppets
This sounds more like a bug in the controllers inside the Flash cards than the actual choice of filesystem. Most Flash card formats (CompactFlash, MemoryStick, MMC/SD) contain a microcontroller that does wear-leveling and ECC. So, logical block zero of the device does not remain physical device zero if that block gets worn out. There are lots of references on the web discussing the microcontrollers in various Flash cards, for example this article (linked via Google cache because the original is a PDF).
These microcontrollers are precisely the reason why it is not a good idea to use these formats in devices that can be powered off suddenly. Look here (search down for "asynchronous power fail" for a mention of these problems. Elsewhere on the site (and in the JFFS author's other online comments), more discussion of this problem is available, including the JFFS author's own experiments.
JFFS works with MTD devices, which are flat Flash arrays with no microcontroller (and the JFFS author doesn't plan on supporting ATA-type Flash cards, although it appears others may be working on this). This gives JFFS complete control over journalling, wear-leveling, and error correction. It is able to do these things in a fashion that is robust in the face of asynchronous power failures. The microcontrollers in various Flash cards do not appear to be this sophisticated.
So, 1) it may not be the choice of filesystem that is the problem, 2) there are documented reasons for not using Flash cards in certain types of systems, and 3) JFFS (and JFFS2), even if they support non-MTD devices now, probably cannot safeguard against the problems in microcontroller-based Flash cards.
Yesterday it worked; today it is not working; Windows is like that...
There's no problem like that with CD-RWs, because they use a special file system -- UDF.
Its a filesystem specifically designed for CD-RWs and would not suffer from a problem like the poster describes.
However, CD-RWs do have a maximum write lifetime, and will eventually degrade. This will take a long time though using UDF.
In the Sharp Zaurus Linux PDA community, many have shied away from SanDisk's SD cards. Their 128MB and 256MB had many problems with the Zaurus. Things are supposedly better now but you still hear about SanDisk SD problems with brand new cards and the latest Sharp ROM.
I've used a 128MB Lexar, formatted as FAT16 and things have worked well. Many have used an SD card as main storage on their Zauruses, formatted as ext2.
Why not utilize a TAR based pseudo FS?
I actually did this once. It's called GBFS, and it's designed to hold graphics, text, and audio assets for programs running from ROM on embedded systems such as the Game Boy Advance compact video game system. I would have used GNU tar, but I dropped it when I saw that the header for each file took half a kilobyte and that I could reinvent a better wheel for my purposes.
The problem with doing a general file system in an archive file format such as .tar, .zip, or .gbfs is that you cannot change the size of a file without copying the whole file system to another file. Nevertheless, .zip and .gbfs do work well as read-only file systems.
Will I retire or break 10K?
If they don't do wear leveling, they
are going to be problamatic, unless
you do the wear leveling yourself.
I might suggest using Compact Flash,
instead. As far as I know, all compact
flash cards do do wear leveling, and
they use a handy IDE interface, and so
should be easy to support.
On a possibly related note, there was
a huge thread on LKML a few days ago,
dealing with flash cards dying. There
may be some information you can glean
from the archives. The original post
was made on Febuary 2, 2003. At any
rate, good luck!
AC
that when we did the formatting with linux, it worked? I thought that was one of the funniest parts of all this. When we used mkdosfs to put a FAT32 fs on the cards, it worked (disregard that FAT32 is probably not appropriate -- we discussed this, and I even recommended we try a linux fs, but since it's an embedded project, jvaigl didn't want to go off and "prove" ext2/3 for this, and I can't blame him).
(I work with jvaigl.)
"Sometimes the truth is stupid." - Lawrence, creator of Prime Intellect