Fatal WeaknessWith High-Capacity MMC/SD Cards?

jvaigl writes "I think I've spotted a pretty fundamental problem with those little Secure Digital and MultiMedia Cards. Since so many people are starting to use them as portable storage and file backup, I thought I'd try run this by the Slashdot readers and either get some corroboration or else have someone knowledgeable point out my misunderstanding." Read on for the trouble -- which has to do with FAT32 formatting and those high-capacity cards -- and the conclusions that jvaigl draws from his experiences.

"I am working on an embedded project where I am using Secure Digital and/or MultiMediaCards to store data. For convenience in developing and updating, I have decided to use a Windows FAT-type file system. This way I can create them, debug them, and update them on Windows development machines using USB card readers.

Since I have to keep around 25,000 files on the card, and since I'd like to minimize disk fragmentation that would result from large cluster size, I would like to use FAT32 with 512-byte clusters. This is no big deal, and certainly supported on windows - "format f: /fs:fat32 /a:512". Done and done.

The interesting thing was that I bought 4 256MB SD cards (Three from SanDisk, one from Lexar Media), and quickly killed 3 of the 4. The SanDisk cards report that track 0 is not readable when I try to format it. Snooping the SD bus shows the card inits OK, and allows writes, but returns an error whenever track 0 is read. The Lexar card's failure is a little more subtle: a format looks like it works, but subsequent chkdsks always fail. The 4th card I'm afraid to repeat this on.

SanDisk (after some weeks of running around) will replace my cards, but hasn't addressed the cause of the failure. I'm also still waiting for a reply from the Lexar 2-day-turnaround support, after 7 business days, including a reminder email.

My theory goes like this: on FAT32, in the first sector (sector 0), there's a field that gives the sector number of the File System Info Sector (FSInfoSec). Every indication I've seen puts this in sector 1, the second physical sector. This sector contains updated counters of used and free clusters on the device. The 256MB cards have about 499,000 512-byte clusters on them. These flash devices have a lifetime of 300,000 write per block, so if I copy 25,000 files to fill the card, the FSInfoSec has been updated either 25,000 times or 499,000 times (depending on when the filesystem updates the counters). If it's the former, I've just eaten up 8% of the lifetime of the card. If the latter, I've killed it before even finishing my write, since a write anywhere also causes a write to sector 1. At best case, once I update this card 12 times, I have to throw it away.

There is some Microsoft documentation that says the FSInfoSec pointer in sector 0 can be set to 0xffff to indicate it's not used. When I used dskprobe.exe from the Microsoft Windows Resource Kit to patch this pointer, Windows 2000 Professional (with a fresh Windows Update applied) blue screens so frequently when I do a dir or chkdsk on the card that I can't do anything useful before I need to cycle power on my PC.

To test my theory, I replaced the dead Lexar card, and repeated the experiment, this time formatting the card FAT16 (no FSInfoSec anymore), and the minimum supported cluster size of 2K. The bad news is of course that I lose about 26MB on the card to fragmentation since the clusters are so large. The good news is that I can write the disk full as many times as I can put up with, and it never fails.

So there are two conclusions: 1) There's a staggeringly high defect rate in the 256 MB cards (SanDisk denies this) and all my ideas about the large cards ever working well with FAT32 are groundless, or 2) even though FAT16 on a 256MB card is hugely wasteful, it's the only way to get the cards to work for very long at all."

Comments

Have you tried...

[KDan]

any other file systems on these (eg NTFS)? Or am I asking a stupid question (better to ask a stupid question and be stupid for a /. thread than to not ask it and be stupid forever!).

Daniel

Fat32 introduction...

[stienman]

PJRC has a nice introduction to the fat32 file system on their website. It's aimed at people writing code for microcontrollers to access fat32 partitions on IDE drives, so it's got the goods.

-Adam

Forgive my Ignorance...

[nathanh]

... but could you use the loopback device to create an image file and then "dd" the image file to the card? This way the 499,000 writes would be made on the host computer and only the final version written to the card.

Re:Forgive my Ignorance...

[andfarm]

He's using Windows on the host machine. No dd or loopback there ;-)

Good idea for Linux users, though.

Re:Forgive my Ignorance...

[babbage]

This is at least partly inaccurate: if you install Cygwin then you can also install DD on a Windows machine. Moreover, I know that even without Cygwin there are ways to do this with native DOS tools -- I know this because the first time I installed Linux, the RedHat documentation described how to write a disc image to the floppy drive from a DOS prompt. I want to say the DOS equivalent was actually something like "dd.exe", but it's been years now and, running "which dd" from a Cygwin shell just gives the POSIX dd that Cygwin itself installed, masking any system version that may also exist.

As for the loopback trick, that I don't know about. Someone cleverer than me might be able to do this within the Cygwin environment (or some other way?) but I have no idea where to even start...

Re:Forgive my Ignorance...

[gmhowell]

I believe you are talking about rawrite.exe. At least that's what I used to make boot floppies with RH in the past.

Same problem with CD-RWs

[andfarm]

I haven't had any experience with this myself, but I've heard about some programs that supposedly allow one to use a CD-RW as a dynamically rewritable drive. (Like a floppy, in other words.) I would expect similar problems with such a configuration. Anybody tried this?

Re:Same problem with CD-RWs

[conway]

There's no problem like that with CD-RWs, because they use a special file system -- UDF.
Its a filesystem specifically designed for CD-RWs and would not suffer from a problem like the poster describes.
However, CD-RWs do have a maximum write lifetime, and will eventually degrade. This will take a long time though using UDF.

Use a filesystem specific for flash

[Lenolium]

Flash disks tend to have filesystems specifically designed for them because they have very different characteristics from traditional drives (ones can change to zeros, but zeros can't change to ones, unless you erase the entire flash sector, and writing a flash sector doesn't matter, it's the erases that count.)
A good flash filesystem will ensure that sectors are only erased when absolutely necessary, and will spread the allocation table out accross multiple sectors FAT16 and FAT32 are horrible about this, and will lead to extremely early flash death. So, if you are going to use flash, please treat it like it is flash, even though it has an IDE interface, it is very different than a standard disk on the other end.

Re:Use a filesystem specific for flash

[GoRK]

The other reply mentioned JFFS, but its successor, JFFS2 is designed to be much better. CramFS is also good but it's read-only. Lots of the current embedded linux stuff (especially the stuff on ARM) is using jffs2.

~GoRK

Sounds like you're right

[morcheeba]

FAT32 is not really suitable for Flash memory for precisely the too-many-erase/write cycles you've noticed. The usualy solution for this is some sort of leveling algorithm so that blocks are rearranged in physical memory so that they are erased an equal number of times. This can be done with a software translation layer, in the hardware (doubtful on such a small, dumb device, but possible with an ide interface), or with an alternative file system designed for the purpose (such as JFFS).

If possible, use one of the these techniques. If not, can you consolidate all the files into one file (easy if all files are the same size) and just rewrite portions of that file? That way, the FAT wouldn't need to be updated. Lastly, is there some sort of caching algorithm you can enable that would delay the write to the FAT+directory until (at best) the time the card needs to be removed?

p.s. you're not losing space to fragmentation; it's actually the slack space at the end of each file that's doing it (I think you just used the wrong term describing it).

Most Flash cards have wear-leveling controllers

[RealTime]

This sounds more like a bug in the controllers inside the Flash cards than the actual choice of filesystem. Most Flash card formats (CompactFlash, MemoryStick, MMC/SD) contain a microcontroller that does wear-leveling and ECC. So, logical block zero of the device does not remain physical device zero if that block gets worn out. There are lots of references on the web discussing the microcontrollers in various Flash cards, for example this article (linked via Google cache because the original is a PDF).

These microcontrollers are precisely the reason why it is not a good idea to use these formats in devices that can be powered off suddenly. Look here (search down for "asynchronous power fail" for a mention of these problems. Elsewhere on the site (and in the JFFS author's other online comments), more discussion of this problem is available, including the JFFS author's own experiments.

JFFS works with MTD devices, which are flat Flash arrays with no microcontroller (and the JFFS author doesn't plan on supporting ATA-type Flash cards, although it appears others may be working on this). This gives JFFS complete control over journalling, wear-leveling, and error correction. It is able to do these things in a fashion that is robust in the face of asynchronous power failures. The microcontrollers in various Flash cards do not appear to be this sophisticated.

So, 1) it may not be the choice of filesystem that is the problem, 2) there are documented reasons for not using Flash cards in certain types of systems, and 3) JFFS (and JFFS2), even if they support non-MTD devices now, probably cannot safeguard against the problems in microcontroller-based Flash cards.

Watch out for SanDisk SD cards

[zsazsa]

In the Sharp Zaurus Linux PDA community, many have shied away from SanDisk's SD cards. Their 128MB and 256MB had many problems with the Zaurus. Things are supposedly better now but you still hear about SanDisk SD problems with brand new cards and the latest Sharp ROM.

I've used a 128MB Lexar, formatted as FAT16 and things have worked well. Many have used an SD card as main storage on their Zauruses, formatted as ext2.

Why .zip, .tar, and .gbfs are not file systems

[yerricde]

Why not utilize a TAR based pseudo FS?

I actually did this once. It's called GBFS, and it's designed to hold graphics, text, and audio assets for programs running from ROM on embedded systems such as the Game Boy Advance compact video game system. I would have used GNU tar, but I dropped it when I saw that the header for each file took half a kilobyte and that I could reinvent a better wheel for my purposes.

The problem with doing a general file system in an archive file format such as .tar, .zip, or .gbfs is that you cannot change the size of a file without copying the whole file system to another file. Nevertheless, .zip and .gbfs do work well as read-only file systems.

jvaigl! Why didn't you mention...

[mgessner]

that when we did the formatting with linux, it worked? I thought that was one of the funniest parts of all this. When we used mkdosfs to put a FAT32 fs on the cards, it worked (disregard that FAT32 is probably not appropriate -- we discussed this, and I even recommended we try a linux fs, but since it's an embedded project, jvaigl didn't want to go off and "prove" ext2/3 for this, and I can't blame him).

(I work with jvaigl.)