Fatal WeaknessWith High-Capacity MMC/SD Cards?

jvaigl writes "I think I've spotted a pretty fundamental problem with those little Secure Digital and MultiMedia Cards. Since so many people are starting to use them as portable storage and file backup, I thought I'd try run this by the Slashdot readers and either get some corroboration or else have someone knowledgeable point out my misunderstanding." Read on for the trouble -- which has to do with FAT32 formatting and those high-capacity cards -- and the conclusions that jvaigl draws from his experiences.

"I am working on an embedded project where I am using Secure Digital and/or MultiMediaCards to store data. For convenience in developing and updating, I have decided to use a Windows FAT-type file system. This way I can create them, debug them, and update them on Windows development machines using USB card readers.

Since I have to keep around 25,000 files on the card, and since I'd like to minimize disk fragmentation that would result from large cluster size, I would like to use FAT32 with 512-byte clusters. This is no big deal, and certainly supported on windows - "format f: /fs:fat32 /a:512". Done and done.

The interesting thing was that I bought 4 256MB SD cards (Three from SanDisk, one from Lexar Media), and quickly killed 3 of the 4. The SanDisk cards report that track 0 is not readable when I try to format it. Snooping the SD bus shows the card inits OK, and allows writes, but returns an error whenever track 0 is read. The Lexar card's failure is a little more subtle: a format looks like it works, but subsequent chkdsks always fail. The 4th card I'm afraid to repeat this on.

SanDisk (after some weeks of running around) will replace my cards, but hasn't addressed the cause of the failure. I'm also still waiting for a reply from the Lexar 2-day-turnaround support, after 7 business days, including a reminder email.

My theory goes like this: on FAT32, in the first sector (sector 0), there's a field that gives the sector number of the File System Info Sector (FSInfoSec). Every indication I've seen puts this in sector 1, the second physical sector. This sector contains updated counters of used and free clusters on the device. The 256MB cards have about 499,000 512-byte clusters on them. These flash devices have a lifetime of 300,000 write per block, so if I copy 25,000 files to fill the card, the FSInfoSec has been updated either 25,000 times or 499,000 times (depending on when the filesystem updates the counters). If it's the former, I've just eaten up 8% of the lifetime of the card. If the latter, I've killed it before even finishing my write, since a write anywhere also causes a write to sector 1. At best case, once I update this card 12 times, I have to throw it away.

There is some Microsoft documentation that says the FSInfoSec pointer in sector 0 can be set to 0xffff to indicate it's not used. When I used dskprobe.exe from the Microsoft Windows Resource Kit to patch this pointer, Windows 2000 Professional (with a fresh Windows Update applied) blue screens so frequently when I do a dir or chkdsk on the card that I can't do anything useful before I need to cycle power on my PC.

To test my theory, I replaced the dead Lexar card, and repeated the experiment, this time formatting the card FAT16 (no FSInfoSec anymore), and the minimum supported cluster size of 2K. The bad news is of course that I lose about 26MB on the card to fragmentation since the clusters are so large. The good news is that I can write the disk full as many times as I can put up with, and it never fails.

So there are two conclusions: 1) There's a staggeringly high defect rate in the 256 MB cards (SanDisk denies this) and all my ideas about the large cards ever working well with FAT32 are groundless, or 2) even though FAT16 on a 256MB card is hugely wasteful, it's the only way to get the cards to work for very long at all."

Comments

Have you tried...

[KDan]

any other file systems on these (eg NTFS)? Or am I asking a stupid question (better to ask a stupid question and be stupid for a /. thread than to not ask it and be stupid forever!).

Daniel

Re:Have you tried...

[Anonymous Coward]

I think NTFS would run into the same problem because it updates the MFT rather frequently. Maybe UDF might be a good file system for Flash memory cards :D

Re:Have you tried...

[hey!]

He's tryin to read them on an embedded system, so unless that system is NT based NTFS won't do him any good.

Fat32 introduction...

[stienman]

PJRC has a nice introduction to the fat32 file system on their website. It's aimed at people writing code for microcontrollers to access fat32 partitions on IDE drives, so it's got the goods.

-Adam

Forgive my Ignorance...

[nathanh]

... but could you use the loopback device to create an image file and then "dd" the image file to the card? This way the 499,000 writes would be made on the host computer and only the final version written to the card.

Re:Forgive my Ignorance...

[andfarm]

He's using Windows on the host machine. No dd or loopback there ;-)

Good idea for Linux users, though.

Re:Forgive my Ignorance...

[KDan]

That's a neat idea though. And surely those cards' drivers support some way of direct-dumping bytes onto them...

Daniel

Re:Forgive my Ignorance...

[babbage]

This is at least partly inaccurate: if you install Cygwin then you can also install DD on a Windows machine. Moreover, I know that even without Cygwin there are ways to do this with native DOS tools -- I know this because the first time I installed Linux, the RedHat documentation described how to write a disc image to the floppy drive from a DOS prompt. I want to say the DOS equivalent was actually something like "dd.exe", but it's been years now and, running "which dd" from a Cygwin shell just gives the POSIX dd that Cygwin itself installed, masking any system version that may also exist.

As for the loopback trick, that I don't know about. Someone cleverer than me might be able to do this within the Cygwin environment (or some other way?) but I have no idea where to even start...

Re:Forgive my Ignorance...

[gmhowell]

I believe you are talking about rawrite.exe. At least that's what I used to make boot floppies with RH in the past.

Re:Forgive my Ignorance...

[babbage]

Yes! Thank you, that's exactly the one I was trying to think of. Poking around on the closest Windows box (a laptop running Win2kPro), there doesn't seem to be a rawrite.exe file under the %SYSTEMROOT% directory, but maybe I'm just not looking hard enough. Oh well, I've got Cygwin anyway so the point is moot here.

Re:Forgive my Ignorance...

[MrResistor]

Every time I've used it (about once a year) I've had to download it. I don't think it's an MS thing, just a DOS utility.

Re:Forgive my Ignorance...

[Scarblac]

Moreover, I know that even without Cygwin there are ways to do this with native DOS tools -- I know this because the first time I installed Linux, the RedHat documentation described how to write a disc image to the floppy drive from a DOS prompt.

Rawrite.exe is a utility that you find on the RH CD. It's not a standard DOS utility.

Re:Forgive my Ignorance...

[Jump]

Loop mounting is a feature of the linux kernel. With
cygwin you still use the microsoft kernel even if you emulate native linux routines using the cygwin library. To my knowledge, there is nothing like loop mounting in windows, so this is why you can't loop mount there (try the mount command with option '-o loop' on a iso image file if you like).

Loop mounting is one of the cool features of linux, I would miss alot in windows.

Re:Forgive my Ignorance...

[gmhowell]

I only ever found it on my RH discs.

Re:Forgive my Ignorance...

[mgessner]

Yes, that's one thing we discussed.

But it turns out (read my other post) that if you use mkdosfs, and copy via a mount point (e.g., "mount -t vfat /dev/sdc1 /mntflash") this all works just fine.

Same problem with CD-RWs

[andfarm]

I haven't had any experience with this myself, but I've heard about some programs that supposedly allow one to use a CD-RW as a dynamically rewritable drive. (Like a floppy, in other words.) I would expect similar problems with such a configuration. Anybody tried this?

Re:Same problem with CD-RWs

[finity]

I've used some programs that claim to do that, but what they really do is queue the stuff to write up and then burn it when you decide to remove the cd (or when you specifically tell it to burn it). I haven't used windows to burn rw's recently, and at the time, this was the closest you got to using a cdrw like a floppy. Also, I don't know all the details of the filesystems used on cds, but it's iso9660 and not fat. I wouldn't imagine it works the same way, however, because you can burn multiple sessions on a cd-r, and this media wouldn't allow you to rewrite parts previously written.
Now, there's an interesting idea though, why use fat filesystem if you can use a cd's type of filesystem.
Someone correct me please if I'm a babbling idiot.

Re:Same problem with CD-RWs

[stienman]

CD-RW disks are not written to with FAT32 or similar file system formats, so you wouldn't have the same problems. I believe DirectCD and others use a proprietary format, which is why you must have directcd installed on any machine you want to access such a cd from - Windows XP comes with this support built in, including writing.

-Adam

Re:Same problem with CD-RWs

[Col. Klink (retired)]

DirectCD doesn't use a proprietary format, it uses the UDF filesystem and packet writing. The 2.4 kernel has read support for such drives, but not write support on CD-R drives due to lack of kernel write support for CDR/CDRWs. For more info.

You need to have the DirectCD software installed on windows to read it because it's not supported in the kernel before WinXP. When you use DirectCD to format a disk, it creates a minimal ISO9660 filesystem with the DirectCD-reading software.

I wasn't able to use the UDF filesystem on older CD-ROM hardware, but it should work fine on all the CD-R and DVD-ROM drives.

Re:Same problem with CD-RWs

[conway]

There's no problem like that with CD-RWs, because they use a special file system -- UDF.
Its a filesystem specifically designed for CD-RWs and would not suffer from a problem like the poster describes.
However, CD-RWs do have a maximum write lifetime, and will eventually degrade. This will take a long time though using UDF.

Use a filesystem specific for flash

[Lenolium]

Flash disks tend to have filesystems specifically designed for them because they have very different characteristics from traditional drives (ones can change to zeros, but zeros can't change to ones, unless you erase the entire flash sector, and writing a flash sector doesn't matter, it's the erases that count.)
A good flash filesystem will ensure that sectors are only erased when absolutely necessary, and will spread the allocation table out accross multiple sectors FAT16 and FAT32 are horrible about this, and will lead to extremely early flash death. So, if you are going to use flash, please treat it like it is flash, even though it has an IDE interface, it is very different than a standard disk on the other end.

Re:Use a filesystem specific for flash

[adolf]

What are some examples of such a splendid, flash-specific filesystem?

Re:Use a filesystem specific for flash

[Lenolium]

JFFS is one, specifically for embedded linux applications, it seems to do pretty well, and has the great advantage of being journaling ( embedded devices are supposed to handle having the power yanked out from under their feet fairly well ) A good writeup of that is availible here: http://www.embeddedlinuxworks.com/articles/jffs_gu ide.html

Re:Use a filesystem specific for flash

[GoRK]

The other reply mentioned JFFS, but its successor, JFFS2 is designed to be much better. CramFS is also good but it's read-only. Lots of the current embedded linux stuff (especially the stuff on ARM) is using jffs2.

~GoRK

Re:Use a filesystem specific for flash

[Kanasta]

that's great to say, but you think people using pocket PC machines format their SD cards?

does anyone actually know what format pocket PC's do to their cards?

Re:Use a filesystem specific for flash

[jvaigl]

The reason we chose SD/MMC cards in the first place was that it's cheap, ahem 'reliable', widely available, and easy to write to from a user's PC in the field with no hardware or software development effort on our part.

It's supposed to be the commodity part of the project were we just buy something that works.

The whole issue of '...if you are going to use flash, please treat it like it is flash', as you suggest, is -- I think -- a clear view of the problem, but the heart of it is that Windows only supports FAT on these cards, so that's what I'm stuck with if I want these to work on Windows without a lot of extra expense in time.

Re:Use a filesystem specific for flash

[GlassHeart]

Windows only supports FAT on these cards, so that's what I'm stuck with if I want these to work on Windows without a lot of extra expense in time.

FAT is utterly ill-suited for use on a flash. In addition to the problem noted by the article, the FAT itself and the root directory are both at fixed locations, and will wear out more quickly than the rest of the disk. Not surprisingly (for a file system designed to run on floppy disks), FAT or root directory media failures are basically not recoverable.

The common solution to this problem is to add a flash file system layer (sometimes known as a flash translation layer) underneath FAT. This yields the simplicity of FAT and the wear leveling of a real FFS, but you're likely going to have to pay for it in money if you don't have the time.

Remember to fire the architect who chose FAT on SD/MMC without having a clue how they work. This is not the kind of problem that should be found when you're already testing your code. As you noticed, you're basically forced to lose 10% capacity, or insert a new expensive layer into your product.

Finally, in case you haven't noticed, you should also try to minimize the times you do write. Use a suitably-sized data buffer, don't flush it too often, and keep files open if you're going to update it many times.

Mixed Media Cards?

[Xunker]

In the olden days of PDAs, we didn't have Flash RAM. Instead, we used so-called SRAM, which was just charge coupled RAM that has to be constantmy charged, like the RAM in your computer. This was great because you could write to the card assumably infinite number of times -- the downside was that it was super-expensive.

I realize that SRAM woudln't be viable with cards of modern styles because of the requirement of a battery to power all the memory, but what about a combination system? Partial blocks are SRAM with a tiny (read embedded Zinc Air or supercapacitor) battery supply, where the data storageblocks are Flash.

I'm not an EE, so I may be not understanding the way allocation tables are stored on a card, but is something like this viable?

Re:Mixed Media Cards?

[swmccracken]

SRAM, or Static RAM is still around. It's just the individual memory cells are a bistable formed from two transistors and other bits and pieces. It has the advantage that data is kept valid and accessable as long as power is applied.

The other option is the cheaper DRAM - where the memory cells are just a capacitor and a transistor. The problem is that the capactior doesn't hold its charge indenfinitly and needs to be refreshed. - It's this need of a refresh that is a pain because it blocks access by other devices, and requires logic circuts.

In a modern PC, cache is typically SRAM, and main core is DRAM.

As for "The olden days of PDAs" - well, Palms use exactly this for their primary storage.

You're right though, a simple battery backed SRAM card would be useful for a lot of situations - especially if an external power supply was available most of the time. (My palm happly keeps 8 MB of such ram valid for months at a time off two AA batteries.)

Of course, if it's read only or infrequently updated then preparing an image on a file and writing that would be a better solution.

I've seen a flash-based embedded system create a ramdisk to store working files - avoids the rewrite the flash issue.

Re:Mixed Media Cards?

[Xunker]

Well, by "olden days", I was meaning expansion cards, not internal storage :)

I'm gazing whistfully at my Newton 1MB SRAM PCMCIA card across my desk :)

FFS?

[glenstar]

What about with FFS? Is this behaviour only being seen with NTFS and FAT32?

Sounds like you're right

[morcheeba]

FAT32 is not really suitable for Flash memory for precisely the too-many-erase/write cycles you've noticed. The usualy solution for this is some sort of leveling algorithm so that blocks are rearranged in physical memory so that they are erased an equal number of times. This can be done with a software translation layer, in the hardware (doubtful on such a small, dumb device, but possible with an ide interface), or with an alternative file system designed for the purpose (such as JFFS).

If possible, use one of the these techniques. If not, can you consolidate all the files into one file (easy if all files are the same size) and just rewrite portions of that file? That way, the FAT wouldn't need to be updated. Lastly, is there some sort of caching algorithm you can enable that would delay the write to the FAT+directory until (at best) the time the card needs to be removed?

p.s. you're not losing space to fragmentation; it's actually the slack space at the end of each file that's doing it (I think you just used the wrong term describing it).

Re:Sounds like you're right

[Kevster]

I think what he's describing is also called "internal fragmentation" in some circles.

Re:Sounds like you're right

[morcheeba]

thanks; I learned something!

Re:Sounds like you're right

[jvaigl]

Thanks for this idea.

Since we'd like the convenience of the separate files, and since we'd like bozo users in the field to be able to update the cards (without us writing or distributing any software to accomplish the update), the FAT16 option is most appealing to me. These considerations aside, file consolidation is the best idea so far, and we've discussed it in the group.

The problem is that I'm not 100% sure that all the varieties of Windows really update the FSInfoSec once for each file rather than once for each cluster added during the file write. This would be stupid, but not out of the question. I spent some time with a a free USB snooper I found on sourceforge.org (thanks!), and it looks like the update is once per file on Win2K pro, but I didn't want to repeat the experiment on '95, '98, ME, XP, and the other flavors of 2K.

It might work to glob all these files into one massive file, and just seek around in it at runtime in the embedded system. That might take some extra work in the filesystem to be efficient, but if it gets back 26MB of wasted space from the cards it's worth considering.

Re:Sounds like you're right

[GlassHeart]

can you consolidate all the files into one file (easy if all files are the same size) and just rewrite portions of that file? That way, the FAT wouldn't need to be updated.

You'll still be rewriting in place. This is bad, because essentially every write is preceded by a slow erase, which will result in horrible write performance. Also, you'll still be wearing out the same spots, just not the FAT. If you perform wear leveling inside that giant file, then you're really just putting the flash translation layer on top of FAT rather than below it.

Lastly, is there some sort of caching algorithm you can enable that would delay the write to the FAT+directory until (at best) the time the card needs to be removed?

Good caching is essential both for performance and wear. Unfortunately, many of these devices don't have physical locks that prevent their sudden removal.

Most Flash cards have wear-leveling controllers

[RealTime]

This sounds more like a bug in the controllers inside the Flash cards than the actual choice of filesystem. Most Flash card formats (CompactFlash, MemoryStick, MMC/SD) contain a microcontroller that does wear-leveling and ECC. So, logical block zero of the device does not remain physical device zero if that block gets worn out. There are lots of references on the web discussing the microcontrollers in various Flash cards, for example this article (linked via Google cache because the original is a PDF).

These microcontrollers are precisely the reason why it is not a good idea to use these formats in devices that can be powered off suddenly. Look here (search down for "asynchronous power fail" for a mention of these problems. Elsewhere on the site (and in the JFFS author's other online comments), more discussion of this problem is available, including the JFFS author's own experiments.

JFFS works with MTD devices, which are flat Flash arrays with no microcontroller (and the JFFS author doesn't plan on supporting ATA-type Flash cards, although it appears others may be working on this). This gives JFFS complete control over journalling, wear-leveling, and error correction. It is able to do these things in a fashion that is robust in the face of asynchronous power failures. The microcontrollers in various Flash cards do not appear to be this sophisticated.

So, 1) it may not be the choice of filesystem that is the problem, 2) there are documented reasons for not using Flash cards in certain types of systems, and 3) JFFS (and JFFS2), even if they support non-MTD devices now, probably cannot safeguard against the problems in microcontroller-based Flash cards.

Re:Most Flash cards have wear-leveling controllers

[jvaigl]

I suspect they're all the same since most seem to be made by SanDisk, Toshiba, or Panasonic and the others are just 'branded' versions of one of these. With that said...

From the "MultiMediaCard Product Manual" (© 2000 SanDisk Corporation):

1.5.3 Endurance

SanDisk MultiMediaCards have an endurance specification for each sector of 300,000 writes ...With typical applications the endurance limit is not of any practical concern to the vast majority of users.

1.5.4 Wear Leveling

SanDisk MultiMediaCards do not require or perform a Wear Level operation.

Wrong term

[aster_ken]

The term you should use is not "fragmentation" but "slackspace". Fragmentation is when one file is "fragmented" between different non-adjacent sectors on a drive. Slackspace is when a 512 byte file occupies 4 kilobytes of space because that is the size of a "cluster" of sectors - thus, you lose 3.5 KB of space in the process.

Re:Wrong term

[asimulator]

I believe there's internal fragmentation - which is what you refer to as slackspace - as well as external fragmentation which you call just fragmentation.

Hey, genius-

[Lord Bitman]

Use tar.
Problem solved.

TARFS

[Anonymous Coward]

Why not utilize a TAR based pseudo FS? Where a single tar file simulates an entire fs.

Surely there are developer resources for this?

[biglig2]

What do the SD card association say? What do the manufacturers say when you complain that their cards have failed?

I suspect that you are failing the cards since the manufacturers do not expect usage patterns like you have; most purchasers of high capacity SD cards are using them because they want to keep a relatively small number of large files on them (JPG on your camera, MP3 in your player, databases in you PDA), not thousands of tiny cards.

Maybe you are outside the design spec, or maybe (especially since the sandisk and lexar fail differently) the specific cards are bad implementations of the standard that "normal" users won't ever notice.

Re:Surely there are developer resources for this?

[jvaigl]

SanDisk offerred to replace the cards (still waiting for the RMA), but ignored my question. Lexar still hasn't replied.

I agree that my usage pattern is atypical, but since SanDisk is now marketing their Cruzer [sic] 256MB MMC USB plug as a PC file backup device, I'm guessing they'll see a whole bunch of these cards fail before their 'expected' lifetime.

The least they could do if they realized this were a problem is stick a note in their FAQ about filesystem support. The internet is FULL of Pocket PC and Palm users' complaints about getting these cards to last, perform well, and work cross-platform. Seems like it's not too much to ask for them to stick a couple paragraphs on their web site for the moderately technical user.

...Climbs off soapbox... --Jim

Watch out for SanDisk SD cards

[zsazsa]

In the Sharp Zaurus Linux PDA community, many have shied away from SanDisk's SD cards. Their 128MB and 256MB had many problems with the Zaurus. Things are supposedly better now but you still hear about SanDisk SD problems with brand new cards and the latest Sharp ROM.

I've used a 128MB Lexar, formatted as FAT16 and things have worked well. Many have used an SD card as main storage on their Zauruses, formatted as ext2.

Stupid Question...

[pythorlh]

Is there anything limiting this problem to flash disks? What's preventing the multiple re-writing from damaging a normal hard drive's sectors? I know that the general increase in hard driv efailure is normally attributed to pushing the storage envelope, but could this also be a factor?

I don't use FAT at all...

[Anonymous Coward]

I've successfully formatted high capacity SD cards with ext2 and ext3 in Linux. You didn't mention that Windows was a requirement, why use FAT at all? I haven't experienced any file system degradation to speak of. I use mine in a SanDisk Cruzer USB reader that works just fine with the usb-storage kernel module, and I "treat it like an IDE disk" all the time.

Linux: less calories, less FAT.

Re:I don't use FAT at all...

[GlassHeart]

I "treat [a flash device] like an IDE disk" all the time.

Try mounting it as /tmp and /var. (Any damage is your own responsibility, please.)

Re:I don't use FAT at all...

[Anonymous Coward]

But he has his swapfile there too.

Why .zip, .tar, and .gbfs are not file systems

[yerricde]

Why not utilize a TAR based pseudo FS?

I actually did this once. It's called GBFS, and it's designed to hold graphics, text, and audio assets for programs running from ROM on embedded systems such as the Game Boy Advance compact video game system. I would have used GNU tar, but I dropped it when I saw that the header for each file took half a kilobyte and that I could reinvent a better wheel for my purposes.

The problem with doing a general file system in an archive file format such as .tar, .zip, or .gbfs is that you cannot change the size of a file without copying the whole file system to another file. Nevertheless, .zip and .gbfs do work well as read-only file systems.

Yes, I am troll.

[Lord Bitman]

All reasonable suggestions are "troll" if they are easy to impliment.

heh, use a proper file system

[sxpert]

You should use a filesystem that is intended for use with flash devices, such as jffs.

Sandisk media

[zero_offset]

Sandisk appears to have a mediocre rep when it comes to camera flash cards. They were nailed for misrepesenting the capcity of their media. After being busted, they pulled that stunt where they conveniently "redefined" what "they" mean by "megabyte".

I did a quick Google search but couldn't find any articles. It's out there, though; I ran into this when I was selecting media for my first digital camera about two years ago. Also I seem to remember certain on-line retailers showing a warning note with Sandisk media about this, ahem, "unusual" definition of megabyte.

Could be unrelated to the thread problem (sounds like it, from some of the other posts about frequent writes wearing the card out), but it might be worth considering.

Must you use MMC cards?

[Anonymous Coward]

If they don't do wear leveling, they
are going to be problamatic, unless
you do the wear leveling yourself.

I might suggest using Compact Flash,
instead. As far as I know, all compact
flash cards do do wear leveling, and
they use a handy IDE interface, and so
should be easy to support.

On a possibly related note, there was
a huge thread on LKML a few days ago,
dealing with flash cards dying. There
may be some information you can glean
from the archives. The original post
was made on Febuary 2, 2003. At any
rate, good luck!

AC

jvaigl! Why didn't you mention...

[mgessner]

that when we did the formatting with linux, it worked? I thought that was one of the funniest parts of all this. When we used mkdosfs to put a FAT32 fs on the cards, it worked (disregard that FAT32 is probably not appropriate -- we discussed this, and I even recommended we try a linux fs, but since it's an embedded project, jvaigl didn't want to go off and "prove" ext2/3 for this, and I can't blame him).

(I work with jvaigl.)