Low-Level Format For a USB Flash Drive?

Luyseyal writes "I unwittingly bought one of these terrible flash cards at Fry's and have managed to nuke two of them, successively. I have a USB flash card reader that will read/write the current one at USB 1.0 speed, but it locks up every Ubuntu and XP machine I've come across in high-speed access mode. I have read that if I low-level format it that it could be fixed, though my current one doesn't support it. My Google-fu must be weak because I cannot seem to find a USB flash reader that specifies that it will do low-level formatting." Can anyone offer advice for resurrecting such drives?

Comments

The official utility, perhaps?

[Anonymous Coward]

http://www.sdcard.org/consumers/formatter_3/

Re:The official utility, perhaps?

[imp]

This formatter won't do anything to help you out. It will just put a new filesystem on the part. You can't 'reformat' FLASH. Bad is bad, and you lose.

Re:The official utility, perhaps?

[dgatwood]

And it's probably not the fault of the flash part anyway. I'm assuming by "hang", the original poster meant that accesses to the device stall indefinitely. Since the device stalls only when accessed at USB 2.0 speeds, that almost completely rules out the flash part as the culprit. Although flash drives do support multiple protocols, there's really no good reason for a flash controller to implement support for more than one protocol, as flash parts have to support all of them. Thus, they're going to pick the fastest protocol and use it every time, without regard to what speed the USB side of the bridge is using for communication. Therefore, the only way the USB speed should realistically trigger a failure in the flash part is if the flash part can't handle high throughput.

It's important to understand that actual cameras write data to flash parts as quickly as the flash part can take it. Thus, a failure caused by high throughput (in the absence of specific workarounds in the camera) would cause the card to be completely and totally nonfunctional in basically any real-world camera hardware. Therefore, the fact that this card is even on the market is a pretty strong indication that the problem is on the other side of the USB bridge---either the USB bridge silicon itself, the USB cable, the host silicon, or drivers.

Yes, lots of people are having trouble with those cards, but in every report, the failure was an outright controller failure, with all data lost, not problems accessing it at certain speeds. I'd be very surprised if that sort of failure were anything other than a junk flash reader.

Now I know what you're thinking. USB card readers "just work". No, they don't. I was rather miserable using dd to manually work around bugs in a USB flash reader just a few years ago. In that particular case, throwing large requests at the thing over a high speed connection caused the device to randomly return a copy of block zero instead of the expected data. Did I mention that this particular controller silicon was used for dozens of products by at least half a dozen major manufacturers for a couple of years before the flaw was discovered? Or that the bug was never fixed in firmware or silicon? :-) So yeah, a thoroughly broken flash reader would not be at all surprising.

First thing I'd do is grab yourself a new flash reader and make sure it doesn't use the same chipset as the one you have. If that doesn't help, *then* you can blame the flash part, and there probably isn't anything you can do about it other than ripping it open and making cufflinks out of it.

If you have access to the flash reader firmware, you might try using a different access mode. I'm guessing that's not an option, though.

Worst comes to worst, if it worked okay with a camera, you could try using the camera's flash controller to read/write the flash part. It will probably be dog slow, but at least you can copy the data off the card before you toss it out.

Re:The official utility, perhaps?

[Skapare]

Yes you can low level format flash drives. There is a layer between the real physical flash memory and the presented hard drive, which does things like level out the wear applied to the flash cells by rotating a pool of blocks (why the drive is always a little smaller than the power of two the flash chips can store). A low level format would clear the flash chips themselves and reset the drive emulation layer to initial state (most likely zero assigned blocks and a full pool of unassigned in the canonical order). But the low level formatter would require low level access to do this, and the computer interface (SD slot pins, USB, etc) may not have a way to do this.

The formatter on the SD Association site, however, appears to be nothing more than a specialized filesystem formatter. Maybe it formats the filesystem and adds some extra stuff afterwards. There is zero indication that it is a low level formatter.

That said, the OPs problem may be more of a case of defective flash chips and/or defective drive emulation and/or defective interface to the computer via the SD pins. It may be defective only at high speed or it may be defective at all speeds (if the speed can be forced lower). This may be a case of the manufacturer doing overclocking with some higher percentage of manufactured devices that just can't make it at that increased speed. Imagine your computer manufacturer taking 1 GHz CPUs, overclocking them at 3.333 GHz, verifying that the units boot the test utility on the assembly floor, and shipping them to consumers with the OS raw sectored onto the hard drive by never actually booted to see if it can run on a CPU that gets an undetected one-bit error every second.

Low Level

[jythie]

Ahm.. has the meaning of this term changed? Last time I heard the term it referred to syncing a drive and it's controller, and thus fell out of usage with the rise of IDE disks.

Re:Hmm...

[marcansoft]

They don't abstract the filesystem, but they abstract the hell out of the lowlevel Flash blocks and just present you with a pretty array of sectors that all work perfectly... until they don't, and then you're screwed.

Good luck, but I wouldn't hold my breath. These things are essentially disposable, and probably more trouble than it's worth reviving.

The specifics:

• Any USB reader is going to present itself as a USB mass storage device. This is the worst possible option, because USB mass storage is horridly generic, and pretty much guaranteed to abstract away any chance of lowlevel operations on particular memory card interfaces. Some readers might have undocumented lowlevel operations, but then you'll almost certainly need a tool specific for your reader or chipset. There's no generic lowlevel format for USB mass storage card readers.
• SD/MMC cards themselves have a high level controller. As far as I can tell from skimming the standard, all you get to do besides read and write is "erase" sectors in blocks, which may or may not help. There may or may not be proprietary commands to do other stuff. Your best bet for a reader is one embedded in a laptop that is not internally SD - that is, one that's a standard SD Host Controller attached to the PCI/PCIe bus. Or an embedded system like some hackable cellphone or a Wii.
• SmartMedia and xD are bare Flash chips in a pretty package, with no controller. If you can get a reader that will expose the low level flash block I/O, you might be able to get somewhere.
• CompactFlash cards are basically IDE/ATA drives with some extra modes. IDE low level format commands may apply. Your best bet here is an actual CF to IDE/ATA converter.
• I don't know about Memory Stick and other cards, but most are along the same lines as SD

dd of course

[v1]

sudo dd if=/dev/zero of=/dev/rdiskxxx bs=1024000

or whatever variation you need for your distro. The above is for mac os x. yes, rdisk is a character device I know I know, but for some reason os x io's a LOT faster o that than the block device. (double or better) No idea why. Block works too tho, whatever works for you. Just plug in the correct disk number for the xxx. Careful which device you're nuking, dd is both swift and unforgiving.

I'd also like to get slightly pedantic and point out that this is NOT a low level format. Low level format refers to laying down the address blocks, and also the data headers and trailers. All dd does is write zeros to the meat of the data block, and update its checksum. There's no such thing as a low level format for non magnetic media because flash drive blocks are electrically addressed, not physically.

FWIW, you can probably tack on "count=20" to make things go much faster. I assume all you need is the partition table completely zapped, and the first 20mb should do it fine. Without this it will wipe the entire device, which for a flash drive may take a little bit. But then again your distro or whatnot may try to find a backup copy of the boot block and partition table etc at the end of the device in which case just wipe the whole thing to avoid it "fixing it" for you.

Re:Darik's Boot & Nuke

[Homburg]

From what I can see, DBAN deletes and overwrites all the data on the device. I don't see why that would help the OP any more than just repartitioning and reformatting would.

Low-Level Format

[Lord Byron II]

There's no such thing as a "low level format" on a flash drive. The term refers to specifying where the tracks are at on a magnetic disk. It was possible, although incredibly stupid, back in the day to perform a low level format on a hard drive and tell it to move the tracks closer together. As a result, you could bump your 10MB disk to 12MB.

This works only because the physical magnetic disk doesn't "know" anything about tracks and sectors. It always drives me crazy when someone who wants to wipe a drive clean, asks me about a "low level format", when what they want to do is zero out the drive (ie dd if=/dev/zero of=/dev/sda).

For a flash drive, each memory cell physically has a 1-to-1 correspondence to a bit (or several bits) of information, so there's no low level format.

Re:Any brand has lemons but some just suck.

[tumnasgt]

Having owned more than 5 Dells, and worked on many ex-lease Dell boxes that were given to my school, I can say that Dell just give you the drive that is cheapest on the day, not a specific brand.

Re:Haven't done a LLF in a while, but...

[Bing Tsher E]

That's the routine for later-generation 8-bit cards. The card in the original IBM PC-XT, however, was a Xebec card. There are registers you need to poke values into, again using DEBUG. It kicks off some code in the 'ROM' but it runs completely blind to the PC, i.e. the code runs entirely in the controller on the Xebec card. You only know that it's done when the LED on the hard drive eventually goes out. You can apply a stethoscope or listen closely to hear the drive stepping, to know that it's still doing real work. After you're satisfied that it's done, you use DEBUG to read the results from registers on the card to check the completion value in them.

I used to have a list of the command sequence to ll format with a Xebec card, but don't anymore. You can 'figure it out' by reading the list of commands, which are documented in the IBM Technical Reference Manual. The docs from IBM from that era are awesome, it lists all the register-based 'command formats' and you can easily figure out what you need to issue to kick off the formatting.

Windows quality file transfer, no easy test tool

[beachdog]

I am responding to your post on the chance that you are seeing a photo import bug because you use gthumb.

The 16GB flash card you link to in your Ask Slashdot question looks like the 8GB flash card I use in my digital camera.

If you are doing digital photography and using Ubuntu or a Linux, take note that the photo import utility in gthumb is broken in Ubuntu 9.10. The gthumb version is 2.10.11 and the specific thing broken is photo import of jpeg images. Photo import fails if there are .avi movie files on the flash card.

I have had a series of flash card aggravations and here is my version of the preceding AskSlashdot comments:

1. Digital cameras format flash memory cards with minor variations or they store image data with minor variations. I work around potential glitches by keeping the card in the camera and connecting the camera to the Ubuntu computer.

2. Use gthumb (note bug above) or the graphical file tool Nautilus. The top level menu item "Places" in Ubuntu starts Nautilus. Copy the files from the camera to the computer.

3. Speaking about USB flash memory, I feel they have devolved into a Windows quality file transfer device = WQFTD That means, they work using the supplied file system. The success of the same devices using Ext2 and Ext3 file systems is problematic.

4. Measuring the read and write reliability of these WQFTDs at the bit level is a difficult problem. As I mention in my journal, I have a big name DVD drive that is a WQFTD. I know it fails when reading huge 8 bit data files. But, building a tool to prove when and where it fails is beyond my available time as an evening hacker.

5. So one answer is "simplify and work around your WQFTD" without challenging it's limits.

Re:Darik's Boot & Nuke

[Unequivocal]

I'm not sure either - but here's a guess: If you write data onto every sector of the drive, perhaps the fubar sectors get noticed by the internal controller at that time and get blocked out from future writes. So by reading/writing out the entire drive, maybe you clean it up a bit.. Until of course more bits go bad which it sounds like would be inevitable in the OP's case.

Re:Any brand has lemons but some just suck.

[ThePhilips]

If you put in an order for a large number of identical machines chances are they'll have components from the same batch.

I'd say it depends.

Our supplier of storage solution "randomizes" the batches. IT was in the beginning checking the HDDs too, only to find that most deliveries contain drives from several batches.

Other company I have worked for got Dell servers and also RAID10. Few months later on Thursday one drive died. Dell provided new drive on Friday and recovery was started - only to find that another original drive failed over the weekend too, rendering the storage dead. All drives in the original RAID10 were from the same batch.

Some companies do get it. Some do not.

P.S. Having *all* drives from different batches, as was explained to me by data recovery specialists, is also bad if one later would want to try to recover information from the dead drive's platters: different batches might have different controllers with different configurations making them irreplaceable. If you have two drives from the same batch and one of them is dead - recovery would be relatively fast and cheap. Recovering information directly from platters is magnitude(s) more expensive.