Slashdot Mirror

← Back to Stories (view on slashdot.org)

Low-Level Format For a USB Flash Drive?

Posted by timothy on from the back-from-the-living-dead dept.

Luyseyal writes "I unwittingly bought one of these terrible flash cards at Fry's and have managed to nuke two of them, successively. I have a USB flash card reader that will read/write the current one at USB 1.0 speed, but it locks up every Ubuntu and XP machine I've come across in high-speed access mode. I have read that if I low-level format it that it could be fixed, though my current one doesn't support it. My Google-fu must be weak because I cannot seem to find a USB flash reader that specifies that it will do low-level formatting." Can anyone offer advice for resurrecting such drives?

27 of 252 comments (clear)

  1. The official utility, perhaps? by Anonymous Coward · · Score: 5, Informative

    http://www.sdcard.org/consumers/formatter_3/

    1. Re:The official utility, perhaps? by imp · · Score: 5, Informative

      This formatter won't do anything to help you out. It will just put a new filesystem on the part. You can't 'reformat' FLASH. Bad is bad, and you lose.

    2. Re:The official utility, perhaps? by dgatwood · · Score: 5, Informative

      And it's probably not the fault of the flash part anyway. I'm assuming by "hang", the original poster meant that accesses to the device stall indefinitely. Since the device stalls only when accessed at USB 2.0 speeds, that almost completely rules out the flash part as the culprit. Although flash drives do support multiple protocols, there's really no good reason for a flash controller to implement support for more than one protocol, as flash parts have to support all of them. Thus, they're going to pick the fastest protocol and use it every time, without regard to what speed the USB side of the bridge is using for communication. Therefore, the only way the USB speed should realistically trigger a failure in the flash part is if the flash part can't handle high throughput.

      It's important to understand that actual cameras write data to flash parts as quickly as the flash part can take it. Thus, a failure caused by high throughput (in the absence of specific workarounds in the camera) would cause the card to be completely and totally nonfunctional in basically any real-world camera hardware. Therefore, the fact that this card is even on the market is a pretty strong indication that the problem is on the other side of the USB bridge---either the USB bridge silicon itself, the USB cable, the host silicon, or drivers.

      Yes, lots of people are having trouble with those cards, but in every report, the failure was an outright controller failure, with all data lost, not problems accessing it at certain speeds. I'd be very surprised if that sort of failure were anything other than a junk flash reader.

      Now I know what you're thinking. USB card readers "just work". No, they don't. I was rather miserable using dd to manually work around bugs in a USB flash reader just a few years ago. In that particular case, throwing large requests at the thing over a high speed connection caused the device to randomly return a copy of block zero instead of the expected data. Did I mention that this particular controller silicon was used for dozens of products by at least half a dozen major manufacturers for a couple of years before the flaw was discovered? Or that the bug was never fixed in firmware or silicon? :-) So yeah, a thoroughly broken flash reader would not be at all surprising.

      First thing I'd do is grab yourself a new flash reader and make sure it doesn't use the same chipset as the one you have. If that doesn't help, *then* you can blame the flash part, and there probably isn't anything you can do about it other than ripping it open and making cufflinks out of it.

      If you have access to the flash reader firmware, you might try using a different access mode. I'm guessing that's not an option, though.

      Worst comes to worst, if it worked okay with a camera, you could try using the camera's flash controller to read/write the flash part. It will probably be dog slow, but at least you can copy the data off the card before you toss it out.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    3. Re:The official utility, perhaps? by Skapare · · Score: 3, Informative

      Yes you can low level format flash drives. There is a layer between the real physical flash memory and the presented hard drive, which does things like level out the wear applied to the flash cells by rotating a pool of blocks (why the drive is always a little smaller than the power of two the flash chips can store). A low level format would clear the flash chips themselves and reset the drive emulation layer to initial state (most likely zero assigned blocks and a full pool of unassigned in the canonical order). But the low level formatter would require low level access to do this, and the computer interface (SD slot pins, USB, etc) may not have a way to do this.

      The formatter on the SD Association site, however, appears to be nothing more than a specialized filesystem formatter. Maybe it formats the filesystem and adds some extra stuff afterwards. There is zero indication that it is a low level formatter.

      That said, the OPs problem may be more of a case of defective flash chips and/or defective drive emulation and/or defective interface to the computer via the SD pins. It may be defective only at high speed or it may be defective at all speeds (if the speed can be forced lower). This may be a case of the manufacturer doing overclocking with some higher percentage of manufactured devices that just can't make it at that increased speed. Imagine your computer manufacturer taking 1 GHz CPUs, overclocking them at 3.333 GHz, verifying that the units boot the test utility on the assembly floor, and shipping them to consumers with the OS raw sectored onto the hard drive by never actually booted to see if it can run on a CPU that gets an undetected one-bit error every second.

      --
      now we need to go OSS in diesel cars
  2. Darik's Boot & Nuke by RhapsodyGuru · · Score: 2, Informative

    Give DBAN a try. This tool never fails me for any kind of disk I throw at it.

    http://www.dban.org/download

    1. Re:Darik's Boot & Nuke by Homburg · · Score: 3, Informative

      From what I can see, DBAN deletes and overwrites all the data on the device. I don't see why that would help the OP any more than just repartitioning and reformatting would.

    2. Re:Darik's Boot & Nuke by Unequivocal · · Score: 3, Informative

      I'm not sure either - but here's a guess: If you write data onto every sector of the drive, perhaps the fubar sectors get noticed by the internal controller at that time and get blocked out from future writes. So by reading/writing out the entire drive, maybe you clean it up a bit.. Until of course more bits go bad which it sounds like would be inevitable in the OP's case.

  3. HP USB Disk Storage Format Tool by Anonymous Coward · · Score: 2, Informative

    Not sure when it locks up the system.

    Try

    HP USB Disk Storage Format Tool

    http://www.pcworld.com/downloads/file/fid,64963-order,4/description.html

    1. Re:HP USB Disk Storage Format Tool by Reece400 · · Score: 2, Informative

      Same effect as Dban, erases all boot records partition maps, etc. It can often correct issues that a simple format can't and could help, that said it definatly isn't a low-level format.

    2. Re:HP USB Disk Storage Format Tool by hairyfeet · · Score: 2, Informative

      If the guy wants a low level flash formatter here you go and it took me all of 5 seconds in Yahoo search. Sheesh, was that really hard enough to require an Ask /. post?

      --
      ACs don't waste your time replying, your posts are never seen by me.
    3. Re:HP USB Disk Storage Format Tool by finarfinjge · · Score: 2, Informative

      What is this w32.SillyFDC.BDJ contained in the ZIP file? Is it there by accident? Did you post in ignorance of a site distributing a worm or are you the author?

    4. Re:HP USB Disk Storage Format Tool by hairyfeet · · Score: 2, Informative

      First I don't know where you're getting a zip file, since there isn't any zip files on that site. If you're gonna troll you might want to go to the site first, then you'd know it is a simple .exe installer. I've also scanned it with no less than 3 A/V (Comodo,ClamWin,AVG) and found exactly squat.

      As for those that want to know what it does, it is similar to the low level format tools that Hitachi and Quantum used to pass out with their drives, only it isn't hard coded to a single HDD manufacturer like theirs were. Considering not a single person here has come up with a better idea, or pretty much any ideas at all, I'd say go for it.

      If you are really so paranoid about the .exe you can always install Comodo Time Machine first, and if you don't like it simply go back in time before you installed and all files/folders and reg entries will be gone like magic. Poof!

      --
      ACs don't waste your time replying, your posts are never seen by me.
  4. Low Level by jythie · · Score: 3, Informative

    Ahm.. has the meaning of this term changed? Last time I heard the term it referred to syncing a drive and it's controller, and thus fell out of usage with the rise of IDE disks.

  5. Re:Do niggers use Flash Drives? by Anonymous Coward · · Score: 0, Informative

    Internet trolls rarely believe their own material. They're like evil comedians. Don't get worked up about it. :)

  6. Not the flash chip by Anonymous Coward · · Score: 2, Informative

    The controller is probably fried, maybe a voltage spike or static electricity occurred. Or it's just cheap Chinese crap. So it can't talk to the USB host controller properly.

    The actual NAND memory is probably fine but unless you want to resolder a controller chip just toss the drive.

    I've used the industrial paper shredder at work to destroy flash drives when they were no longer recognizible by the host yet they still had potentially sensitive personal or corporate data on them.

    Take the PCB out of the housing, snap off the USB connector and feed the board into it, the flash chip gets ground to bits so no adversaries can recover your data. :D

  7. Re:Hmm... by marcansoft · · Score: 5, Informative

    They don't abstract the filesystem, but they abstract the hell out of the lowlevel Flash blocks and just present you with a pretty array of sectors that all work perfectly... until they don't, and then you're screwed.

    Good luck, but I wouldn't hold my breath. These things are essentially disposable, and probably more trouble than it's worth reviving.

    The specifics:

    • Any USB reader is going to present itself as a USB mass storage device. This is the worst possible option, because USB mass storage is horridly generic, and pretty much guaranteed to abstract away any chance of lowlevel operations on particular memory card interfaces. Some readers might have undocumented lowlevel operations, but then you'll almost certainly need a tool specific for your reader or chipset. There's no generic lowlevel format for USB mass storage card readers.
    • SD/MMC cards themselves have a high level controller. As far as I can tell from skimming the standard, all you get to do besides read and write is "erase" sectors in blocks, which may or may not help. There may or may not be proprietary commands to do other stuff. Your best bet for a reader is one embedded in a laptop that is not internally SD - that is, one that's a standard SD Host Controller attached to the PCI/PCIe bus. Or an embedded system like some hackable cellphone or a Wii.
    • SmartMedia and xD are bare Flash chips in a pretty package, with no controller. If you can get a reader that will expose the low level flash block I/O, you might be able to get somewhere.
    • CompactFlash cards are basically IDE/ATA drives with some extra modes. IDE low level format commands may apply. Your best bet here is an actual CF to IDE/ATA converter.
    • I don't know about Memory Stick and other cards, but most are along the same lines as SD
  8. dd of course by v1 · · Score: 5, Informative

    sudo dd if=/dev/zero of=/dev/rdiskxxx bs=1024000

    or whatever variation you need for your distro. The above is for mac os x. yes, rdisk is a character device I know I know, but for some reason os x io's a LOT faster o that than the block device. (double or better) No idea why. Block works too tho, whatever works for you. Just plug in the correct disk number for the xxx. Careful which device you're nuking, dd is both swift and unforgiving.

    I'd also like to get slightly pedantic and point out that this is NOT a low level format. Low level format refers to laying down the address blocks, and also the data headers and trailers. All dd does is write zeros to the meat of the data block, and update its checksum. There's no such thing as a low level format for non magnetic media because flash drive blocks are electrically addressed, not physically.

    FWIW, you can probably tack on "count=20" to make things go much faster. I assume all you need is the partition table completely zapped, and the first 20mb should do it fine. Without this it will wipe the entire device, which for a flash drive may take a little bit. But then again your distro or whatnot may try to find a backup copy of the boot block and partition table etc at the end of the device in which case just wipe the whole thing to avoid it "fixing it" for you.

    --
    I work for the Department of Redundancy Department.
  9. Re:Hmm... by marcansoft · · Score: 1, Informative

    That should be not internally USB. D'oh.

  10. I have used: dd by jemc · · Score: 2, Informative

    First find out the device of the flash disk.

    The following fixed a USB disk which was hosed:

    dd if=/dev/random of=/dev/sda
    or
    dd if=/dev/zero of=/dev/sda

    It's not a format, but fixes corrupt files, which can cause the disk to be unuseable.

  11. Low-Level Format by Lord+Byron+II · · Score: 4, Informative

    There's no such thing as a "low level format" on a flash drive. The term refers to specifying where the tracks are at on a magnetic disk. It was possible, although incredibly stupid, back in the day to perform a low level format on a hard drive and tell it to move the tracks closer together. As a result, you could bump your 10MB disk to 12MB.

    This works only because the physical magnetic disk doesn't "know" anything about tracks and sectors. It always drives me crazy when someone who wants to wipe a drive clean, asks me about a "low level format", when what they want to do is zero out the drive (ie dd if=/dev/zero of=/dev/sda).

    For a flash drive, each memory cell physically has a 1-to-1 correspondence to a bit (or several bits) of information, so there's no low level format.

    1. Re:Low-Level Format by Anonymous Coward · · Score: 1, Informative

      That's still not the same as a low-level format described (correctly). It might erase or change *more* data than simply writing 0 to every block-device-accessible byte, but it doesn't let you change the way that data is stored on the hardware the way a traditional low-level format does.

  12. Re:same question but w/ USB sticks by Anonymous Coward · · Score: 1, Informative

    Check out HDD wipe tool and HDD low level format tool from HDD Guru.
    I've used them several times with great success and the fact that they are free as well, doesn't hurt

    http://hddguru.com/software/

  13. Re:Any brand has lemons but some just suck. by tumnasgt · · Score: 4, Informative

    Having owned more than 5 Dells, and worked on many ex-lease Dell boxes that were given to my school, I can say that Dell just give you the drive that is cheapest on the day, not a specific brand.

  14. Re:Haven't done a LLF in a while, but... by Bing+Tsher+E · · Score: 3, Informative

    That's the routine for later-generation 8-bit cards. The card in the original IBM PC-XT, however, was a Xebec card. There are registers you need to poke values into, again using DEBUG. It kicks off some code in the 'ROM' but it runs completely blind to the PC, i.e. the code runs entirely in the controller on the Xebec card. You only know that it's done when the LED on the hard drive eventually goes out. You can apply a stethoscope or listen closely to hear the drive stepping, to know that it's still doing real work. After you're satisfied that it's done, you use DEBUG to read the results from registers on the card to check the completion value in them.

    I used to have a list of the command sequence to ll format with a Xebec card, but don't anymore. You can 'figure it out' by reading the list of commands, which are documented in the IBM Technical Reference Manual. The docs from IBM from that era are awesome, it lists all the register-based 'command formats' and you can easily figure out what you need to issue to kick off the formatting.

  15. Windows quality file transfer, no easy test tool by beachdog · · Score: 5, Informative

    I am responding to your post on the chance that you are seeing a photo import bug because you use gthumb.

    The 16GB flash card you link to in your Ask Slashdot question looks like the 8GB flash card I use in my digital camera.

    If you are doing digital photography and using Ubuntu or a Linux, take note that the photo import utility in gthumb is broken in Ubuntu 9.10. The gthumb version is 2.10.11 and the specific thing broken is photo import of jpeg images. Photo import fails if there are .avi movie files on the flash card.

    I have had a series of flash card aggravations and here is my version of the preceding AskSlashdot comments:

    1. Digital cameras format flash memory cards with minor variations or they store image data with minor variations. I work around potential glitches by keeping the card in the camera and connecting the camera to the Ubuntu computer.

    2. Use gthumb (note bug above) or the graphical file tool Nautilus. The top level menu item "Places" in Ubuntu starts Nautilus. Copy the files from the camera to the computer.

    3. Speaking about USB flash memory, I feel they have devolved into a Windows quality file transfer device = WQFTD That means, they work using the supplied file system. The success of the same devices using Ext2 and Ext3 file systems is problematic.

    4. Measuring the read and write reliability of these WQFTDs at the bit level is a difficult problem. As I mention in my journal, I have a big name DVD drive that is a WQFTD. I know it fails when reading huge 8 bit data files. But, building a tool to prove when and where it fails is beyond my available time as an evening hacker.

    5. So one answer is "simplify and work around your WQFTD" without challenging it's limits.

  16. Re:Any brand has lemons but some just suck. by ThePhilips · · Score: 3, Informative

    If you put in an order for a large number of identical machines chances are they'll have components from the same batch.

    I'd say it depends.

    Our supplier of storage solution "randomizes" the batches. IT was in the beginning checking the HDDs too, only to find that most deliveries contain drives from several batches.

    Other company I have worked for got Dell servers and also RAID10. Few months later on Thursday one drive died. Dell provided new drive on Friday and recovery was started - only to find that another original drive failed over the weekend too, rendering the storage dead. All drives in the original RAID10 were from the same batch.

    Some companies do get it. Some do not.

    P.S. Having *all* drives from different batches, as was explained to me by data recovery specialists, is also bad if one later would want to try to recover information from the dead drive's platters: different batches might have different controllers with different configurations making them irreplaceable. If you have two drives from the same batch and one of them is dead - recovery would be relatively fast and cheap. Recovering information directly from platters is magnitude(s) more expensive.

    --
    All hope abandon ye who enter here.
  17. Re:Windows quality file transfer, no easy test too by John_Sauter · · Score: 2, Informative

    2. Use gthumb (note bug above) or the graphical file tool Nautilus. The top level menu item "Places" in Ubuntu starts Nautilus. Copy the files from the camera to the computer.

    Rather than gthumb or Nautilus, I use the command-line tool rsync, as follows: rsync -avc (source) (destination), with the source being the flash memory card. When the copy is complete I remove the card, re-insert it, and run the command again. The "c" in "-avc" means to checksum the corresponding files, and copy them again if the checksums don't match. When this triggers a re-copying of a file I know that the card is failing, and discard it.

    I bought a bunch of cheap cards when I first got my camera. After some bad experiences with losing pictures I now test each card that I buy, filling it with data and using the above technique to verify that I can get the same data from it twice running. If I can't, the card is discarded. If I ever buy cards locally, I will return bad ones for replacement instead of discarding them.