Slashdot Mirror

← Back to Stories (view on slashdot.org)

IBM Speeds Storage With Flash: 10B Files In 43 Min

Posted by timothy on from the you-sure-have-a-lot-of-mp3s dept.

CWmike writes "With an eye toward helping tomorrow's data-deluged organizations, IBM researchers have created a super-fast storage system capable of scanning in 10 billion files in 43 minutes. This system handily bested their previous system, demonstrated at Supercomputing 2007, which scanned 1 billion files in three hours. Key to the increased performance was the use of speedy flash memory to store the metadata that the storage system uses to locate requested information. Traditionally, metadata repositories reside on disk, access to which slows operations. (See IBM's whitepaper.)"

6 of 76 comments (clear)

  1. Re:File Sizes? by GuldKalle · · Score: 4, Informative

    As far as I can see, the files themselves were not read, only the metadata (who has access, modification time, position on the spinning platter, etc.).

    --
    What?
  2. 43 min for 10 bytes? by Tei · · Score: 5, Insightful

    Thats very slow.

    Also, please, better technical expertise writing the articles.

    --

    -Woof woof woof!

    1. Re:43 min for 10 bytes? by impaledsunset · · Score: 4, Funny

      Come on! Adobe Flash has always been slow, that's a massive improvement!

    2. Re:43 min for 10 bytes? by maxwell+demon · · Score: 2

      Make 10 Billion files on your ext3 filesystem and see how long an ls takes you

      Ext3 can store 10 billion files in 10 bytes? Must be the new Whoosh feature, which avoids reading metadata like the comment title.

      --
      The Tao of math: The numbers you can count are not the real numbers.
  3. Re:Define Scanning in... by MichaelSmith · · Score: 2

    I wonder how google would go indexing the contents of 10 billion files.

    --
    http://michaelsmith.id.au
  4. Alternative summary by tulcod · · Score: 2

    IBM throws a lot of hardware at a problem; problem gets solved.