IBM Speeds Storage With Flash: 10B Files In 43 Min

CWmike writes "With an eye toward helping tomorrow's data-deluged organizations, IBM researchers have created a super-fast storage system capable of scanning in 10 billion files in 43 minutes. This system handily bested their previous system, demonstrated at Supercomputing 2007, which scanned 1 billion files in three hours. Key to the increased performance was the use of speedy flash memory to store the metadata that the storage system uses to locate requested information. Traditionally, metadata repositories reside on disk, access to which slows operations. (See IBM's whitepaper.)"

Comments

Re:File Sizes?

[GuldKalle]

As far as I can see, the files themselves were not read, only the metadata (who has access, modification time, position on the spinning platter, etc.).

43 min for 10 bytes?

[Tei]

Thats very slow.

Also, please, better technical expertise writing the articles.

Re:43 min for 10 bytes?

[impaledsunset]

Come on! Adobe Flash has always been slow, that's a massive improvement!

Re:43 min for 10 bytes?

[MichaelSmith]

IBM are selling ClearCase with a straight face.

Re:43 min for 10 bytes?

[maxwell demon]

Make 10 Billion files on your ext3 filesystem and see how long an ls takes you

Ext3 can store 10 billion files in 10 bytes? Must be the new Whoosh feature, which avoids reading metadata like the comment title.

Re:43 min for 10 bytes?

[physicsphairy]

43 min for 10 bytes.

I see they've copied the poorly hobbled together config for my SAMBA server.

Re:43 min for 10 bytes?

[kno3]

Oh, BURN!

Re:43 min for 10 bytes?

[maxwell demon]

Oh, BURN!

Well, for burning I'd prefer ISO9660 with RockRidge extension to ext3. :-)

Huh....

[Demena]

Traditional filesystems hold their metadata on disc? Ermmm... Exactly what do you think that the 'sync' command does. Traditionally metadata is held in memory and periodically written to disc for storage.

Re:Huh....

[SuricouRaven]

Not all of it. Just that which has been recently accessed. Enough for most purposes, as usually only a tiny bit of the stored data is ever needed at once. Doesn't hold up well in some scientific and engineering uses though, and if you need fast response times even on files that haven't been accessed in weeks then it becomes a potential problem.

Re:Huh....

[Demena]

That really depends on the directory layout and directory sizes. Study the i-node structure to understand why.

Re:Huh....

[Demena]

There is s difference between filesystem metadata and file metadata. You mention scientific and engineering uses as being particularly bad when it is my belief that the system architects are the cause of this. It is common to find bad architects in those fields. Directory structure is important. If you do not understand the particular filesystem architecture you cannot design for good and fast access. If you want a good fast access system it is absolutely necessary to understand things at that level. Most delays are not in accessing the file itself or the data within it but actually finding the file (or the bit you want) in the first place.

Re:Huh....

[smallfries]

Are you confusing a system that stores something in memory, and a system that caches a copy of a small part in memory for fast access?

Re:Huh....

[Demena]

I'm not confusing anything. I know exactly how it works.

Re:Huh....

[smallfries]

It doesn't sound like you do. Sync is used to flush the cache of metadata back out to the disk. The metadata is actually stored on disk.

Re:Huh....

[Demena]

Which is precisely what I said. The filesystem metadata that is _used_ is in memory. It is periodically _saved_ to disk iff there have been changes (i-node 0 for standard unix filesystems).

Re:Huh....

[smallfries]

So now you are shifting in your claims. Yes, when metadata is used it is in memory - the same is true of any data. But it is held (to use your term) on disk, where it is loaded into memory on use, changed and saved back to disk. The primary store of metadata, the one that persists between boots, is held on the disk. A small local cache is changed, as with any data. So going back to your original (erroneous) claim: traditional file-systems *do* hold their metadata on disk, even if they cache a portion of it in memory as an optimisation.

Re:Huh....

[Demena]

No. There is no need to retract anything. I made no erroneous claim. Stop trolling.

Re:Huh....

[anamin]

You be trollin.

Re:Define Scanning in...

[MichaelSmith]

I wonder how google would go indexing the contents of 10 billion files.

Demand

[wesleyjconnor]

......Is this kind of performance in scanning in high demand?

cost/performance

[maxwell demon]

They noted that while solid-state storage can cost 10 times as much as traditional disks, they can offer 100 percent performance boost.

So you get 2 times the performance for 10 times the price? I'd say that's still 5 times as expensive. What would be the performance boost with a RAID of 5 disks?

Re:cost/performance

[FishTankX]

I think you misunderstood the point of the statement in that article.

It's referencing using solid state as a cache, and how even though solid state memory costs 10x as much, when used for caching duty, it can increase the performance of the disk array by 100%. This would be in line with the numbers alot of sites are getting from intel's new hard disk SSD caching tech.

Re:Define Scanning in...

[Sulphur]

Maybe they are scanning them to see if they contain a 1 or a 0. That way they can claim insane numbers like 10B. Whatever a 10B is.

2?

You can DIY it in linux.

[elsJake]

Some filesystems allow you to store the journal on a different disk , such as a SSD

Numbers

[kramulous]

Now, some of my maths might be (a little) off, but ...

I've just spent half the day processing financial files ... 133KB average file size and processed (by process, I mean every byte is 'looked' at in c++ code) 4000 per second. I did this on a single file (compressed tar.gz) that when expanded is 7857 files and just over 1GB in size. The compressed file is temporarily stored in /dev/shm. The parallelisation is around one thread processing the ram drive file while the other file copies the next file (1GB file uncompressed, 65MB compressed) from a 5400rpm notebook drive (Thinkpad X60) to the ram drive.

Now, this latest in file processing by a giant of the industry has 'achieved' 3.55 million per second files 'processed' (and by processed it is never said what - but I'll assume the same as me) of files that are 650bytes in size (PDF says dataset was 6.5 TB).

I was processing on a notebook that is about 7 years old architecturally and achieved 544MB bytes processed per second and the latest IBM can do is 2.3GB per second.

Is this a *big* step forward? I should log into our cluster and do a test on memory a little more advanced and see how their numbers stack up.

I guess what i'm saying is, there is just no substitute for writing software properly.

Re:Numbers

[smallfries]

Your lack of understanding is quite simply astounding. You have completely missed the point of their research, which is to reduce the latency in randomly accessing information in a large dataset. They are not measuring throughput (or bandwidth) although the article does state that they hit 4.9GB/s. If you made your files much much smaller and then repeated your test you would find that your performance drops drastically as your program comes limited by a different IO bound. Instead of being bounded by the bandwidth you would find that the number of separate IO operations became the bottleneck. The reason that SSDs have provided such a large increase in performance is not the increase in bandwidth of 5-10x spinning disks - the decrease in latency has increased the number of IO operations by four, or five, orders of magnitude.

Using a RAM disk you should be able to hit a much higher number of IO operations than even an SSD, but it won't come close to the 3.9M+/s that IBM have reported. For database transactions where the amount of information is each record is small, but the number of records accessed is high, this measure is a much better indicator of performance than bandwidth alone.

I guess what i'm saying is, there is just no substitute for writing software properly.

Well, if you have a complete lack of understanding of a subject, but you continue on regardless talking absolute bollocks then perhaps the world would seem like that to you.

Re:Numbers

[Salamander]

Doing something for 7857 files and doing it for 10 billion are very different situations. 7857 files, including metadata, can easily be sucked into memory in one big chunk and unpacked/examined from there. That simply doesn't work for datasets larger than memory. At the higher scale, modern filesystems do tend to fall apart, badly, so different approaches are needed. Comparing your paper airplane to an F-22 doesn't make it look like you know anything about writing software properly. Quite the opposite.

Re:File Sizes?

[TheRaven64]

It says in the heading. They copied 10 byte files in 43 minutes. Not very impressive, even the old Mac troll copied files faster than that...

Try it for yourself

[BlackPignouf]

time sudo ls -lAR / | grep -E '^[ld\-]+' | wc -l

It should give you the number of files on your filesystem and the time it took to "scan" them all.

Re:Try it for yourself

[pakar]

Well, you probably need to make sure you dont have any of the files or metadata in the buffercache before starting.. Also limit the search to the actual filesystem you want to test..

# echo 3 >/proc/sys/vm/drop_caches
# time find / -xdev -printf "%p %y %s %n %i %m %G %U %c %b %a\\n" |wc -l
621847

real 0m36.738s
user 0m6.031s
sys 0m12.737s

This on a simple 40Gb Intel SSD with a ext4 fs

Re:Try it for yourself

[reset_button]

FYI: "drop_caches" only drops clean pages, so you need to run "sync" first if you want to properly flush your cache.

Re:10B

[isorox]

Did anyone else read that as "10 byte files?" that seemed mighty slow lol

Nope, I read 267

Alternative summary

[tulcod]

IBM throws a lot of hardware at a problem; problem gets solved.

SUN were doing this in 1990

[petes_PoV]

I have a vague memory of Sun producing an NFS accelerator about 20 years ago. This worked by caching remote file data in non-volatile memory.

something strange in the title?

[Anonymous Coward]

I was wondering what does it mean 10B files... Ok, the article talk of 10 Billion files... But 1 Billion is 10^9 or is 10^12. So If you have to use a symbol, use a sensible one... What about 10G files? :D

Comparison of several big UGG Deckers

[linyanyun]

UGG Boots Australia is the United States under the company, in terms of degree of market recognition, to seize the market early, large leafy tree. However, in China, Ugg Australia is a trading company agent in product promotion maintenance, to ensure credibility, the existence of defects, which led to the present, very few pure through the headquarters of the United States authorized UGG authentic Cheap Ugg. Jumbo ugg from Australia Australia is a relatively big, and reflects the Australian tradition of craftsmanship and style, fashionable, unique, fine workmanship. But it is worth Unfortunately Jumbo ugg Australia brand is currently only in home sales has not been involved in China. Xiaobian that: If no relatives, friends in Australia, then forget it, travel expenses, postage, flowers you have blood Aukoala Australia ugg sale was founded in the 1970s, when only an initial small workshop, has now been developed to favored by European and American fashion star to become the essential thing. Product imaginative, low-key luxury and fashion, not only continuation of the classic design, but also into the tassel, rivets, Bohemia, feathers and other fashion elements, not only in the order of a high-end products brand, is on the taste laid in the fashion industry's extraordinary. ugg cardyXiao Bian broke the news: Aokoala enter the Chinese market in 2010, by a regular brands company, the overall evaluation was very good, recommended to has a try. It's written by xSteven on 7.26 tag: UGG Boots Cheap Ugg ugg ugg sale ugg cardy

Coach high performance in China in 2010

[linyanyun]

Coach bags in the Chinese market is booming, the report shows same-store sales showed double-digit growth. The U.S. market grew only 6.3%. According to Bain & Company survey, China's luxury market in 2009 the total capacity of 23.3 billion U.S. dollars, Coach said he accounted for 5% of the total market. Lew Frankfort said: "Coach handbags brand awareness in China is not very high, about 8%, we decided to catch up over the next five years, China will surpass Japan to become the Coach after the United States the second largest Coach outlet market. Recently, high-end accessories manufacturer and retailer discount Coach bags said it expects fiscal year 2011 sales in China rose 75 percent year on year. goal is to develop the Chinese middle class consumer level. Coach's CEO Lew Frankfort said in an interview: "As of 2010 In July, China has 25 new store openings in China, the number of stores totaled 65. "authentic bags will focus future development strategy in the Asian market, before the Coach has been very high-profile entry into the cheap Coach European, U.S, UK, Ireland and Portugal markets. Lew Frankfort also said:" As the middle class was growing rapidly, so consumption will rise. China will become the first global luxury market, but now Coach bags is just out of infancy. ". Coach also plans to launch in China in mid-2011, e-brand shopping site. It's written by xSteven on 7.26 tag: Coach bags Coach handbags Coach outlet discount Coach bags authentic bags cheap Coach Coach bags