Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Design Of The Google File System

Posted by timothy on from the library-of-babel dept.

Freddles writes "This is an interesting paper (PDF) describing the design approach to Google's file system. The design had to take account of requirements for huge file sizes, a highly responsive infrastructure and an assumption that hardware components will always fail."

4 of 210 comments (clear)

  1. In case you don't like PDF by Brahmastra · · Score: 4, Informative

    Here's the html link

  2. PDF mirror by Tyler+Eaves · · Score: 4, Informative

    PDF mirror on my server /Feels sorry for the Rochester cs server

    --
    TODO: Something witty here...
  3. Just to make it clear.. by Doodhwala · · Score: 4, Informative


    Okay, so I read this paper as a part of the SOSP reading group here at school. Just want to make it clear that this is not the file system used by the front end that we all see. It is used by internal dev groups as well as the web spiders that they employ. Their unique usage has definitely led to a number of interesting choices (such as the atomic appends) for the file system design. Read the paper for more details :-)

  4. Fabulous Insights by dolo666 · · Score: 4, Informative

    I really enjoyed that read about the file system Google uses. The fact that they usually append to their files, is of special note. By appending data you only need to know a simple pointer address. Seems quick enough. Add a bunch of threaded concurrent writes and you could get into trouble on other systems... The "atomic append" seems interesting because of the use of multiple machines to append simultaneously (hazard free).

    64meg chunk size is pretty huge, but I'm guessing that's blocked out based on continual threads of data, not typical files.

    At first glance, this file system seems fairly wasteful. But hey, Google likely require speed and reliability over cost. Right?

    This reminds me of the discussions about not-so-far-off database filesystems coming to an OS near you.