The Cleversafe client is designed to be embeddable into mobile linux devices like mobile phones, media players, cameras, cars, etc. We (I am one of the Cleversafe developers) believe that more data and more critical data will move to mobile devices over time and a dispersed storage grid to store all that data will thus become increasingly important.
Chris Gladwin
In reading through a lot of the posts, I thought it might be useful to elaborate on how Cleversafe compares to current copy-based data storage systems as well as previous projects using similar techniques for data storage and communications...
Effectively all digital data storage in use today works by making copies of data and redundant copies of data with the use of parity bits when stored on a RAID array. Cleversafe does not store a copy of the original data and definitely does not store copies of data. Cleversafe 'disperses' data which is different than copying data. Original data files are turned into a set of 'dispersed files' or 'slices' -- each of which contains too little information to be useful on its own. These slices are then stored in different locations. On the current Cleversafe test grid, each file is dispersed into 11 slices which are each stored by a separate storage hosting providers in separate geographic locations as shown at http://www.cleversafe.org/wiki/Cleversafe_Research _Storage_Grid.
In order to ensure ultra-high availability, the dispersal algorithms are designed in such a manner that any majority of these slices can be used to perfectly recreate all the original data. This technique is similar to methods often employed in data communications where data is broken up into some number of packets by the sender in such a manner that the receiver does not need all the packets to recreate all the original data.
Over the past 25 years, a number of projects have looked at storing data using information dispersal or similar techniques. Many of these projects have used Reed Solomon or similar encoding / decoding techniques, including OceanStore, PAR/PAR2 and others. The Cleversafe project is not only developing algorithms for information dispersal, but is also creating a complete system to enable the benefits of Dispersed Storage to be practically used on a generally-available-to-everyone scale. So in addition to creating new, computationally-efficient algorithms for information dispersal, the Cleversafe project includes:
- a metadata management system for managing files stored on the grid
- grid management tools, including 'rebuilder' processes that enable the grid to 'self-heal'
- interfaces to enable dispersed storage to work in various existing environments, including a general API, a command line interface, a file system interface (which we began demonstrating at Linux World last week) and an upcoming GUI interface
- integration with existing methods for encryption
- live dispersed storage grids running on nodes operated by various storage hosting companies in various locations
- etc.
So, the focus of Cleversafe is to build on the previous work in dispersed storage (which has mainly been academic research) to create a practical and complete open source system to better store the world's data.
Slashdot Mirror
The Cleversafe client is designed to be embeddable into mobile linux devices like mobile phones, media players, cameras, cars, etc. We (I am one of the Cleversafe developers) believe that more data and more critical data will move to mobile devices over time and a dispersed storage grid to store all that data will thus become increasingly important. Chris Gladwin
In reading through a lot of the posts, I thought it might be useful to elaborate on how Cleversafe compares to current copy-based data storage systems as well as previous projects using similar techniques for data storage and communications...
h _Storage_Grid.
Effectively all digital data storage in use today works by making copies of data and redundant copies of data with the use of parity bits when stored on a RAID array. Cleversafe does not store a copy of the original data and definitely does not store copies of data. Cleversafe 'disperses' data which is different than copying data. Original data files are turned into a set of 'dispersed files' or 'slices' -- each of which contains too little information to be useful on its own. These slices are then stored in different locations. On the current Cleversafe test grid, each file is dispersed into 11 slices which are each stored by a separate storage hosting providers in separate geographic locations as shown at http://www.cleversafe.org/wiki/Cleversafe_Researc
In order to ensure ultra-high availability, the dispersal algorithms are designed in such a manner that any majority of these slices can be used to perfectly recreate all the original data. This technique is similar to methods often employed in data communications where data is broken up into some number of packets by the sender in such a manner that the receiver does not need all the packets to recreate all the original data.
Over the past 25 years, a number of projects have looked at storing data using information dispersal or similar techniques. Many of these projects have used Reed Solomon or similar encoding / decoding techniques, including OceanStore, PAR/PAR2 and others. The Cleversafe project is not only developing algorithms for information dispersal, but is also creating a complete system to enable the benefits of Dispersed Storage to be practically used on a generally-available-to-everyone scale. So in addition to creating new, computationally-efficient algorithms for information dispersal, the Cleversafe project includes:
- a metadata management system for managing files stored on the grid
- grid management tools, including 'rebuilder' processes that enable the grid to 'self-heal'
- interfaces to enable dispersed storage to work in various existing environments, including a general API, a command line interface, a file system interface (which we began demonstrating at Linux World last week) and an upcoming GUI interface
- integration with existing methods for encryption
- live dispersed storage grids running on nodes operated by various storage hosting companies in various locations
- etc.
So, the focus of Cleversafe is to build on the previous work in dispersed storage (which has mainly been academic research) to create a practical and complete open source system to better store the world's data.
Chris Gladwin