A Move to Secure Data by Scattering the Pieces

uler writes "The NY Times has an article about an interesting new open source storage project. Unlike data storage mechanisms today that work 'by making multiple copies of data,' the Cleversafe software takes an 'approach based on dispersing data in encrypted slices.' It's an elegant solution and one that's been a long time coming: the software uses algorithmic techniques known by mathematicians since the 70's. Adi Shamir (of RSA) first wrote of information dispersal is his 1979 paper 'How to Share a Secret (pdf).'"

Comments

Hmmm....

[Anonymous Coward]

PAR? PAR2?

Re:Hmmm....

[Disoculated]

Whomever marked this as offtopic was a little quick on the gun. I believe the coward is referring to PAR files, a method of breaking up data and reassembling it commonly used on newsgroups.

Wasn't this Al Gore's idea?

[andrewman327]

Although the goal was different, this is in the spirit of the creation of the Internet. DARPAnet was designed to scatter information to maintain communications. to use a different example it reminds me of RAID.

With all of this encryption technology, people still need to remember basic security tips. Use good passwords ("password" could be cracked very quickly even with 128 bit AES), maintain physical security (hardware keyloggers can find out about the manifesto you're writing before you even save the file) and use common sense.

Before you all ask, yes it does run Linux. The company was actually at Linuxworld.

Windows ME: Most Secure OS Ever?

[nick_davison]

Storing data in random locations, often garbled beyond all recognition?

Clearly Windows ME's memory -l-e-a-k-s- management made it the most secure OS ever. If only they had some way of reconstructing that data when you wanted it back again.

Number 1 of 4

[Red Flayer]

This concept just adds another layer

Freenet?

[BigZaphod]

Isn't this basically what freenet does? It encrypts the data into chunks and spreads it around all over the place.

I was working on a p2p system that worked in a similar manner. I was even thinking of repurposing it for the sake of doing online backups - but frankly the bandwidth just doesn't seem to be there yet to do that sort of thing in a practical manner. That, and I got bored with the project... (but nevermind that). :-)

Re:Freenet?

[mrogers]

Freenet uses forward error correction, which guarantees that the original data can be reconstructed given a sufficient number of pieces. Shamir's information dispersal algorithm makes the additional guarantee that nothing can be learned about the original data unless you have enough pieces to reconstruct it.

6 of 11

[Bonker]

After RTFA, it occurs that this is mostly a research project. The goals (and downloadables) include libraries that allow PCs to mount a distributed encrypted filesystem and others.

In a business example where you know that you can ultimately control the sites where you're storing your partial data, this would be a very good thing.

For the single user attempting to secure his information by using the existing network, there are some downfalls. 6 of 1l slices of the data are needed to recontstruct the whole. Therefore if a party intent on obtaining secret data obtains the majority of the servers, he has the data.

Also, if a disaster wipes out the majority of the servers, leaving five or less of the eleven, the data is gone.

This is a very, very important concept for business storage, but I have to wonder if it scratches any geek itches not already soothed by Truecrypt and Par2.

Number 2 of 4

[Red Flayer]

See Comment 15948676

Of complexity, but also adds

Number 3 of 4

[Red Flayer]

See Comment 15948695Another layer of inefficiency and

Like mnet?

[haeger]

If I'm not mistaken, this was one of the goals with the (now dead?) mnet project.
From what I remember they split up data into multiple pieces, encrypted it and distributed it over a number of nodes, with some redundancy in it. If you know python and are intrested in p2p I'm sure there's a lot to be learned from that project.

.haeger

Number 4 of 4

[Red Flayer]

See comment 15948718

an increased risk of loss of data.

Burma Shave.

new implimentation of an old idea

[bingbong]

Ross Anderson of the Computer Security Group at Cambridge University wrote a paper called the Eternity Service. It has had a few different attempts at implementation, as well as some reworks in terms of design. The primary difference is in the Eternity Service - you had no idea what data you had, nor did you have access to the keys. This new concept/design seems to provide more control/granualirity for the user. Given the new proposed encryption laws in the UK, I'm not sure this is a good idea.

Re:aaaaaaaaaarrrrrrrgggggggggghhhhhhh!

[Red Flayer]

It's '70s not 70's.

Not really -- it should be '70s' in all likelihood. The first apostrophe is to represent the missing "19", the second is to denote the possessive that is implied. The term "the 1970's" is a shortening of "the years of the decade we call the 1970s," or "the 1970s' years."

This gets messy, however, since the word 'years' is implied, and to say during the '70s' will make people wonder which 70 seconds you're talking about, and why it needs to be encapsulated with apostrophes -- is it an idiomatical 70 seconds? Kinda like the Biblical '40 days'?

For that matter, if you really want to get pedantic, what's the use of referencing the 70s at all if you're not going to bother denoting the scale? I mean, surely not mentioning that it's AD (or CE) is going to confuse people using other calendars... more so than misusing an apostrophe, right?

Along the same lines, it's just horrific that they'd abbreviate the decade anyway, how are we to know that the writer didn't intend the 1870s, or the 2070s even, if he happens to be living backwards in time?

Bah, there are grammatical rules, and it's great if everyone follows them, but really, it makes no difference if he spelled it 70's, '70s, or seventies (which is the proper spelling, btw).

Re:Ignorance is no excuse

[psmears]

Ignorantia legis neminem excusamus [sic] That's Latin for "ignorance of the *law* is no excuse"

(emphasis mine) That legal principle only means that you can't murder someone, and then claim “...but I didn’t know that was against the law” and hope to be let off. In order to commit a crime, you have to have mens rea (to use yet more Latin)—that is, a “guilty mind”—so storing data that you genuinely didn't know was illegal isn’t a crime. And in this case, there are plenty of possible, perfectly legal reasons for wanting to store encrypted data (for example, there’s a huge market for offsite backup for corporations, who tend to want to keep their stuff secret), so the storage providers are probably OK (disclaimer: the above is about criminal law, civil law is a bit murkier, IANAL etc). On the other hand, it would be very hard to be certain that no trace of the illegal data remained on the client machine (especially if you don’t want to type a twenty-word passphrase every time you want to listen to a new track ;-)

Sharing a secret in the offline world

[davidwr]

A friend taught me this. The secret in his case was a proprietary industrial process.

You take the secret and divide it into 3 pieces. You have a team of 3 people to each carry or memorize two of the 3 pieces.

Amy carries pieces 1 and 2
Bob carries pieces 2 and 3
Charlie carries pieces 3 and 1

If any one of them is compromised by bribery or other means, 1) the information is not lost and 2) the enemy has only an incomplete picture of what is going on.

This can be extended to more people to achieve greater redundancy or less exposure:

More redundancy: 4 people with 4 peices, each person knows 3 elements. Any 2 of 4 people needed to put the pieces together.

Less exposure: 4 people with 4 pieces, each knows 2 elements. Any 3 of 4 people needed to put the pieces together. Loss of 1 person exposes 1/2 of the total secret.

There's no reason to stop with 4 people and 4 pieces.

Think of this as RAID for human-knowledge.

Shamir? No

[pedantic bore]

The Information Dispersal Algorithm is due to Michael Rabin.
Shamir's secret-sharing algorithm uses a similar idea (it's
essentially the same as Rabin's algorithm, except that the
data is padded with random gibberish).

Notes from the Cleversafe lead developer

[mengland]

(Fyi: this link to the New York Times article bypasses any need to login/register with the nytimes.com website.)

I'm the Cleversafe Dispersed Storage software-development project leader. I work with Chris Gladwin (mentioned in the New York Times article) as a fellow manager at Cleversafe.

I offer some comments below to help outline some of the unique aspects of the Cleversafe technology.

Encryption is not dispersal. Cleversafe provides both, and then some. The Cleversafe Dispersed Storage software disperses any "datasource" (typically a file) into several slices (our current software current uses 11 slices in an 11-lose-any-5 scheme; future versions may use additional schemes with "wider" slice sets). Additionally, our software also encrypts, compresses, scrambles, and signs the datasource content, but we are not trying to reinvent the wheel: other software technologies exist to do these things, and we leverage them extensively.

We found that a bigger challenge than creating or managing dispersal algorithms was to make the entire storage system regardless of the dispersal algorithm used (and we design the system to be dispersal-scheme agnostic). The meta-data management system and many other things took us far longer to implement then the Cleversafe IDA. It's not hard to use Reed-Solomon, or some other algorithm on a single file or a small set of files and disperse the slices by hand onto several different system (or use variants of this like the 3-piece secret story with Amy, Bob, and Charlie mentioned above). It's much harder to manage this across an entire file system (with hundreds of thousand of files--or many more depending on the file system) for an unlimited number of file systems from all the various users across to be stored on heterogeneous set of an unlimited-number of geographically-dispersed, commodity-storage nodes in a completely-decentralized way with no dependence on the original source of the data (eg, you could sledgehammer your laptop and not lose any data that's stored on our grid/storage service). (I apologize for that run-on sentence.)

Further, dispersed-storage systems do not require replication. (Dispersal systems may replicate data for performance purposes, if at all, depending on the application/configuration/installation/context.) If a system replicates entire copies of the data (be they encrypted or not) then it, by (our) definition is not a dispersed-storage system. So a continual question I have when evaluate other systems: do they replicate the data in whole or not? Most systems replicate.

Cleversafe is not the first to present a dispersal system, but we like to think we are the first to make it broadly usable by people and inter-operable with other systems. See our cmdline client (which will soon have continous-backup and XML-programmable policy management), our Dispersed Storage API, our dsgfs file system, a soon-to-be released GUI client, and future "connectors" (what we call the applications that leverage our technology) to come, all available at http://www.cleversafe.org.

A side note: "revision management" is built into the Cleversafe system to address what I call "soft" failures (accidental deletes, application failures, etc) vs. "hard" failures (hard disk crashes) as well as archival requirements.

I believe that the concept of "dispersed storage" will eventually change how the world thinks about storage systems--regardless of whether or not these are Cleversafe-based systems (I think Cleversafe presents the best such system, but I of course am biased).