A Move to Secure Data by Scattering the Pieces

uler writes "The NY Times has an article about an interesting new open source storage project. Unlike data storage mechanisms today that work 'by making multiple copies of data,' the Cleversafe software takes an 'approach based on dispersing data in encrypted slices.' It's an elegant solution and one that's been a long time coming: the software uses algorithmic techniques known by mathematicians since the 70's. Adi Shamir (of RSA) first wrote of information dispersal is his 1979 paper 'How to Share a Secret (pdf).'"

Comments

Hmmm....

[Anonymous Coward]

PAR? PAR2?

Re:Hmmm....

[Disoculated]

Whomever marked this as offtopic was a little quick on the gun. I believe the coward is referring to PAR files, a method of breaking up data and reassembling it commonly used on newsgroups.

Doesn't FreeNet do this?

[mb10ofBATX]

I've been out of the freenet loop for a long time, but I thought I remembered reading in its documentation a few years ago that it did this same kind of encrypting and dispersing chunks of data.

Wasn't this Al Gore's idea?

[andrewman327]

Although the goal was different, this is in the spirit of the creation of the Internet. DARPAnet was designed to scatter information to maintain communications. to use a different example it reminds me of RAID.

With all of this encryption technology, people still need to remember basic security tips. Use good passwords ("password" could be cracked very quickly even with 128 bit AES), maintain physical security (hardware keyloggers can find out about the manifesto you're writing before you even save the file) and use common sense.

Before you all ask, yes it does run Linux. The company was actually at Linuxworld.

Re:Wasn't this Al Gore's idea?

[Detritus]

Although the goal was different, this is in the spirit of the creation of the Internet. DARPAnet was designed to scatter information to maintain communications.

Cite? From what I've read about the original Arpanet, it was designed to allow the sharing of computer resources and data among DoD researchers. It wasn't designed to be a failure-tolerant network, although DARPA funded quite a bit of research in that area.

Windows ME: Most Secure OS Ever?

[nick_davison]

Storing data in random locations, often garbled beyond all recognition?

Clearly Windows ME's memory -l-e-a-k-s- management made it the most secure OS ever. If only they had some way of reconstructing that data when you wanted it back again.

...like network RAID?

[xxxJonBoyxxx]

The Project uses information dispersal algorithms (IDAs(TM)) to separate data into 11 unrecognizable DataSlices(TM) and distribute them, via secure Internet connections, to 11 storage locations throughout the world, creating a storage grid. With dispersed storage, transmission and storage of data is inherently private and secure. No single entire copy of the data is in one location, and only 6 out of the 11 nodes need to be available in order to perfectly retrieve the data.

...like network RAID? The site needs spellchecking - badly - but the encryption seems to be based on a key derived after you do some kind of RSA public/private key sign on.

Re:...like network RAID?

[Gary W. Longsine]

Interesting. Why 11?
"It's one louder."

Number 1 of 4

[Red Flayer]

This concept just adds another layer

Freenet?

[BigZaphod]

Isn't this basically what freenet does? It encrypts the data into chunks and spreads it around all over the place.

I was working on a p2p system that worked in a similar manner. I was even thinking of repurposing it for the sake of doing online backups - but frankly the bandwidth just doesn't seem to be there yet to do that sort of thing in a practical manner. That, and I got bored with the project... (but nevermind that). :-)

Re:Freenet?

[mrogers]

Freenet uses forward error correction, which guarantees that the original data can be reconstructed given a sufficient number of pieces. Shamir's information dispersal algorithm makes the additional guarantee that nothing can be learned about the original data unless you have enough pieces to reconstruct it.

aaaaaaaaaarrrrrrrgggggggggghhhhhhh!

[Anonymous Coward]

It's '70s not 70's.

Re:aaaaaaaaaarrrrrrrgggggggggghhhhhhh!

[joebutton]

It's '70s not 70's

Is it, though?

According to Lynne Truss's Eats, Shoots & Leaves:

Until quite recently, it was customary to write "MP's" and "1980's" - and in fact this convention still applies in America. British readers of The New Yorker who assume that this august publication in in constant ignorant error when it allows "1980's" evidently have no experience of how that famously punctilious periodical operates editorially.

Having said which, 1980s clearly makes more sense. It's a plural, not a possessive, innit?

Re:aaaaaaaaaarrrrrrrgggggggggghhhhhhh!

[Red Flayer]

It's '70s not 70's.

Not really -- it should be '70s' in all likelihood. The first apostrophe is to represent the missing "19", the second is to denote the possessive that is implied. The term "the 1970's" is a shortening of "the years of the decade we call the 1970s," or "the 1970s' years."

This gets messy, however, since the word 'years' is implied, and to say during the '70s' will make people wonder which 70 seconds you're talking about, and why it needs to be encapsulated with apostrophes -- is it an idiomatical 70 seconds? Kinda like the Biblical '40 days'?

For that matter, if you really want to get pedantic, what's the use of referencing the 70s at all if you're not going to bother denoting the scale? I mean, surely not mentioning that it's AD (or CE) is going to confuse people using other calendars... more so than misusing an apostrophe, right?

Along the same lines, it's just horrific that they'd abbreviate the decade anyway, how are we to know that the writer didn't intend the 1870s, or the 2070s even, if he happens to be living backwards in time?

Bah, there are grammatical rules, and it's great if everyone follows them, but really, it makes no difference if he spelled it 70's, '70s, or seventies (which is the proper spelling, btw).

6 of 11

[Bonker]

After RTFA, it occurs that this is mostly a research project. The goals (and downloadables) include libraries that allow PCs to mount a distributed encrypted filesystem and others.

In a business example where you know that you can ultimately control the sites where you're storing your partial data, this would be a very good thing.

For the single user attempting to secure his information by using the existing network, there are some downfalls. 6 of 1l slices of the data are needed to recontstruct the whole. Therefore if a party intent on obtaining secret data obtains the majority of the servers, he has the data.

Also, if a disaster wipes out the majority of the servers, leaving five or less of the eleven, the data is gone.

This is a very, very important concept for business storage, but I have to wonder if it scratches any geek itches not already soothed by Truecrypt and Par2.

Number 2 of 4

[Red Flayer]

See Comment 15948676

Of complexity, but also adds

Number 3 of 4

[Red Flayer]

See Comment 15948695Another layer of inefficiency and

Like mnet?

[haeger]

If I'm not mistaken, this was one of the goals with the (now dead?) mnet project.
From what I remember they split up data into multiple pieces, encrypted it and distributed it over a number of nodes, with some redundancy in it. If you know python and are intrested in p2p I'm sure there's a lot to be learned from that project.

.haeger

but. but, but...

[Anonymous Coward]

Secure Data by Scattering the Pieces

You mean to tell me that all those hours of defragging my HD's on Windows 98 were actually a waste of time?? ;-)

Number 4 of 4

[Red Flayer]

See comment 15948718

an increased risk of loss of data.

Burma Shave.

dispersion, section four

[thrillseeker]

... the ...

I thought of this a few years ago

[greg_barton]

I thought about a system to do this a few years ago, but with a little twist: distribution of the pieces would be via computer virus. The pieces would be stored in user's computers, but more importantly in intrusion logs of "secure" systems as well. Retrieval would be a social act, kind of like a treasure hunt. "Hey, geeks of the world, there's this important information out there. Go figure out how to get it!"

This system could be used for high profile secrets, like government whistle-blower data and the like. Storage would be secret and nearly undetectable because of all the other virus noise. Retrieval would be highly public by necessity, both to make retrieval possible and to publicize the contents of the data.

dispersion, section five

[thrillseeker]

... novelty.

You invented Lotus Notes

[Gary W. Longsine]

Lotus Notes doesn't really do this on purpose, but an artifact of the system architecture and user behavior (they want local copies of everything) seems to combine to provide a rudimentary capability of data recovery from widely distributed stores such as you describe. I observed a client once restore all the data on a Lotus Notes server following a catastrophic data loss (short chain of events meant no backups could not be recovered following the loss of a RAID filesystem, many GB of data gone). They put a call out their user community for the replicated copies of the various "database" thingies which existed on user laptops and desktop systems. They were able to recover all of the data that anyone cared about, anyway, if not actually all of it.

new implimentation of an old idea

[bingbong]

Ross Anderson of the Computer Security Group at Cambridge University wrote a paper called the Eternity Service. It has had a few different attempts at implementation, as well as some reworks in terms of design. The primary difference is in the Eternity Service - you had no idea what data you had, nor did you have access to the keys. This new concept/design seems to provide more control/granualirity for the user. Given the new proposed encryption laws in the UK, I'm not sure this is a good idea.

I don't see whats so new

[Alistar]

I've been doing something like this for years.

First I would encrypt the original file, split it up into 10-100 pieces, encrypt those, hide them in other files, encrypt those, then store them in random locations around the internet either by emailing a piece to a webmail or uploading to a server somewhere, posting the binary or hex sequence to a forum, things like that.

Heck sometimes I'd repeat the repeat the encrypt/split/hide process several times, or even put the last step as hidden. Yes I realize anyone with any computer talent could find a file hidden in another one, but it keeps it out of plain sight.
I also remove any identifiable information on what order the pieces go in, I rely on myself to remember. Or leave clues elsewhere.
I'll admit sometimes it takes like 3 days to gather and assemble them if I need them, though.

I use it for things that are better off gone forever than being leaked.

Re:I don't see whats so new

[Nerdfest]

Osama?

Do not mod up

[Anonymous Coward]

Copy

The problem...

[Fulkkari]

The problem with this idea is bandwidth and speed. You think your broadband is fast, but if you have to download the 27 gigabytes of photos, music and stuff, it won't be exactly fast on a 8 Mbps DSL, not to talk about 1 Mbps or less. You might wait a couple of hours, but you won't wait a couple of days.

Okay. So you tell me that amount of available bandwidth will increase? But so will the amount of data that needs to be backed up. And it will grow faster than the bandwidth. Think of homemade movies. You can already fill up your average drive in no-time. What do you then do, when you get a HD camera?

Although the idea isn't a new one, I think it is still neat. It might work for some stuff, but I don't see this becoming mainstream with technologies like Time Machine coming to the end-users.

That's how CDs work - distributed data

[Animats]

Not quite, but the coding scheme that makes CDs and DVDs resistant to dust and scratches works much like that. Big blocks have an error correcting code appended, and then the bits of the data plus error correcting code are rearranged and spread widely across the block. So when you lose a contiguous set of bits, you can replace it by using data distributed across the block.

It's a good error correction scheme, but it's not exactly new. Every CD player in the world has this. CDs aren't encrypted (there's no key, just an well-known algorithm), but you could mix encryption in if you wanted. This wouldn't help the error recovery.

Ancient

[Salamander]

This is so not-new it's not even funny. I've already seen FreeNet and MNet mentioned as precursors, which is appropriate. Dozens of other P2P "filesystems" (in quotes because I don't believe it's truly a filesystem unless it's fully integrated into the OS) and block-level data stores have done this. Probably the one that most thoroughly examined the inherent tradeoffs, and that's most directly based on Shamir's IDA work, is PASIS at CMU. Presenting Cleversafe as the first to move in this direction is an insult to those who have gone before.

Re:Ignorance is no excuse

[psmears]

Ignorantia legis neminem excusamus [sic] That's Latin for "ignorance of the *law* is no excuse"

(emphasis mine) That legal principle only means that you can't murder someone, and then claim “...but I didn’t know that was against the law” and hope to be let off. In order to commit a crime, you have to have mens rea (to use yet more Latin)—that is, a “guilty mind”—so storing data that you genuinely didn't know was illegal isn’t a crime. And in this case, there are plenty of possible, perfectly legal reasons for wanting to store encrypted data (for example, there’s a huge market for offsite backup for corporations, who tend to want to keep their stuff secret), so the storage providers are probably OK (disclaimer: the above is about criminal law, civil law is a bit murkier, IANAL etc). On the other hand, it would be very hard to be certain that no trace of the illegal data remained on the client machine (especially if you don’t want to type a twenty-word passphrase every time you want to listen to a new track ;-)

Sharing a secret in the offline world

[davidwr]

A friend taught me this. The secret in his case was a proprietary industrial process.

You take the secret and divide it into 3 pieces. You have a team of 3 people to each carry or memorize two of the 3 pieces.

Amy carries pieces 1 and 2
Bob carries pieces 2 and 3
Charlie carries pieces 3 and 1

If any one of them is compromised by bribery or other means, 1) the information is not lost and 2) the enemy has only an incomplete picture of what is going on.

This can be extended to more people to achieve greater redundancy or less exposure:

More redundancy: 4 people with 4 peices, each person knows 3 elements. Any 2 of 4 people needed to put the pieces together.

Less exposure: 4 people with 4 pieces, each knows 2 elements. Any 3 of 4 people needed to put the pieces together. Loss of 1 person exposes 1/2 of the total secret.

There's no reason to stop with 4 people and 4 pieces.

Think of this as RAID for human-knowledge.

Shamir? No

[pedantic bore]

The Information Dispersal Algorithm is due to Michael Rabin.
Shamir's secret-sharing algorithm uses a similar idea (it's
essentially the same as Rabin's algorithm, except that the
data is padded with random gibberish).

Notes from the Cleversafe lead developer

[mengland]

(Fyi: this link to the New York Times article bypasses any need to login/register with the nytimes.com website.)

I'm the Cleversafe Dispersed Storage software-development project leader. I work with Chris Gladwin (mentioned in the New York Times article) as a fellow manager at Cleversafe.

I offer some comments below to help outline some of the unique aspects of the Cleversafe technology.

Encryption is not dispersal. Cleversafe provides both, and then some. The Cleversafe Dispersed Storage software disperses any "datasource" (typically a file) into several slices (our current software current uses 11 slices in an 11-lose-any-5 scheme; future versions may use additional schemes with "wider" slice sets). Additionally, our software also encrypts, compresses, scrambles, and signs the datasource content, but we are not trying to reinvent the wheel: other software technologies exist to do these things, and we leverage them extensively.

We found that a bigger challenge than creating or managing dispersal algorithms was to make the entire storage system regardless of the dispersal algorithm used (and we design the system to be dispersal-scheme agnostic). The meta-data management system and many other things took us far longer to implement then the Cleversafe IDA. It's not hard to use Reed-Solomon, or some other algorithm on a single file or a small set of files and disperse the slices by hand onto several different system (or use variants of this like the 3-piece secret story with Amy, Bob, and Charlie mentioned above). It's much harder to manage this across an entire file system (with hundreds of thousand of files--or many more depending on the file system) for an unlimited number of file systems from all the various users across to be stored on heterogeneous set of an unlimited-number of geographically-dispersed, commodity-storage nodes in a completely-decentralized way with no dependence on the original source of the data (eg, you could sledgehammer your laptop and not lose any data that's stored on our grid/storage service). (I apologize for that run-on sentence.)

Further, dispersed-storage systems do not require replication. (Dispersal systems may replicate data for performance purposes, if at all, depending on the application/configuration/installation/context.) If a system replicates entire copies of the data (be they encrypted or not) then it, by (our) definition is not a dispersed-storage system. So a continual question I have when evaluate other systems: do they replicate the data in whole or not? Most systems replicate.

Cleversafe is not the first to present a dispersal system, but we like to think we are the first to make it broadly usable by people and inter-operable with other systems. See our cmdline client (which will soon have continous-backup and XML-programmable policy management), our Dispersed Storage API, our dsgfs file system, a soon-to-be released GUI client, and future "connectors" (what we call the applications that leverage our technology) to come, all available at http://www.cleversafe.org.

A side note: "revision management" is built into the Cleversafe system to address what I call "soft" failures (accidental deletes, application failures, etc) vs. "hard" failures (hard disk crashes) as well as archival requirements.

I believe that the concept of "dispersed storage" will eventually change how the world thinks about storage systems--regardless of whether or not these are Cleversafe-based systems (I think Cleversafe presents the best such system, but I of course am biased).

Potentially great for internal use...

[WoTG]

I've often wondered when someone would get around to perfecting a dispersed backup system for LAN's. With the average workstation toteing 100GB drives, and the average use of a handful of GB's, there seems to be a surplus of cheap disk space on the LAN... at least compared to backup tapes or other media. Though, in hindsight, I guess a single fire or building disaster would still be catastrophic...