Slashdot Mirror
Open Source Moving in on the Data Storage World
pararox writes "The data storage and backup world is one of stagnant technologies and cronyism. A neat little open source project, called Cleversafe, is trying to dispell of that notion. Using the information dispersal algorithm originally conceived of by Michael Rabin (of RSA fame), the software splits every file you backup into small slices, any majority of which can be used to perfectly recreate all the original data. The software is also very scalable, allowing you to run your own backup grid on a single desktop or across thousands of machines."
The data storage and backup world is one of stagnant technologies and cronyism.
It's not offtopic, dumbass. It's orthogonal.
Editors please note!
Editors, please note that there is some incorrect information in this post. Firstly, the original concept of the IDA was designed by Shamir of RSA fame, not Rabin.
Also note that the Cleversafe IDA is a custom algorithm, and is only similar to Shamir's initial concept.
I can't find this in the FAQ -- is there a "creator/seeder" in the whole process? Which means a particular group of slices can only be unlocked by a particular seeder created by Turbo IDA.
If there is a creator/seeder, then we are still burdened by having to keep this seeder safe so that we can retrieve the distributed slices.
If there is no creator/seeder, is this safe enough so that people cannot patch slices together by way of trial-and-error?
Please stop entering code 2,2,7,6,6,4
At work we're looking into this to store critical data on out intranet which spans several states and facilites. Looks great, but only time will tell.
I seem to remember a project months ago that was going to use P2P to backup your data on other P2P users computers which to me sounds quite insane. Anyone know if this is related?
http://religiousfreaks.com/While Michael Rabin was inventor of the Rabin cryptosystem in 1979, it was Ronald Rivest, Adi Shamir and Len Adleman behind RSA two years earlier.
Speak for yourself. I have all my old business buddies back up my data for me.
Using the information dispersal algorithm originally conceived of by Michael Rabin (of RSA fame), the software splits every file you backup into small slices, any majority of which can be used to perfectly recreate all the original data.
It seems like this can be tuned to provide varying levels of fault tolerance. According to the abstract (I don't have an ACM web account, and I couldn't find the full text), it seems like I can take a file and make it so that any four chunks can be used to rebuild the file. I can then take those chunks and distribute them eight times to different machines. Thus, five of the eight machines would have to be rendered inoperable before I were unable to retrieve my data.
If I understand it correctly, then this is really slick.
Companies are crying out for new storage solutions all the time. If the answer is slow in coming it is not due to "cronyism" and "stangnation". Rather the causes include the facts that distributed storage is hard, and people don't like loosing their data.
Since all we need is a majority of files, its a realtime compression scheme of 51%. ------ Thats what I would do. You do whatever you want.
The software is also very scalable, allowing you to run your own backup grid on a single desktop or across thousands of machines.
Not good enough for the website to avoid being slashdotted. Maybe the technology is still Beta?
This sounds like Rar, Par, and BitTorrent got merged in some freak transporter accident...
Par files (for use with QuickPar, etc) are great, saving all sorts of extra posting on binary newsgroups.
Lose Weight and Feel Great with Isagenix
so it's sort of like parchive http://parchive.sourceforge.net/ which is software splits every file you backup into small slices, any majority of which can be used to perfectly recreate all the original data
Well, their webserver seems like it's been smoked, here's a link to their sourceforge page, where you can grab the actual software:
http://it.slashdot.org/it/06/04/26/2039224.shtml
It's a perfectly cromulent word.
While R in RSA stands for Ron Rivest, it is Adi Shamir (S of RSA) you have in mind. He came up with a wonderful secret sharing scheme which allows a bunch of folks or computers to keep pieces of secret in such a way that no N of them have any idea what the secret is, even if they collude. OTOH N+1 of them can easily figure out the secret. RSA can help you keep important secrets safe this way: if the owner is OK, the secret cannot be recreated; if the owner quits or dies, all-important secret holders can recover his password and unencrypt critical company data. And if a couple of them cannot participate, you still can get your secret back.
Even more amazingly Shamir's secret sharing scheme allows computing math functions, such as digital signatures, without ever recovering secret keys. This is called threshold cryptography, some of you may be interested to learn about its many wonders. Shamir rocks and so is threshold crypto!
Maybe one day vendors will stop pushing overly expensive and utterly bland storage solutions. i.e. Last time I had a meeting about storage the product was: 2x Servers 2x Disk Arrays with possible storage of a little under 2TB (using 24 80Gb SCSI HDDs) with RAID 5, Oh and the storage was presented as 4 @500Gb drives to the OS (Some proprietary thing). all in at a cool £27.000, (and that was before the license for CIFS) guess how it was billed - innovative... Its a joke, so the solution? In the meantime lots of SATA Drives and file replication, eventually? maybe we can make use of all that storage that sits on every machine on the LAN that is never used...
Why bother?
I just rely on Echelon for my data backups...
If you disagree with me on social issues, then it's pretty clear that you are a narrow-minded bigot.
Related companies/projects happened in this order: MojoNation .. MNet .. HiveCache .. AllMyData
good luck!
In the early 90s, a company made a virtual file server for networked Macs. Each client Macintosh had a file on its hard drive, and when a request was made through the driver, a number of Macs were contacted, and files were read and written to in a fairly load balanced fashion. I'm pretty sure it used some decent (think single DES) encryption at the time too, so someone couldn't just dig through the server's file on their Mac's hard disk and glean important data. It also added some redundancy, so if a Mac or two wasn't up on the network, it wouldn't kill the virtual Appleshare folder.
By chance, anyone remember this technology? I have no idea what happened to it, but it would be a blockbuster open source app if done today, and was platform independant. If done right, one could create data brokerage houses, where people could buy and sell storage space, and also reliability, where space on a RAID or server array would be of higher value than space on a laptop that is rarely on the Internet.
generally, speaking the more copies of something you have floating around, the larger the probability they get into the wrong hands. so this whole redundancy thing is just going to be viewed as a huge security breach, and never really become popular...
Jesus said to his disciples: "If you don't have a sword, sell your cloak and buy one" - Luke 22:36
sounds a lot like content addressable storage.. oh wait, thats what it is... nevermind, i've already deployed that were i work...
I was immediately visualizing a Borg Cube regenerating after a hit from the Enterprise.
regardless, it sounds cool.
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
http://labs.google.com/papers/gfs.html
Very roughly, this is what GFS does. I dn't have 25,000 servers at my disposal, so I haven't been able to test it though. Maybe next week. Meanwhile, I muddle through with tape.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
i haven't remembered the name yet, but the company was bought by novell shortly before NDS came out. i always thought it was how NDS replicated itself around w/o eating up the network while trying to take care of itself.
The most interesting link here is behind a pay-wall. Do the editors bother to follow the link in articles? Do they just assume we all have ACM access? Come on, this place used to be a bit better than this, didn;t it?
http://www.acooke.org
I just hope they don't patent it!
Am I part of the core demographic for Swedish Fish?
We all knew that.
http://www.networkmirror.com/ekXdq3RqHwPz5vfh/www. cleversafe.org/index.html
The others are there too.
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
By chance, anyone remember this technology? I have no idea what happened to it, but it would be a blockbuster open source app if done today, and was platform independant.
That's very interesting. If I understand what you're saying, was it something like this? That's a description I wrote up for a system I'd like to build if I every get the time.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
This can be done quite easily with Reed-Solomon coding. In fact, you don't need the majority of the nodes, but simply an arbitrary N set of nodes, with an arbitrary M nodes as redundancy. N=1 and M=1 is basically RAID1. N = n and M = 1 is simply RAID5, N=n and M=2 is RAID 6.
In fact, I wrote a RSRaid driver for Linux for my thesis and did some performance testing on it. I'll save you the 30 pages and just tell you that the algorithm is far too CPU intensive to scale up very well for fileserver use (my original intent,) but I did conclude it could be used as a backup alternative to tape. Hmmmm.
Direct Link
Google Cache
Please forgive the double brackets, I fought witH Word and lost.
Contact me if you'd like to play with the code. I never did any reconstruction code, but the system did work in a degraded state, and was written for the Linux 2.6 kernel.
Tell me how you restore from Echelon, and I'm sure many of us will start using the service ;)
The basis of the method lies in the Byzantine General's Problem and related mathematical puzzles. A derivative is used in cryptography for distributed keys. As a backup strategy, it looks interesting - you don't need any higher level of trust than you would need in the Byzantine General's Problem, for exactly the same reasons. This includes not just backup devices but also all connections to backup devices (so you have security against SAN failures, packet corruption and other such problems). The price you pay for this added security and reliability is that it is going to be either extremely slow or more expensive.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
It's nice to see another attempt that's free. Free speech requires anonymity.
Friends don't help friends install M$ junk.
As they appear to be toast now...
And how can you say backing up to a *single* desktop pc is of any value?
---- Booth was a patriot ----
One point that's been brought home to me in a very real way, in my position in senior support for one of the major storage system vendors: the hard disks themselves really do make a difference. SCSI disks are much more expensive because of their construction, the duty cycles they can perform to over long periods. You can NOT hammer a SATA disk at 90% of the time, 24/7, and expect it to last the way an enterprise-class SCSI disk does. My company sells low-cost SATA disk systems too, and some customers find that the lower price is a false economy for what they need the system to do.
I'm kinda missing the point of the "editorializing" in this article: when a storage system is doing its job, it IS boring. You put bytes in, assured they will be stored, and you get them out on demand. You want nothing "interesting" to happen to the data that your business is built on! Sure, the technology is stagnant, if that means customers can get access to the data, reliably, year after year. We Slashdotters are prepared to take "bleeding edge" risks that enterprise customers are not.
(this is not a
...for my alma mater.
Cleversafe's headquarters are located at the new University Technology Park at IIT...no, not that IIT, this one.
Comment removed based on user account deletion
if they find out you think their products are boring.....
Anyone who has used usenet in the last decade or so knows most binaries are split into multiple parts (RAR's now-a-days) with PAR and PAR2 recovery volumes. So instead of making this sound like an awesome new development, why not be honest about what it is: a slightly different application of a very old technology/algorithm.
"However, if you take a k dimensional vector and compute the dot product with l mutually orthogonal vectors (where l > k), then any k dot products are enough to reconstruct the original vector."
Do you mean that we have a k-dimensional vector space V, a vector on this vector space and calculate the dot product with l mutually orthogonal vectors where l>k?
Is that it? Because if it is it's strange to say the least.
https://www.accountkiller.com/removal-requested
Slashdotted! Can't check the site contents or the wiki.
From the summary : "the software splits every file you backup into small slices, any majority of which can be used to perfectly recreate all the original data."
So, basically it is like RAID 5 striping and parity applied to the file level.
Neat concept.
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.
I'm guessing that someone who is "hooked on phonics" was trying to say anachronism.
Nice fileserver you 'ave there. Shame if somefing were to 'appen to it. Know what I mean, 'squire?
Looks like it's back up!
Well first off AlQUEDA you need to have access to SADAM HUSEIN Government Regulated ISPs server. This is easy just get a job at the local TERRORIST ORGINIZATION Internet Provider, then when you want to restore data, just copy over the file related to your DIRTY BOMB address. Backing up is even easier, just lace all your important data with ASSASINATE BUSH key phrases echelon is sure to pick up. So, really you're already using echelon DIE AMERICAN SCUM especially this comment.
The paper for a backup system is excellent -- it covers all bases of what it should be, encryption, redundancy, and robustness in case the master controller is lost. (I like your idea of the master controller storing metadata, similar to how TSM, Networker, and Retrospect store backup catalogs, and if the master is down, having the clients "recatalog" themselves to a new master.)
As for this Mac backup program, it was around around '91-'92, about the time of System 7's release. It was "merely" a control-panel extension at the time, where one installed it, rebooted, set how much HD space to give to the virtual file server, and went on your merry way. It was not intended to be scaled to an enterprise, or the public internet, but was intended for Appletalk networks (as in the old hardware and broadcast protocol which worked remarkably well in its time), instead of purchasing a dedicated Mac with Appleshare on it. (At the time, Appleshare was a server that took over the whole machine that was similar to Netware, but was intended for Macs.)
I wonder what ever happened to the company's IP and source code. Hopefully its not sitting on some old SCSI-1 drive in some clearinghouse, slowly bit-rotting away. Even worse, the code of this program ending up lost to history. Even if it were written in 680x0 assembly or THINK Pascal, it could be translated, probably with a lot of effort, to work supporting generic UNIX VFS, and Windows drives. For the Appletalk code, hopefully someone could gut it out, replace it with either a broadcast protocol, or something better for auto-discovering clients.
Very interesting - thanks for posting the link.
I'd love to see some figures for that baby running on a multi processor system with Gig-E to other nodes......
Keep up the good work!
Is this algorithm of interest to biologists who are working on how information is stored in brains? It seems likely to me that this could be interesting for that type of research.
Slashdot is becoming almost as bad as LtU!
The editor I hired after I sacked the last one, has been sacked.
KFG
(This is a repost from an earlier part of the thread so that I can get these comments on the toplevel.)
Hello-
I am the lead designer of the first Cleversafe dispersed-storage system (aka a grid-storage software system) and am one of the project's co-founders. The Cleversafe system never stores a complete copy of the data in any one place (or "grid node" in our terminology). At most 1/11th of the file data--we call it a file "slices"--is stored at any one grid node in a "scrambled" (i.e., non-contiguous), compressed, and encrypted/signed fashion. The grid _never_ stores more than one copy of the data on the grid, and that one copy is never stored all in the same place--it's dispersed using an optimized information-dispersal algorithm that we created but has similar properties to the previously-published info-dispersal algorithms (IDAs).
If a grid node and its associated content--i.e., the user's file slices on that node--are ever completely compromised (firewall comes down, all encryption and scrambling is cracked, etc), then the cracker acquires at most 1/11th (one-eleventh) of the data users data.
Further, if any half (or at least 5 out of any 11) of the grid nodes are for any reason destroyed or otherwise unavailable, all of the user's data is still accessible. This is done by generating a "coded" file slice for every data slice that we store on the node, and regenerating missing file slices from down nodes by pumping the available data and coded slices through our info-dispersal algorithms (which are all open-sourced, by the way) that are executed on the client side or when the grid "self heals" for destroyed nodes.
The system can also be implemented in a cost-effective fashion. The grid system can sustain so many concurrent, per-node outages that the availability/uptime requirements for each node are minimal. Also, the grid-node servers need not support much processing capability, for the client offloads much of the work from the servers.
We feel this system provides a powerful combination of reliability, scalability, economy, and security.
The hardest part of the design, imo, is to be able to reliably track all of these file slices across a large and heterogeneous set of grid-node machines housing these info-dispersed file slices. We designed the grid meta-data system from the ground up to do this and to be capacity-expandable, performance-scalable, and easily serviceable. More details for the open-source flavor of the grid-software design can be found here:
http://wiki.cleversafe.org/Grid_Design [cleversafe.org]
There's much more that I can say about this system; I plan to add additional comments to this thread as more questions and comments arise. I'm sure there are new comments I have yet to read, for they're coming in pretty quickly...
I also encourage further discussion at our newly-created web forums: http://forums.cleversafe.org/ [cleversafe.org]
Mailing lists (that will be synchronized with the web forums) will also be available at cleverafe.org in the near future.
-Matt
Cleversafe project lead
Then with my 11 GMail accounts I get something like 10GB of free, secure, offsite data backup!
-- Moderation in all things, exceptions to all rules --
I wonder what ever happened to the company's IP and source code. Hopefully its not sitting on some old SCSI-1 drive in some clearinghouse, slowly bit-rotting away. Even worse, the code of this program ending up lost to history.
This is why I think software vendors that wish to obtain copyright protection for their software should be required to publish source code along with the binaries. They wouldn't be required to grant any license whatsoever, so no one would have any right to use the source -- not even to compile it, which is preparation of a derivative work -- but it would greatly increase the probability that the code would not simply be lost.
If that were the case, we'd probably have that source code somewhere, and even if the copyright holder weren't around to give permission to use the code, or wasn't willing to, at least we could read the source and get ideas about how various problems were addressed. And that, by the way, is exactly what copyright is for... copyright exists in order to enrich the public domain by increasing the dissemination of ideas. It does this by restricting copies of particular expressions of ideas, but the core goal is (or once was, anyway) to encourage the spread of ideas.
Anyway, if you can't tell this is one of my pet issues :-)
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
The last storage proposal I heard was from Lefthand Networks for their iSCSI based SAN systems.
They basically sell a stack of drive arrays which you can configure as volumes as you see fit. Some notable features:
Ability to configure multiple RAID types within the stack. So you could have RAID 10 and RAID 0 within the same stack of drives depending on if you need speed or redundancy.
The ability to stripe the data and parity across units in the whole stack (RAID 10 level 2 and 3). So if you have 3 4 drive systems in a stack a volume can be spread across the entire stack so that even 2 drive failures in a single unit of the stack cause no data loss or downtime.
Storage provisioning that can grow and shrink (thin provisioning I think is the current catch phrase). So if you allocate 100GB to a department and they go over, you get notified but they don't get a disk full message. The storage is allocated automatically but can be reduced back down.
Snap shots of changed data. You can set it to create a snap shot of the data at certain times of day allowing an employee to retrive a file from earlier if they accidentally hose it without the need to go to tape. You can even snap shot your data to a non-RAID volume (to conserve space) just to be used during a backup to tape. The snap shot is backed up and you don't worry about open file backup since the live volume is not the one being backed up.
You can also do bit level replication over small data pipes to other location. Great for transfering data from remote offices back to a central SAN for transfer to tape. Also useful for offsite backup for critical data. Because it isn't doing full file replication you can do this over a T1 line.
Another cool thing is that you can add on additional units to increase the overall storage capacity, but you don't need to create new network shares or volumes. You can expand your existing volumes so that it is transparent to the end user. Plus each unit is running 2 gigabit Ethernet NICS together increasing stack performance as you scale the size of the SAN. From the latest benchmarks I've seen, iSCSI appears to be getting pretty close to 2Gbps fiber channel in performance, and you don't need a special switch for it.
In general iSCSI is starting to push SAN technology down into the small/medium business space. It definately isn't cheap but it is cost comparable to the 4TB of NAS (2 mirrored 2TB units in RAID 5) we had put in for our last round of storage upgrades 3 years ago - and has a ton more features.
Lefthand definately isn't the only company in the iSCSI space by a longshot - but I did really like the feature set. The only thing I have a real issue for is the apparent current lack of an iSCSI iniator for Mac OSX. I'm still researching but I may have to lose some features and go with a fiber channel XSAN on the production side of the business.
Sometimes my arms bend back.
3Par InServe is built on Linux.
We have a couple in a closet at work.
www.3par.com
You are right. The "R" in RSA stands for (Ron) Rivest. On the other hand, Prof. Rabin won the Turing award many many years ago...
Facts:
1) Information dispersal algorithm (IDA) was invented by Professor Michael Rabin at Harvard. IDA is an algorithm for distributed storage.
2) R in RSA stands for Professor Ronald Rivest at MIT. This article has nothing to do with RSA.
More notes on our IDAs compared with others:
s
The Cleversafe information dispersal algorithms (IDAs) were designed to provide real-time performance with large amounts of data storage and retrieval (gigabytes, petabytes and above). Previous algorithms, like Rabin, Shamir and Reed-Solomon, are very effective at storing smaller amounts of data (kilobytes), but their computational overhead which is proportional to the square of the data block size or greater arent well suited for quickly dispersing/restoring larger amounts of data. The Cleversafe algorithms encode AND decode data with a computational overhead that is linearly proportional to the size of the data blocks. Specifically, the Cleversafe encoding algorithms for an 11 node grid with a threshold level of 6, required 5 operations per byte to encode data. For decoding on this dispersed storage grid, the Cleversafe algorithms require 4 operations per byte to decode data greater than 99% of the time and no more than 13 operations per byte in rare cases.
Another Cleversafe contributor, Chris Gladwin, developed our IDAs. For more info:
http://wiki.cleversafe.org/Turbo_IDA_Technology
On can also read an Excel spreadsheet (found in the above wiki page) and C++ source code that represents the "guts" of our 11-Pillar IDA code module.
For more info about Cleversafe contributors:
http://wiki.cleversafe.org/Cleversafe_Contributor
You can see Chris and I at the bottom of the page which is ordered with the most-recent contributor listed first.
-Matt England
ps: We are finishing up our project announcement at this week's MySQL User's Conference where we drew significant interest. We have engaged some MySQL core developers regarding integrating the their technology with ours.
Well, according to the spec you can only restore from ECHELON by going through the FISA_COURTS interface, but everyone's known for a while now it's been backdoored if you have sufficient privileges.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
1. It does seem like RAID5+ at the file level. More redundant, so it seems cooler to me.
2. Has anyone read an article on Google's file system? This sounds a lot like it. Multiple stripes, recovery with less than N-2 parts, and Google uses it to improve performance first, with copies worldwide more or less. I think the article was in Wired, but I'm too lazy to look it up.
New? Maybe. Improved? Maybe. Cool? I want.
rick
deleting the extra space after periods so i can stay relevant, yeah.
Anybody know of any implementations of this? Seems like there would be a lot of rather obvious uses, but I've never heard of it being used. (Or maybe I have and just didn't recognize it as such.)
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Still stagnant technology. Recovery volumes for various archival utilities have been around a long time. This is just the first time that I know of where they use the RSA algorithm instead of an older algorithm.
ou backup into small slices, any majority of which can be used
... I won't trust my data to something I don't even understand. You can say to RTFM, but hey, this is the first paragraph about the software, it should be catchy and clear.
Ok, I'm numb in the morning, but what the hell does that mean ?
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
I have always wanted to post that :)
Read, refresh, repeat.
Rivest, Shamir, Adleman invented RSA.
Shamir invented secret sharing.
Rabin invented the Rabin public key cryptosystem, and IDA.
IDA is not like secret sharing.
With secret sharing, you have a secret, which you break up into shares. You can decide how many shares you need to reconstruct the secret when you break it up. Without the right number of shares, you know nothing about the secret. But the big difference is that EACH SHARE IS SLIGHTLY BIGGER THAN THE INITIAL SECRET.
With IDA, you have lots of data. You break it up into chunks. EACH CHUNK IS SMALL COMPARED TO THE SIZE OF THE INITIAL DATA. The total size of the chunks is bigger than the size of the data. When the chunks you have add up to a size slightly bigger than the initial data, you can reconstruct the initial data.
That was a lot of confusion to untangle.
Xenu loves you!
a) Make a backup of your data and encrypt it.
b) Call the file "Britneys secret sex tape - the real thing!" (or whatever)
c) Share it on eMule.
That's it. Your valuable data will replicate itself around the world ad-infinitum. You'll be able to access a copy from anywhere in the world with an Internet connection.
No sig today...
I skimmed through the links in the post and did not find answers to following questions:
what is the storage size to datasize ratio? I am talking about practically meaningful numbers ensuring storage reliability comparable to the competitors.
what is the storage reliability at the storage size to datasize ratio comparable to the competitors.
Theoretical estimates will suffice.
PS I do not have access to the full text of linked ACM paper
I do not believe in karma. "Funny"=-6. Do good and forbid evil. Yours, Oft-Offtopic Flamebaiting Troll.
No windows binary ? I can see this taking off quickly...
Capacity Networks have been doing this for ages.
http://www.capacitynetworks.com/
Good to see some open source guys doing the same thing. The CapFS idea is just awesome.
Why not replace the AppleTalk portion with ZeroConf/Rendezvous/Bonjour? That seems to be what AppleTalk grew up to be anyway.
Media that can be recorded and distributed can be recorded and distributed.
-kfg
Small world,
I also did my master's thesis on a concept like this. I was doing this splitting and syncing using virtual disks. They communicated on AES encrypted TCP packets. I idea was it would be a system simple enough for any to set up as a bunch of friends or families.
I've been too busy to work too much on it since graduating, but I opened all of the code and it's on sourceforge.
http://sourceforge.net/projects/netraid/
It may be inaccurate (I don't know), but how can you say a priori that the word is not being used to describe what it defines?
Imagine a scenario where the Legato sales rep is buddy-buddy with the CIO: takes him out golfing and dinners on Legato's dime, in exchange the CIO sticks with Legato no matter how much the product may diverge from the company's true needs. The sales rep keeps a lucrative contract and the CIO gets free stuff: regardless of the performance of each.
I don't think genuine friendship is needed wrt to cronyism. If you want to label this strictly bribery, than how is this any different than two old frat brothers who collude to enrich each other?
I believe Microsoft does the same things under the covers in their BitTorrent alternative, plus some consideration for locality.
Article here
I recall a mid-to-late 90s product called Mango. It was a distributed network storage program for Windows. Though the company's namesake is still around, I'm pretty sure the product effectively vanished. Seemed a neat idea at the time.
Method of processing duck feet
Yes, it's hard to dig anything up on them anymore. Their product was only compatible to Windows 95, and never got rolling on the NT kernel. I bought a couple of copies surplus hoping to run them on something, but never got around to it. Neat in concept.
Found an old slashdot post by a couple of former Mango engineers: link
http://research.microsoft.com/sn/Farsite/
Pretty cool stuff, check this out:
Lots more questions answered on the FAQ: http://research.microsoft.com/sn/Farsite/faq.aspx
From: http://research.microsoft.com/sn/Farsite/
...store an entire file/data/content set (encrypted or otherwise) in one place? ...make multiple copies of the data?
It does this by distributing multiple encrypted replicas of each file among a set of client machines.
Therein lies the key. There exist many systems that copy entire files (or sets of data) to many machines/nodes. I have been introduced to several references to many other projects that claim similar things with similar language to projects like Farsite.
The Cleversafe system never stores an entire file (or data/file set) in any one place, encrypted or not. Only portions of any file (known as file "slices") are stored anywhere on the Cleversafe dispersed-storage grid. In our (Cleversafe's) opinion, this reduces complexity (by not having to synchronize multiple copies) and increases security and privacy (by never storing all of the data in one place), among other things.
In short, some key differentiating question I typically ask when investigating Cleversafe-competitive systems:
Does the system...
*
*
If either answer is yes, then I tend to view the project as significantly different then the Cleversafe technology. I have found full-replication-based methods in many various forms are quite prevelant in many applications.
-Matt
Taking an excerpt from a previous post I made on another sub-thread:
I felt it worth noting at the top level of this thread:
The Cleversafe meta-data system was designed with an attempt to be able to easily use any information-dispersal algorithm (IDA) available today (including the Reed-Solomon, Shamir, and current Cleversafe methods) or in the future. In fact, the current Cleversafe IDA represents a small part of the code; the vast majority of the code and development effort can be found in the meta-data-management system to track data slices from an unlimited number of files originating from an unlimited number of users and computing systems; this also needs to be done in a way such that the entire system can tolerate and tremendous number of concurrent failures from the underlying system components.
Is Cleversafe the first one to design a "hyper-redundant," grid-like, meta-data-management system? No. However, we believe we are the only ones to have built such a robust system based on an IDA mechanism with absolutely no replication of the data--and therefore, we contend, much less complexity. Further, I believe that this reduced complexity (when compared with other distributed file/meta-data systems) enables many powerful features, including performance scalability and better human serviceability.
Will the Cleversafe system prove to be uniquely valuable? We believe so. However, as at least one other post on this thread mentions: time will tell.
It's also important for me to reiterate: I personally designed much of the current meta-data system, so I present an obviously-biased perspective.
-Matt
Anonymous writer writes:
Recovery volumes for various archival utilities have been around a long time. This is just the first time that I know of where they use the RSA algorithm instead of an older algorithm.
To be clear:
Dispersal is not encryption. (Cleversafe uses both.)
While we (Cleversafe) do use public-private key methods to encrypt the data/content, this is still a separate operation from the data *dispersal*.
Moreover, if the content encryption is somehow cracked/broken (and public-private key encryption can be broken), the cracker acquires at most 1/11th (in our current IDA scheme) of "scrambled"/non-contiguous data.
This is the major reason why we feel that our system provides unique, security-and-privacy-based value over encryption-only based systems. If the encryption breaks, you still can't get the data. (And of course, we use the encryption mechanism, too.)
Note that a different RSA key can be used to encrypt each file Slice (ie, for each Cleversafe "Pillar," as per our terminology for our grid design) such that if a cracker breaks one slice/Pillar's key, they still have to break the key for other Pillars (and there are 11 total Pillars in the current IDA scheme)...*in addition* to the "toplevel" key we use to encrypt the file before it's sliced/dispersed. Note: we don't have this post-dispersed-encryption feature in our current alpha4.1.3 code (we only encrypt the toplevel file before it's compressed and dispersed), but we believe it will not be difficult to add.
Also: we will be signing each slice as well, for data-integrity purposes to prevent both malicious and non-malicious data change/vandalism. This also will be a feature added in the near term.
One can read more about the open-source flavor of the Cleversafe grid design.
-Matt
ps: I encourage interested parties to continue discussions at http://forums.cleversafe.org/ (as well as to soon-to-be-available email lists that will synchronized with these forums).
Heh heh. You said "Cocks". Heh heh.
Yours truely,
Binkus or Buttwipe (I'm not sure witch)