Slashdot Mirror

if you organization has good geographic distribution, why not consider something like:

http://www.hivecache.com/home.html

This allows you to use the excess capacity you already have (believe it or not, having 2+ gigabytes taken up by the same operating system/programs files distributed across all of your desktops falls under the catatory of "excess capacity"). The average corporate desktop has gigabytes upon gigabytes of unused diskspace and oddles of unused cycles (that's what the grid computing fad, in full inflamation about 2 years ago, was all about). Still, it's good to see something actually positive and useful come out of the p2p area.

It has encryption and allows users to self-service themselves with regard to restores.

Farsite. HiveCache. I even worked on a commercial offering: Mangomind (called Medley at the time). Some of these weren't positioned as backup solutions but, structurally, they're just like what Cringely describes. There have been many others, but I'll let people Google for themselves.

Still in startup stages, but growing bigger, and with a solid technology core. Portable to any platform which can run Python.

Disclaimer: No, I am not an employee. Yes, I probably would not know this if I didn't know a couple of their employees.

Something that caught my eye a while ago in this area was HiveCache. Never used it, don't know anyone that has, but it looks like a pretty cool system.

While a pure linux solution seems to score the most points here, this particular one lets you combine your windows, OS X, and linux systems into a single distributed storage mesh. There is safety in numbers, and the more systems you can add to these sort of distributed storage systems the more reliable they become.

HiveCache is more of a backup solution, but I do know that it is possible to use this with a webDAV front-end for archival storage and other intersting storage possibilities.

Tapes are also slow, expensive, hard to verify, labor intensive, and did I mention slow?

Now, when a hard drive fails, what're you going to do?

Clever solutions spread the risk around multiple hard drives and allow you to recapture some of the idle disk space on desktop system. When a tape drive fails are you even going to know about it? I have had numerous tape drives fail silently, and the only time you know that you weren't getting a good backup was then the time came to restore the data and you found your tapeset to be useless. Disk drives are cheap, and with RAID and distributed-RAID solutions you can use error correction techniques to eliminate the problems of losing multiple drives across the system -- and when any component fails you know about it and can act to eliminate the potential for cascading failures that might actually lead to data loss.

The biggest win from dumping tapes is that you can restore data quickly and eliminate the IT burden by turning most data restore operations into a user self-help situation. Let the user restore their own data and give them the additional advantage of being able to do file versioning and online backup verification. Offline media (like tapes) will continue to have a small role as a solution for catestrophic failures like the building burning down, but when it comes to day to day disaster recovery (and the all-too-frequent "pilot error" events) tape backup's time has come and gone.

Disk-based backup is also much more efficient than tape, allowing you to do all sorts of fun tricks like single-instance storage (e.g. only one copy of word.exe needs to be backed up, the rest of the systems can just point to this copy) and using disk-based backup systems as an intermediate cache (a la Data Domain boxes) is the smarter solution if you can't quite give up your tape habit just yet.

Tapes are cheap and high density.

Oh really? I can buy 1 TB of IDE disk for 1K and drop it into a cheap Linux box that was previously acting as a doorstop. How much does your tape drive cost? And the tapes? And the server to front the tape drives? And the additional networking hardware to provide a fat pipe to this data funnel you have just put into place? Tape is no longer even the cheap solution, it is just the one you are used to....

Ask and you shall receive...

Check out HiveCache for grid-based backups for your enterprise. There are lots of great distributed systems out there that do more than just provide a cheap supercomputer replacement. Some of them can actually save your biz money and eliminate some of the more unpleasant tasks your IT staff has to deal with.

It should also be noted that the FEC libraries that are being passed around now (including the Java library from Swarmcast) are all based upon the Vandermonde FEC library provided by Luigi Rizzo. This seed has sprouted the Java library previously mentioned as well as Python bindings that will appear soon coming from the MNet and HiveCache efforts.

Hivecache is an outgrowth of the Mojo Nation P2P project. Mojo was mainly a file sharing environment; Hivecache is pointed towards business data backup environments (partly because Mojo didn't reach the ...5 Profit!!! stage...)

Hivecache is a P2P distributed backup system that grew out of Mojo Nation. Files are encrypted and shredded into multiple RAID-like pieces, so no individual piece can be used to reconstruct the original data. You don't know what's on there, and you can't find out, because you don't have the information to do it, which provides you some protection as well as providing protection to the people whose data you're storing.

The system you describe already exists. Check out HiveCache for a system that does what you describe and adds nice features like strong encryption to stored data, error-correction to create a distributed RAID across the PCs on the LAN, and efficient storage by only keeping enough copies of redundant files (e.g. word.exe, windows DLLs, etc.) to ensure reliable recovery.

If you want to see what a system like this looks like when it is applied to the proper environment (like across PCs within the enterprise) then check out HiveCache.

I think that people who worry about "putting their files on other people's machines" should go over the docs once more.

Users can backup and restore their own files; user self-help for cases of "pilot error" and random system crashes means that IT does not get the "hey, I accidentally deleted my presentation that I have to give to the board in 2 hours, can you help me out?" call that interrupts whatever they were doing. Doing daily/hourly snapshots to an online storage mesh also means that the backup tape monkey does not need to spend time trying to balance backup runs to fit within various backup windows, you can let the distributed system handle the snapshots of current data and use the tape for weekly/monthly offsite archive.

There are lots of cool and interesting uses for p2p outside of simple content distribution, you just need to look a little harder.

Unfortunately, the legal work we did early on when designing the system only prevented people from suing us (the code creators) and it did so by pushing liability off on to the users. That was the closest that one could really get to safety given the structure of the DMCA. Contrary to the widely held fantasy among decentralized p2p systems, "willful blindness" is not a valid defense against DMCA attacks -- something that I think the upcoming Kazaa et al. trial is eventually going to reveal after all of the appeals and other legal wrangling is worked out.

BTW, the only app that mnet provides is a publish-retrieve shared data system identical to the old mojonation (sans distributed resource management), file sharing is not "one of" the apps for the API, it is the only app.

Check out Hivecache. Their software supposedly uses all your free bits of disk space for online backups. Not exactly what you're after, but a pretty cool idea none-the-less.

I will try to answer as many of the good questions and points of discussion that have been brought up as I can over the next few hours, but I wanted to shoot out a quick overview of what HiveCache is to try to set the story straight here.

First of all, HiveCache is an enterprise backup utility that uses a parasitic peer-to-peer data mesh as its backup media. Simple enough really. The goal of the software is not to replace tape or other offline backup tools, the goal is to serve as an alternative tool for users to make most file restoration requests ("hey, I accidentally deleted my Powerpoint presentation fo the big meeting that starts in 30 minutes...") a user self-help operation rather than something that needs IT assistance. Users restore most files via the p2p mesh and tape/CD-R is only needed for really old stuff or if the building burns down.

The HiveCache distributed online backup system is currently targetted at small to mid-sized enterprises (100-1000 seats) as a way for these companies to increase the ROI on existing IT investment (they already paid for the disk space, so why not use all of it) and to decrease the burden that daily backup and restore operations place upon IT staff. Right now the clients are win32 but agents that serve up disk space to these clients from OS X and Unix hosts are also available. By using good error-correction mechanisms it is possible to maintain five "nines" of reliability for retrieving any particular file even if 25% of the network drops offline. As the backup mesh grows larger reliability keeps increasing while the data storage burden for adding a new node drops (because the level of redundancy among the nodes grows.)

Lastly, the relationship between HiveCache and MojoNation. Basically, there are two branches off of the work HiveCache (nee Evil Geniuses) did on MojoNation, one early branch went on to become the backup product, a later fork pared off some of the non-essential bits (payment system, etc.) and became MNet. The MojoNation public prototype helped to work out the kinks of the data mesh but for the last year and a half of the life of MojoNation most of our internal coding effort was on behalf of this other project which shared some back-end components with the LGPL codebase we were also supporting. For those who complained earlier that the MojoNation user experience sucked I must humbly appologize, we were spending the cycles working on a different UI. Since the MojoNation project went into hibernation on our side, a former MojoNation coder and several other very sharp people have been continuing the MNet project based upon the open codebase (and with a much nicer UI than we ever provided for MojoNation.) I do appologize if the patent and licensing language appears a bit heavy-handed, it was a cut and paste job from some email with legal counsel and will be made clearer this weekend when the site is updated.

ObPlug: We are still seeking a variety of enterprise environments for our upcoming pilot test and in addition to getting to experience the benefits of the HiveCache system for your company you will also be able to purchase the Q4 release version at OEM prices! Sign up now by sending mail to pilot@hivecache.com

Jim McCoy HiveCache, Inc.

Mojo Nation was conceived by Jim McCoy and Doug Barnes in the 90's. At the end of the 90's they hired hackers and lawyers and started implementing.

Their company, Evil Geniuses For A Better Tomorrow, Inc., opened the source code for the basic Mojo Nation node (called a "Mojo Nation Broker") under the LGPL.

During the long economic winter of 2001, Evil Geniuses ran short of money and laid off the hackers (the lawyers had already served their purpose and were gone).

One of the hackers, me, Zooko, and a bunch of open source hackers from around the world who had never been Evil Geniuses employees, forked the LGPL code base and produced Mnet.

Now there is a new commercial company, HiveCache. HiveCache has been founded by Jim McCoy.

BTW, if you try to use Mnet, be prepared for it not to work. Actually the CVS version works a lot better than the old packaged versions. We would really appreciate some people compiling and testing the CVS version (it is very easy to do, at least on Unix).

It would be really good if someone would compile the win32 build. We do have one hacker who builds on win32, but we need more.