Slashdot Mirror
Distributed Data Storage on a LAN?
AgentSmith2 asks: "I have 8 computers at my house on a LAN. I make backups of important files, but not very often. If I could create a virtual RAID by storing data on multiple disks on my network I could protect myself from the most common form on data failure - a disk crash. I am looking for a solution that will let me mount the distributed storage as a shared drive on my Windows and Linux computers. Then when data is written, it is redundantly stored on all the machines that I have designated as my virtual RAID. And if I loose one of the disks that comprise the raid, the image would automatically reconstruct itself when I add a replacement system to the virtual RAID. Basically, I'm looking to emulate the features of hi-end RAIDS, but with multiple PCs instead of multiple disks within a single RAID subsystem. Is there any existing technologies that will let me do this?"
I believe that Windows 2000's Distributed File System allows you to do just this.
Vintage computer games and RPG books available. Email me if you're interested.
The obvious answer for this is nbd, as pointed out in another post -- but I would have concerns about speed with that kind of setup. I'd be interested in hearing reports on that.
But if you don't want to get into nbd, you can tolerate delayed writes to your virtualized disks, and all you want is the network equivalent of RAID level 1, then you could always just set up an rdist script that synchronizes your local data disk with a remote repository (or eight) every so often...
--ZS
-- sigs cause cancer.
It's called the Andrew File System.
http://www.psc.edu/general/filesys/afs/afs.html
There's another alternative with a different name, but I forget what it's called.
Reeses
And since the guy is also using windows-boxes, an NBD-server for windows can be found here:
http://www.vanheusden.com/Loose/nbdsrvr/
This version enables you to also export partitions/disks.
www.vanheusden.com - home of Multitail, HTTPing, CoffeeSaint, EntropyBroker, rsstail, bsod, listener, nagcon, nagi
Intermezzo is designed for this and a bit more - if one of the machines is a laptop you can take it away and work on it, and it'll resync when you get back.
It isn't particularly high-performance, from what I know, and may be more complexity than you need.
I imagine you'll need gigabit ethernet or multiple NICs in bonded mode. Then you have the performance of each individual system to take into account. Especially if one of the systems is heavily used. I would recommend getting one BIG HONKIN' SERVER and putting it in a central location. Give it gigbit and let everything else connect to it at 100. Then, make sure it has a hardware RAID controller. Use SAMBA for the cross platform connectivity you desire, and viola! protected data with redundancy and high speed performance. If you go with remote display (RDP with Windows Terminal Server or X with *nix) then you have an even better appraoch as all the data will exist on the secure RAID box.
I get what you mean though... it's a nice idea, but it would be costly to implement vs. what I suggested above.
When I went to see a presentation on HP's SAN solutions last year, I was very impressed with the ideas they had. One big hardware box with multiple disks that are controlled by the hardware. They are then presented to any systems over a fiber link as any number of drives you wish for any OS. Finally, their "snapshot" ability was pretty impressive. (Also called Business Copy) All they would do is quiesce the data bus, then create a bunch of pointers to the original data. As data is altered on the "copy" (just the pointers, not a real copy), the real data is then copied to the "copy" with changes put in place. I imagein something similar could be accomplished with CVS...
Un-news
see http://drbd.cubit.at/ DRBD is described as RAID1 over a network.
Rsync with a cron script would work too. I think there is a recipe in the linux hacks books to do something like what you are looking for: #292.
http://plan9.bell-labs.com/sys/doc/venti/venti.
Abstract
This paper describes a network storage system, called Venti, intended for archival data. In this system, a unique hash of a block's contents acts as the block identifier for read and write operations. This approach enforces a write-once policy, preventing accidental or malicious destruction of data. In addition, duplicate copies of a block can be coalesced, reducing the consumption of storage and simplifying the implementation of clients. Venti is a building block for constructing a variety of storage applications such as logical backup, physical backup, and snapshot file systems.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
NBD *is* standard Linux kernel. It's built right in: /usr/src/linux-2.4/Documentation/nbd.txt
D _w ork_with_heartbeat
If you're curious about using the enhanced NBD w/ failover and HA, you can read about it at:
http://www.it.uc3m.es/~ptb/nbd/#How_to_make_ENB
This is the way I do it, and although a little clunky, it allows me to keep remote backups of certain directories one three different servers.
/home/user/.ssh/authorized_keys file.
.last-sync | grep '.' 1>/dev/null 2>/dev/null .last-sync
First, setup ssh to use pubkey authentication instead of interactive password. You can read the man pages for details but it basically boils down to running keygen on the trusted source:
ssh-keygen -b 2048 -t dsa -f ~/.ssh/identity
Then copy|append the newly created ~/.ssh/identity.pub to the remote hosts into their
Now you can run rsync with ssh as the transport (instead of rsh) by exporting:
export RSYNC_RSH=ssh or also passing --rsh=ssh on the command line.
So to sync directories you could use a find command to update regularly:
while true; do
find . -follow -cnewer
if (( $? == 0 )) ; then
rsync -rz --delete . destination:/some/path/
touch
fi
sleep 60
done
Obviously this is pretty hackish and could be improved. But the point is that with ssh and rsync you could do automatic mirroring of specific filesystems or directories to remote locations securely.
Just to clarify what this guy is saying:
1) Make all your machines NBD servers. NBD for Linux, NBD for Windows. NBD stands for "network block device" and allows a client to use a server's block device.
2) Set up a master client/server (using Linux or something else with a decent software RAID stack). This machine will be the only NBD *client*, and it will use all the NBD block devices exported by the rest of your network.
3) On the master set up in 2), create a Linux MD RAID array overtop all the NBD devices that are available.
4) Create a filesystem on the brand-spanking-new multi-machine RAID array.
5) Export it back to the other machines via Samba or NFS or AFS or what have you.
Why does only one machine (the "master server") access the NBD devices, you ask? Because for a given block device, there can only be one client accessing it safely. Thus, if you want to make the RAID array available to anything other than the machine which is *running* the array off the NBD devices, you need to use something which allows concurrent access; something like NFS, Samba, or AFS.
Hope that clears it up a bit.
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
Instead of xcopy, try RoboCopy, included in the windows NT/2k/xp/2k3 resource kit available here. It gives you almost as much control as rsync, including directory synchronization, touch control, ageing, network failure support, and others. I use this at work to move around copies of live production data to backup servers located offsite via vpn without any issues. More information on syntax can be found here.
And now for something completely different...a man with three buttocks.
http://www.parl.clemson.edu/pvfs/
n dF ileSystems.html
"The goal of the Parallel Virtual File System (PVFS) Project is to explore the design, implementation, and uses of parallel I/O. PVFS serves as both a platform for parallel I/O research as well as a production file system for the cluster computing community. PVFS is currently targeted at clusters of workstations, or Beowulfs."
"In order to provide high-performance access to data stored on the file system by many clients, PVFS spreads data out across multiple cluster nodes, which we call I/O nodes. By spreading data across multiple I/O nodes, applications have multiple paths to data through the network and multiple disks on which data is stored. This eliminates single bottlenecks in the I/O path and thus increases the total potential bandwidth for multiple clients, or aggregate bandwidth."
Or there are many others to chose from, google for clustered filesystems:
http://www.yolinux.com/TUTORIALS/LinuxClustersA
We have looked at various distributed filesystems for use in a clustered setup of webservers. We wanted to remove the single point of failure from a central NFS server - Intermezzo was one of the filesystems we had a look at.
The idea behind Intermezzo is fairly simple and the documentation is good. The Intermezzo system looked like an ideal solution for our setup (Coda and OpenAFS are far to complex for use in a distributed filesystem on a closed internal net).
We tested the system but sadly it's not really production stable and I can't advise that you use it.
If you are looking for a SAFE solution then Intermezzo is not for you - you will just end up with garbled data, deadlocks and tons of wasted time ...
My 2 cents.
0) Mirroring (RAID 1) takes double the disk space; but you could use RAID 5 instead. A 4 disk RAID 5 would take 4/3 as much disk space as you get to use.
1) You could make a partition that is 10% of your disk, make another identical one on another disk, and mirror those. Then put your 10% critical data in there.
2) Do what I do: set up a RAID server, and keep all critical data on that. This is good if you have a home network with multiple computers. It also makes data sharing easy among the computers.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
As I always chime in at this point:
Use rdiff-backup!
http://rdiff-backup.stanford.edu/
Configurable, secure, distributed, versioning incremental backups.
It's not a replacement for RAID, but is good for nightly inter-machine backups.
There's also a related project where the far-end repository is encrypted, so you can have it on any public server without fear of having your data read by the wrong people.
Very cool. It's saved my ass a few times.
Eloi, Eloi, lema sabachtani?
www.fogbound.net