Slashdot Mirror
Distributed Data Storage on a LAN?
AgentSmith2 asks: "I have 8 computers at my house on a LAN. I make backups of important files, but not very often. If I could create a virtual RAID by storing data on multiple disks on my network I could protect myself from the most common form on data failure - a disk crash. I am looking for a solution that will let me mount the distributed storage as a shared drive on my Windows and Linux computers. Then when data is written, it is redundantly stored on all the machines that I have designated as my virtual RAID. And if I loose one of the disks that comprise the raid, the image would automatically reconstruct itself when I add a replacement system to the virtual RAID. Basically, I'm looking to emulate the features of hi-end RAIDS, but with multiple PCs instead of multiple disks within a single RAID subsystem. Is there any existing technologies that will let me do this?"
http://nbd.sourceforge.net/
/dev/nd0, it will send a request to the server via TCP, which will reply with the data requested. This can be used for stations with low disk space (or even diskless - if you boot from floppy) to borrow disk space from other computers. Unlike NFS, it is possible to put any file system on it. But (also unlike NFS), if someone has mounted NBD read/write, you must assure that no one else will have it mounted.
"Network Block Device (TCP version)
What is it: With this thing compiled into your kernel, Linux can use a remote server as one of its block devices. Every time the client computer wants to read
Limitations:It is impossible to use NBD as root file system, as an user-land program is required to start (but you could get away with initrd; I never tried that). (Patches to change this are welcome.) It also allows you to run read-only block-device in user-land (making server and client physically the same computer, communicating using loopback). Please notice that read-write nbd with client and server on the same machine is bad idea: expect deadlock within seconds (this may vary between kernel versions, maybe on one sunny day it will be even safe?). More generally, it is bad idea to create loop in 'rw mounts graph'. I.e., if machineA is using device from machineB readwrite, it is bad idea to use device on machineB from machineA.
Read-write nbd with client and server on some machine has rather fundamental problem: when system is short of memory, it tries to write back dirty page. So nbd client asks nbd server to write back data, but as nbd-server is userland process, it may require memory to fullfill the request. That way lies the deadlock.
Current state: It currently works. Network block device seems to be pretty stable. I originaly thought that it is impossible to swap over TCP. It turned out not to be true - swapping over TCP now works and seems to be deadlock-free.
If you want swapping to work, first make nbd working. (You'll have to mkswap on server; mkswap tries to fsync which will fail.) Now, you have version which mostly works. Ask me for kreclaimd if you see deadlocks.
Network block device has been included into standard (Linus') kernel tree in 2.1.101.
I've successfully ran raid5 and md over nbd. (Pretty recent version is required to do so, however.) "
I'd argue the point that the most common form of data loss is a crashed hard disk.
/. user knows what they're doing with their data far more than my average user does and is less likely to cause self-inflicted damage.
In my 14 years as a Network Administrator I think I've restored backups due to failed hard disks about twice (RAID catches the rest).
But I restore data accidentally deleted or changed by a user at least weekly! A distributed storage system won't help you there.
However, I will grant that the average
I hope you're looking at some fast lines to put between those boxen. Even at 100Mb/sec, doing RAID across a LAN could get slow.
I'm against picketing, but I don't know how to show it.
I have often wanted the same thing, kind of like RAID on files, call it RARF (Redundant Array of Remote Files). I was thinking along the line of a device driver that presents an ATA/IDE interface to the file system on one side and passes the requests to multiple copies of virtual disks. The virtual disks would be like VMWare disks, and potentially each on a different machine/location. Each virtual disk could even be encrypted differently.
This would be really useful for SOHO type places to allow me to have a hot offsite backup at multiple friends (and vise versa).
Hmmmm, what happens if your house catches fire ?
8 copies of the same document all nicely toasted!
Instead of trying to implement a shoestring SAN, go the simple route: throw up a Linux box running Samba for your "backup server;" it doesn't need much horsepower, just fairly fast drives and a network connection. Then schedule copies of your documents and home directories (using a cron-type tool on Linux and XCOPY called by the Task Scheduler on Windows, you should be able to hack something together that copies only changed files) every night at midnight, or some other time when you aren't using your computers. Although you might lose a bit of work if the system goes down, you won't ever lose more than 24 hours' worth.
If you have more money to blow, then I would suggest that you invest in an honest-to-dog hardware RAID card and some good drives and put them into a server, then do everything across the network (put the /home tree and My Documents folders on the server). You can of course mount the /home directory in Linux via NFS or smbmount, and Group Policy in Windows 2K/XP will allow you to change the location of the My Documents folder to whatever you choose. You might be able to do the same via the System Policy Editor on 9x; it's been a while and I can't find the information after a brief Google.
To sum up:
That's it. I'm no longer part of Team Sanity.
Really. If you're on a 100-megabit LAN, that gives you a max of about 10 megaBYTES per second. So, if you have to transmit information to two other computers for every disk write, you're effectively limitting yourself to a maximum of about 5 megabytes/second disk transfer. And that's under GOOD situations. If you're doing random I/O, where the latency will be the determining factor, then take the latency of the hard drives, add in the latency of the networking, and the latency of the software layers, and you're looking at some pretty abysmal performance.
Using rsync in a cron job will solve your backup problems. In fact, your script can use rsync to do the synchronization, and tar/gzip to archive the backup - giving you "point in time" snapshots for when someone says "I deleted this file 4 days ago, can you get it back?"
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
In my experience, it's one of those "it would be a wonderful thing if it worked."
I've been using it for years. I've found nothing that works better. I've got ``clients'' that are IRIX, NetBSD, Solaris, SunOS 4, NetBSD, MacOS X and FreeBSD and I use it to serve my web root, home directories, various applications (my mail server etc...) I can't imagine using something else.
It requires it's own partition for each mount of it; you can't just share disks you've already got.
This is very misleading. A file server has to have a dedicated partition. Clients need nothing but OpenAFS or similar installed. Mount points are global and management is distributed. Thinking that AFS is anything like NFS would certainly lead to a bad experience. It solves many, many problems with NFS.
The servers time has to be matched exactly, so it's also best if you've got an NTP server running and clients on all the machines.
And your AFS server and client comes with them. I can't imagine what the problem would be with having times matched, anyway. I've gone through the horrors of tracking down log entries from systems that didn't have time synchronized. I don't want to do that again.
It's also about ten times slower than Samba (which you might use instead to share with Windows machines), and it chokes when you try to move/copy/delete large files.
Slower at what? Access times? Add another server, it's not like you have to tell the clients. Write times? I don't know about that, I wouldn't want to run a database off the thing, but that's not what it's for. I have no idea what you're talking about regarding it choking on large files. I haven't seen that.
I tried it for a month before it completely corrupted it's own partition and I switched back to NFS and Samba.
How exactly did it corrupt its own partition? I've never seen such a thing. Perhaps you did something you were not supposed to do (like anything in its own partition).
I can't wait for the day when these problems are but a memory and such a system works flawlessly.
There have been some *very* large AFS installations for years (MIT, CMU, etc...). I wouldn't think that would be the case if such problems were common.
-- The world is watching America, and America is watching TV.
It seems to be a great problem solver for what you're trying to do. First off, on initial start it only connects to computers it knows, or downloads info about a couple of nodes from the main website, but if you were to export your noderef and import it into all of your other systems instead of the default noderefs, then you could have a distributed storage network set up among all of your computers.
Granted, you'd have to have a bit more storage dedicated than you'll be storing, but if you want every file to have a decent backup, then that's one of the prices you'll have to pay. Also, it's self cleaning when it comes to backups, because it automatically pushes out the old, less requested files in favor of the newer, more requested files.
Another solution, should your systems be using Linux is maybe something like GNUnet, which is built upon the sharing of files in both a distributed and an anonymous manner.
Listen, Sonny Jim. You'll not be getting any mod points from us by bringing up the last contender to Windows, which failed miserably. We're feeling good about ourselves right now, and we don't need bringing down.
Get your own free personal location tracker
I take it you've thought about speed issues? RAID over a 100mbit link doesn't sound like great fun - leastways I wouldn't put my swap on such a drive :)
Gigabit might work though.
"'I pass the test,' she said. 'I will diminish, and go into the West, and remain Galadriel.'"
- JRR Tolkien.
I understand many of the comments here which say "put in a big honkin' server and hardware RAID". That would be a better solution from a purely 'let's serve files and protect data' standpoint if you can accomodate a single, large server and want the best performance.
However, I see a use for a network LAN storage system. Every machine these days comes with a 72G drive or larger installed locally, yet we are trained as IT personnel to say 'don't store anything locally, it's not secure or safe, put it on one of our nice big honkin' servers'. Unfortunately, those big servers cost alot of money, often require specific admins (eg SAN experts to deal with the management software, dividing up LUNs, etc), and may involve alot of red tape to justify additional storage allocation for your project.
What to do with all that local disk space that, if unused as most centralized IT would rather have you do it, would be a vast untapped storage resource?
The concerns regarding latency are well understood, but this might not be a factor if this LAN storage array was used for 'archive' storage where real-time high speed access isn't the driving factor. A RAID 5 system would be far too fragile, as if two nodes were offline/rebooting the entire network storage LAN would be unavailable. You'd need to have more redundancy than that.
I could see an interesting application using multiple nodes each contributing disk space to a LAN archive storage array which would be 'written to' and retrieved with similar expectations as writing to a tape drive. The bonus would be that you could work on files in realtime over such a network, just quite slowly (many vendors used to offer archive file systems which worked this way using tape or optical drives as the storage medium - AMASS was one such vendor).
The lustre project (www.lustre.org) is supposedly going to be the end all/be all of distributed parallel file systems, but I believe it is still fairly unstable and not ready for production use. In the meanwhile, the best one out there is PVFS(www.parl.clemson.edu/pvfs/). Fat chance trying to find Windows clients, but you can always re-export it with Samba.
What if you reboot one of the NBD servers? While you'll still have access to the data since it's a raid, I would well imagine that you would have to rebuild the entire "disk" once it comes back online.
Assuming a Raid5 with three nodes, and two go down not at the same moment, will all your data be lost?
I would think very carefully about these issues before putting all your valuable data on it. RAID isn't really designed for frequently unreliable connections like this. It's meant to prevent data loss if a hard drive crashes, which should be a fairly uncommon thing within a single system.
- It's not the Macs I hate. It's Digg users. -
You'll be wanting a distributed cluster filesystem. There are several available, with their pros and cons. They are also all aimed at enterprise / HPTC installations. For home use you'll be better off with a set of RAID disks.
GPFS from IBM. This is free for academic use, but you pay for commercial use. Linux or AIX only.
GFS from sistina. Commercial offering. Linux only.
Lustre. This is beta quality code, but is freely available. It might work wonderfully, or it might eat your files.
(open)AFS. Works, but has limitations. It does not support large files and clients aren't available for all OSes.
I think you just described the Freenet project