Slashdot Mirror

← Back to Stories (view on slashdot.org)

Sharing a Subset of Data Between 2 Sites?

Posted by Cliff on from the keeping-your-files-with-you dept.

eldrich asks: "We have two labs: a main lab (lab 1) has 1.2Tb of on-line data storage -- two machines with 600Gb RAID-5s hung off of them. These happily service about 30 Linux machines via NFS over fast ethernet. There are 5-6 WinXP machines that connect via SMB and Samba. The lab is on a private network with a single firewall between it and the world, and we use LDAP for practically everything (hostname, usernames, password, autofs, etc). The students' lab (lab 2) is 40 miles away, with 8 workstations and 2 WinXP machines. This lab also has a small RAID-5 Linux server with 180GB space which serves via NFS and Samba. Sometimes we have people from lab 2 at lab 1 and while they are at the main lab, they need their files. What I want to do is make lab 2's 180GB RAID a subset cache of the 1.2Tb one in lab 1. This puts everyone's main storage at lab 1 (which is backed up weekly) but a local copy can be cached on the lab 2 raid system. This gives the students a local copy for fast access, but all the safety of the backups made from our system. Does anyone know of a filesystem or programs that can help with this?"

"Some people spend 95% of their time in lab 2, so that is their 'home' server, but when they come to lab 1 for a week's stay or so, they scp/rsync their files to the lab 1 server, and at the end of the week push the changes back to lab 2. When people login to a workstation, they usually remain logged in for days at a time and xlock the screen. [If we can get this caching system working], it would mean that people moving between the labs would not need to copy files around since there would always be a 'local' copy.

The network between the labs is not fast enough for direct automounting of lab 1's server on the lab 2 workstations, especially since some files can be over 300Mb in size. We have a VPN (via freeswan) between the different labs, so all data transmitted is encrypted. Also, because lab 2 has 1/6 the capacity of lab 1's RAID it needs to be cached copies of in-use or probable in-use data only.

Crontab entries set for night copies are not useful because people often appear from both places on any given day.

The 3 servers currently run 2.4.18 with XFS so any solution should be compatible with XFS but at a real push we could consider changing the filesystem to another one."

4 of 23 comments (clear)

  1. CODA and AFS by Lomby · · Score: 3, Informative

    If you have a very reliable connection you may want to go for AFS

    In case the connection is not realiable (or not fast enough), you may want to try CODA which is a distributed filesystem which supports disconnected operations. Beware: AFS is a mature project, while CODA may still be a work-in-progress.

  2. Keep it simple use CVS or rsync by Bob+Bitchen · · Score: 3, Informative

    Don't over-engineer, keep it simple use CVS or rsync.

    --
    http://tinyurl.com/3t236
  3. unison by martin · · Score: 3, Informative

    http://www.cis.upenn.edu/~bcpierce/unison/

    works very well and is designed for this kind of thing.

    BTW - weekly backups!!!! daily surely?

  4. Intermezzo might be a solution by narensankar · · Score: 3, Informative
    http://www.inter-mezzo.org/

    Similar to afs and coda suggested before, but with local caching to allow much higher performance. Also works in disconnected mode.