Sharing a Subset of Data Between 2 Sites?

eldrich asks: "We have two labs: a main lab (lab 1) has 1.2Tb of on-line data storage -- two machines with 600Gb RAID-5s hung off of them. These happily service about 30 Linux machines via NFS over fast ethernet. There are 5-6 WinXP machines that connect via SMB and Samba. The lab is on a private network with a single firewall between it and the world, and we use LDAP for practically everything (hostname, usernames, password, autofs, etc). The students' lab (lab 2) is 40 miles away, with 8 workstations and 2 WinXP machines. This lab also has a small RAID-5 Linux server with 180GB space which serves via NFS and Samba. Sometimes we have people from lab 2 at lab 1 and while they are at the main lab, they need their files. What I want to do is make lab 2's 180GB RAID a subset cache of the 1.2Tb one in lab 1. This puts everyone's main storage at lab 1 (which is backed up weekly) but a local copy can be cached on the lab 2 raid system. This gives the students a local copy for fast access, but all the safety of the backups made from our system. Does anyone know of a filesystem or programs that can help with this?"

"Some people spend 95% of their time in lab 2, so that is their 'home' server, but when they come to lab 1 for a week's stay or so, they scp/rsync their files to the lab 1 server, and at the end of the week push the changes back to lab 2. When people login to a workstation, they usually remain logged in for days at a time and xlock the screen. [If we can get this caching system working], it would mean that people moving between the labs would not need to copy files around since there would always be a 'local' copy.

The network between the labs is not fast enough for direct automounting of lab 1's server on the lab 2 workstations, especially since some files can be over 300Mb in size. We have a VPN (via freeswan) between the different labs, so all data transmitted is encrypted. Also, because lab 2 has 1/6 the capacity of lab 1's RAID it needs to be cached copies of in-use or probable in-use data only.

Crontab entries set for night copies are not useful because people often appear from both places on any given day.

The 3 servers currently run 2.4.18 with XFS so any solution should be compatible with XFS but at a real push we could consider changing the filesystem to another one."

Comments

Re:Here's a thought.

[borgboy]

Now, about that job.....how's $50M in stock options and $2M/year in salary? I'll even let you run Linux at work (in a Virtual PC, of course)
All you have to do...is nothing!

--Bill