Remote Data Access Solutions?

magoldfish asks: "Our company has several terabytes of data, typically chunked into 2-6 GB files, that we need to share among users at sites around the US. Our users range in technical skill from novice to guru, data access needs to be secure, and we require client access from multiple platforms (Linux, Mac, and Windows). These files will likely be accessed infrequently, and cached on local hard drives by our power users. We've been running WebDAV, primarily to accommodate non-savvy users and guarantee access from firewalled environments, but the users are really complaining about download speed — maybe 50 KB/sec serving from a Mac. Any suggestions for better alternatives, or comments on the pros and cons of alternative access techniques?"

Comments

How will the data be used?

[kperrier]

Without knowing how the data will be used it will be hard for anyone to supply you with some recommendations.

One word:

[Pig Hogger]

rsync.

(As always, Google is your friend).

Profiling?

[barnetda]

I'm no network / data access guru, but this seems like a typical case of profile first, optimize later.

The idea is simple. Don't just go in and change stuff, first measure the pieces under typical load. Look where the bottle-neck is, address it, and move to the next bottle-neck. Repeat as often as needed.

Are you disk I/O bound? Buy faster disk / better controllers / spread the load over more machines / .....

Are you CPU bound? Is the CPU on your server spending so much time with I/O requests, that it has no cycles available to address additional requests? Buy more / faster / better CPUs.

Are you network bound? Which piece of the network is the hold-up? Your switch? Get a better / faster one. Your ISP? Get a fatter pipe.

Have you optimized all of these? What about setting up remote servers that are updated hourly/daily/weekly/whatever so the machine is close to the user network-wise for faster download speeds.

Some of the above adds complexity. Are you equipped to handle that complexity? Can you become equipped to handle it? If not, re-consider your options.

Hope this helps.

Cheers,
Dave

Remote Desktop Solution(s)

[Gunfighter]

If possible, write the app to run centrally and then use a remote desktop solution like LTSP, Citrix, or Windows Terminal Services to feed access to the app out to clients.