Slashdot Mirror

I used to dream of setting up an office network environment based on Linux and FOSS. Only, there there was one thing missing: a proper file system. That's why I think OpenAFS -- the distributed file system -- deserves more attention.

Most *nix fans use either NFS, which is simple, but scales badly and lacks encryption, or Samba, which was designed to support Windows clients. OpenAFS, on the other hand, offers file sharing and replicated read-only content distribution, provides location independence, scalability, security, and transparent migration capabilities. Client software includes support for UNIX, Linux, MacOS X, and Windows. The code base is very stable and it has an active development and support community.

E.g. http://www.openafs.org/, http://www.gluster.org/

A good system would let me switch on a new system/pc and it would automatically share all it's resources (storage, ram, cpu, I/O) with a defined cluster of other systems/PCs.

It's not entirely hopeless though: things like AFS, various distributed shared memory systems with a good API, task and process migration and so on have been around for quite some time.

Sorry, helps to provide the site:

http://www.openafs.org/

You may be interested to know that AFS has implemented a variant of this feature. The conceit is that filenames can contain a magic string @sys, which gets substituted with the "sysname" of a particular system. This means if someone publishing software over AFS wants to have multi-platform support, they merely have to setup a directory divided by sysname and have compiled versions of the software for each system type they wish to support.

Coda, and before that, AFS. Oh, and Lustre.

It's not a new idea. The only real difference here is that it's associated with BitTorrent and The Pirate Bay, and is designed to handle a whole set of problems you won't have, like untrusted machines communicating over the Internet, and how to compensate people for using their hard drive to store your stuff.

There are all kinds of free AFS for Windows:
http://www.openafs.org/

http://www.openafs.org/

3G is pretty hot shit-- I've been using one of Sprint's cards for the last couple of months. OK, maybe the US is ass-backward compared to the rest of the mobile world, but it's still pretty damn cool to have an SSH session open while I'm on the train. I'm presently looking into OpenAFS as that should deal with occasional dead spots better than SSH, but in general, the experience is much better than I was expecting. Not quite up to Hulu-watching speeds, but it's definitely making me depend more on my laptop-- and I pretty much hate laptops.

That's a long way of saying that the 3G is worth it even if you have to deal with the outrageous SMS pricing.

OpenAFS is a distributed file system. It seems to fit your bill. No personal experience, so don't know how well it actually works.

For a filesystem, I would recommend openafs. For printing, I recommend setting up CUPS servers. AFS lets you have distributed servers that are centrally maintained. AFS is location agnostic. The filesystem is split into volumes, which can be located on any server and seamlessly moved between servers without needing to change the file path. Check it out at http://www.openafs.org/

It also has manual read-only replication so that you can have a local read-only copy of frequently accessed files.

Are the magic words, but please do prepare your brain for a roller-coaster ride.

OpenAFS
and
Kerberos

www.openafs.org

Its not only a nice idea, it works fine too.

Windows support?
http://www.openafs.org/windows.html

OpenAFS for Windows 1.5.30 (1.5.3001.0) using MIT Kerberos for Windows 3.2.2 for authentication.

SFTP is a crude hack?

May I suggest AFS? It is somewhat complicated to set up, but it is very good and used on many large and very large sites.

Which is...

Not unlike.. http://www.openafs.org/

Which is the open version of the Andrew File System, about which wikipedia says..

"Additionally, a variant of AFS, the Distributed File System (DFS) was adopted by the Open Software Foundation in 1989 as part of their Distributed computing environment.'

I suggest that you take a look at AFS, it might have the featureset that you want. It may not be the easiest system to setup, but it is powerful and feature-rich.

Personally, I'd like to see methods like OpenAFS with a RAID/SAN data store. A great benefit of AFS is that it's ideal to work over a large IP network. Every night issue a update for all the nodes, a little like rsync I suppose in this respect, but it's ideal for a large infrastructure. Of course things like MD5 sums should be used on the files, perhaps split the large files with RAR or something, maybe use a .PAR file also. You know.. I think the pirate world has this sort of thing sorted already. Why don't the media giants take a leaf out of their book and see how others in the volatile world cope? Maybe they could use newsgroups for data retention?

I have set up and supported remote sites and home based telecommuting. Listen to my advice, listen very carefully and save your sanity.

If your organization is large enough then it is likely that you will have a few older desktop PCs that have been or are due for replacement during an upgrade cycle. PCs that are inadequate for Microsoft XP and Office2003 are more than powerful enough for many current versions of Linux, especially for the role of server. Also second hand PCs with the required specifications are very cheaply acquired.

1) Find an older PC, at least a PII 300 with 256 MB memory, to set up as a headless ( no display or keyboard ) server and firewall. A simple web based interface ( or even an external hardware push button ) can be used by the local users to start/stop the server and internet connection. All other maintenance should be handled remotely via ssh, webmin and VNC.
2) Install a second NIC or connect the modem directly to the server. Connection to the Internet should be through the server and connection to the Office should be through a VPN on the server. Use a dynamic IP service for each site so you can remotely log on to the local server via ssh.
3) Install a new IDE hard drive in a 3.5" removable rack and tray. The drive should be than big enough for the operating system (Linux of course) and copies of some of the local desktop partitions. A telecommuter can shut down the server and bring in the drive during the day to resync and repair.
4) Install a DHCP demon on the local server to allocate local IP addresses, DNS and gateway settings. If the desktops are network boot capable then install TFTP to remotely boot and use Knoppix via PXE and the network. If the desktop OS is constantly crashing, or is infected by malware, the user can select PXE/network boot via the BIOS, and boot into Knoppix. The user can then be instructed over the phone to enable the ssh server to allow remote scan,repair and reimaging of the desktop partitions. The user can use the Knoppix desktop to continue working with full access to files while the the remote administrator fixes/reimages the drive in the background.( Consider hiring someone who knows how to customise Knoppix or another live Linux system for your setup )
5) Partition the desktops with as small as required C: partition ( or in the case of Linux the root partition ) for software. When software is install, use dd and netcat via live Knoppix to copy/clone a snapshot of the partition to the server. You can allocate the remaining free space as a persistent partition where documents are stored.
6) Install and enable remote VNC service on all the platforms, but only allow incoming connections from the local server ( which is redirected over a SSH tunnel ).
7) For local backup, create share directories on the desktop accessible by the server. On the local server create loopback encrypted file systems, unmount and copy the images to the desktops shares in chunks, using redundancy if enough space is available on the desktops. Checksum ( MD5 is enough ) each piece.
8) If the network load to the Office is taking up all the available internet bandwidth or the connection is just too slow then install proxy servers on the local server. You can also consider using a distributed filesystem ( OpenAFS is still the best ) wi

Agreed -- try OpenAFS. More complex, but scales well.

There is already a distributed file system providing location independence, scalability, security. It is called AFS.
http://www.openafs.org/

What does the rest of Slashdot think? Is OpenAFS anything like GoogleFS?

A very good, free, cross platform network file system has already been written and is in use by (at very least) large universities. It's called AFS and the free version is at http://www.openafs.org/ .

Have a look at this article: http://www.linuxplanet.com/linuxplanet/reports/436 1/1/ then choose amongst the more mature projects: Coda http://coda.cs.cmu.edu/ and OpenAFS http://www.openafs.org/. Intermezzo looked promising but hasn't been updated in a long while so it's probably dead.

Hope this helps.

On that note, see if AFS is workable. It's not perfect*, but it does have good clients for Windows, Mac, and *nix, the ability to work from anywhere on the Internet, decent speeds on the local network, a fancy ACL system, and some amount of encryption.

*I've had problems recently when putting my laptop to sleep on one subnet, waking it up in another subnet (new IP), and trying to save a file that's already open.... but that's admittedly not a common situation. It works as well as a slightly-laggy local disk otherwise.

WebDAV is also a good option for Windows usage, although it's not as secure or customizable (AFS on Windows works like a mapped network drive).

I can set up Samba, etc. on just about any box. What defies me is setting up OpenAFS. How about a server that supports OpenAFS or Coda?

There's more to be concerned about than throughput. For example, there's shared file access semantics (such as whether the filesystem allows concurrent reads and/or writes, and how strong the cache consistency is, etc.). I'd be willing to bet that Samba doesn't allow more than one computer to have the same file open at a time, for one thing, while some other distributed filesystem would.

I'm looking into running AFS on my home network, because it's got support for synchronization with intermittantly-disconnected nodes (like my laptop).

Distributed filesystems, for one thing (see OpenAFS). We use that here (well, it was developed here) and it is used at several universities. I have used AFS over wireless, and it is really really bad, but connecting from our network (Andrew) in Pittsburgh to MIT's (Athena) is fairly snappy. It is also nice for software transfers and such from mirrors at different universities. And there is no p0rn in AFS.

Why not give OpenAFS from http://www.openafs.org/ a try? It has its own permissions model, and (if you choose to have it so) is completely Kerberos-5 secured. Local root means literally nothing to AFS. It may be a bit beyond your needs, but in terms of scalability and security it beats NFS any day...

Google's job won't be to create this, it already exists (http://www.openafs.org/). They will make it fast by meticulously mirroring 100,000,000TB of the worlds data to their innocuous looking cabinent/cellserver on your street corner, and your mom's streetcorner, and GW's streetcorner, etc... and pouring resources into integrating it better.

1) Install a second NIC or connect the modem directly to the server. Connection to the Internet should be though the server and connection to the Office should be though a VPN on the server.
2) Install a new IDE Hard drive in a 3.5" removable rack and tray. The drive should be than big enough for the operating system (Linux of course) and copies of some of the local desktop partitions. A telecommuter can shut down the server and bring in the HD during the day to resync and repair.
3) Install DHCP demon to allocate local IP addresses, DNS and gateway settings. If the desktops are network boot capable then install TFTP to remotely boot KNOPPIX via PXE. IF the desktop OS is constantly crashing, the user can select PXE boot, network KNOPPIX. The user can then be instructed over the phone to enable ssh server to allow remote repair and reimaging of the desktop partitions from copies on the local server.
4) Partition the desktops with as small as required C: ( or in the case of Linux the root ) partition for software. When software is install, use dd and netcat via live KNOPPIX to copy a snapshot of the partition to the server. You can allocate the remaining free space as a persistant partition where documents are stored. ( Consider hireing someone who knows how to customise Knoppix for your setup.)
5) Install/Enable VNC on all the platforms, but only allow incoming connections from the local server ( which is redirected over a SSH tunnel ).
6) For local backup, create share directories on the desktop accessable by the server. On the local server create loopback encrypted file systems, unmount and copy the images to the desktops shares in chunks, using redundantcy if enough space is available on the desktops. Checksum ( MD5 is enough ) each piece.
7) If the network load to the Office is takeing up all the available internet bandwidth or the connection is just too slow then install proxy servers on the local server and consider using a distributed filesystem ( OpenAFS is still the best ) .
8) If phone charges are eating into the budget, and the internet connection is good enough, then install Asterisk on the local server ( upgrade the server to a Celron 800Mhz or better ) and a card with enough FXS ports for each local user. Don't bother with software based phones/headsets. The phone will work when the desktop does not.
9) Set up a Linux server at the Office that operates as a thin client application server. Allow remote access though both FreeNX and VNC. Create login accounts and logins that operate as virtual meeting rooms, with multiple users logging in via VNC. Use VNCserver with a screen size of around 1000x600, that will operate via a VNC viewer on any 1024x768 desktop. Use phone based conference calling for voice -- it's a lot less hassle for the users
10) Add the ususal list of cross platform applications: Firefox, Thunderbird, Gaim, OpenOffice etc.

Do the open ten step and save yourself and your santity from all those hours driving from site to site.

http://www.openafs.org/

Check out AFS.

Perhaps a better link is to the OpenAFS (Open Andrew File System) implemenation: another IBM contribution to Open source. It is continuing to make available current releases. They're working right now on a new stable release.

The link is http://www.openafs.org/

Steve

You might check out OpenAFS. I'm not sure it meets all your requirements, though.

I don't see how GFS can scale as well as something like OpenAFS. With AFS, you get an entire infrastructure. I wish more people would be investing time and effort into improving filesystems like AFS, where all systems can share a common namespace without requiring the availability of a SAN. The two have slightly different uses, but it'd still be nice to see more force behind AFS now that it is opensourced.

AFS outages are twice as fun.

You can try openafs its mounts a filesystem for you as a network drive and its open source. Its a distributed filesystem product, pioneered at Carnegie Mellon University, this is what we use at the University of Alberta for students, well this and ssh. http://www.openafs.org/

Compared to network file systems. Slow, clunky, inconvenient. In a university, any reason AFS won't do the trick? It can even run encrypted if you don't trust the network you're running over.

How about OpenAFS ? It is sort of like NFS on steroids, with redundancy, scaling, cacheing, Kerberos-based security ... I've just started looking at it myself, but it seems pretty slick.

I'm one of the two people here at UMBC who run the core servers for the campus.

We use AFS here for everyone's home directory, mail spool, web space, and other things. To maintain this, we currently have about 6 servers with direct-attached storage serving everyone's AFS home directory volumes. These servers are a mix of Dell and Sun gear running Linux and Solaris. Both platforms have run well over the years, but each server's direct-attached SCSI storage is limitting and, well, aging.

So we can better use our storage and improve things for everyone in general, I'm in the process of rolling out a fiber channel SAN with new servers and RAID arrays to replace what's currently running. The new server gear we chose? Sun's V20z Opteron server running Solaris 10 . Linux is right out.

Why no more Linux, or rather, why Solaris? A few reasons. Solaris's storage management is TONS easier to deal with and do interesting things with than what is available in Linux. Namely, we've found and have been fustrated by Linux's software RAID. Yeah, it works... but that's about it. Weee look, I can make a mirror! Solaris's SVM (aka DiskSuite) is no VxVM, but it does allow us to do things such as disk sets to share between hosts and monitor our metadevices in detail. Linux's raidutils on the other hand are poorly documented and toublesome (usage options don't match reality, etc)

Another aspect on Linux vs. Solaris in mass storage is (as far as I know) a lack of multi-pathing in Linux. Multi-pathing is a no-brainer especially in the context of Fiber Channel networks and Solaris's MPxIO is in-built and works quite well.

But I'm just poo-pooing Linux here on this specific point. We offer Linux workstations in every one of our computing labs. Linux replaced SGI/IRIX workstations there many moons ago and work well for that purpose. Linux servers also are used for our general shell login servers. But on the backend, where we need reliable features, consistency, and heavy-lifting... we're enthralled with Sun x86 servers and Solaris 10. The V20z Opteron hardware actually is cheaper (for us) than a Dell 2650 and offers a ton more features all-alround.

There is an irony, though. The service processor on the Sun V20zs run Linux. Ah well ;)

"we represent as open-source friendly a commercial organization as you are *ever* going to see"

"Unlike the Marine corp, the open source community is more than willing to ignore their bad apples as 'not my problem' (the Marine corp punishes the group for the behavior of the bad apples, pretty soon there are no bad apples)."

Larry suggested, "if Linus and Andrew and the others moved elsewhere, we'd glady comp them licenses", referring to their current employment with OSDL.

Ever heard of Active Directory? Or seen MS's educational pricing? Both of those are much more compelling than you might think.

If you want off the shelf, then Novell's NDS or eDirectory is the way to go.

If you have some system administration background or really want Free Software, then Kerberos or Kerberos + LDAP is the way to go. It's not the bear people make it out to be.

For file sharing there's Samba or OpenAFS

Do an incremental rsync of their home directories everynight and if something ever goes wrong just delete their home and replace it with a good copy.

One way to aproach this situation is to put a large multi-user box in each individual department or workgroup. Keep the windows desktops there, add X-servers, and run some apps centrally.

If the question is licensing, a net-booted corporate (Linux/FreeBSD/NetBSD) desktop wouldn't be too difficult. Run locally what the machine can handle, make sure the network is super fast, and run the apps requiring performance on the workgroup server.

This would reduce the need to upgrade lots of hardware (which will surely be obsolete again soon) and minimize downtime in the process. If need be, some legacy apps could survive under Wine locally or, again, at the workgroup server. Keep storage centralized to facilitate backups across the entire organization (OpenAFS?).

It's an old way of doing things but overlooked far too often. You've obviously got to run the numbers but, surely, "a few good men" handling things on the server (they would handle the app server AND the centralized, consistent-across-the-organization, netboot image(s)) would be much less expensive than the workload on an IT staff required by an office full of people and their problems on Windows machines.

Maybe not *THE* solution, but certainly worth a look for many.

1. Active Directory to Novell eDirectory, although that doesn't really give you much. No real Open Source functional alternative.
2. Exchange server to Open-Xchange
3. NTFS to perhaps XFS or Reiser, orOpenAFS, although OpenAFS is really lots better, and has tons more functionality

https://lists.openafs.org/pipermail/port-netbsd/20 05-January/000011.html discusses AFS on NetBSD, if that'd be useful.

https://lists.openafs.org/pipermail/openafs-devel/ 2005-January/011370.html

looks like somebody noticed.

I use Andrew File System (specifically, http://www.openafs.org/) for my files, since I was used to using it at school, and I'm fond of its access control system. It allows you to designate redudant sites for your volumes for backup or load balancing purposes. However, its major downside is that it's optimized for reads but not for writes (PVFS would probably work better if you need optimal write performance), and it can be a real bitch to set up for the first time. I've also yet to figure out how to get it to work through my NAT, though it's supposed to be possible. It beats the hell out of NFS (v2, at least, I haven't really taken a look at NFS v3) in terms of reliability, security, and scalability, though.

I didn't realize that it had automatic expiration. I must have missed that somewhere while reading the documentation.

The cache going away does not lead to data loss. It does lead to really shitty performance while the cache repopulates, but all of the data will still be in the database which is completely separate from the cache. If it was considered necessary, it wouldn't really be hard to load up a bunch of key objects into the cache from a script but that would be guessing which objects are going to be needed while just letting it repopulate and suffering some slowness for a few hours gets the right objects into the cache. Different applications have different needs.

Don't feed me bullshit. memcached dies and so does your entire cache. That's significant data loss no matter how you want to spin it.

I don't know what HA-NFS and AFS are, but I know that using Squid (assuming you're talking about the HTTP proxy) would be caching at the wrong level. Caching constructed pages is pointless because most pages are completely different for each logged in user. memcache caches the atoms of data necessary to build the page, such as information about users and journal templates.

Squid doesn't just cache pages, you know. I can cache a wide range of data that's served over http. Sound familiar? If you've read the memcached protocol documentation, it should.

As for the others: OpenAFS and HA-NFS. So much for "evaluated other solutions". These are both lightning fast high-availability NFS replacements - AFS sports numerous features such as client-side caches. And yes, they are open source.

Whoop de doo. Slashdot is looking at memcached. Their DBMS is notorious for corrupting itself, so that tells me quite a bit about their availability concerns.

Like I said - this may work great for LJ and Slashdot, but there are enormous e-commerce sites (that believe it or not, use a heckuva lot of OSS) that have a little more to worry about than losing ad revenue for the 10 minutes it takes to repopulate memcached. Having that kind of downtime simply is not possible. You not only lose sales, depending on your caching strategy, you can get unrecoverable orders, or just outright lose customers because your site is slow. It's not uncommon, either, it's pretty much a guarantee if your site gets slow or goes down for any extended period of time - your full-service uptime directly correlates to sales for sometimes several months, and god knows you're fucked if it happens during the christmas season.