Slashdot Mirror
Organizing Data Across a Heterogeneous Net?
angst_ridden_hipster asks: "Like many people, I have a bunch of machines I use regularly. These include Linux machines, BSD machines, a Mac OS X machine, and a Windows machine. These machines are on a number of networks. All have internet connectivity. Some of them are always powered on. A few of them are not. Obviously, I have a bunch of accounts. And, it goes without saying, I have a bunch of data. What are the best approaches to sharing data? I want to be able to securely access my home data while at work, and from one machine to another, etc. Opening ssh terminals is the approach I have traditionally used, but I'm beginning to wonder if some mirroring software (e.g., Unison) might be in order. It'd provide the function of backups, as well as guaranteeing availability. Would it be wiser to tunnel nfs over ssh? Or is there some better option?
Assuming I actually start mirroring data across multiple machines, I'll need to organize it in a portable taxonomy. This is almost easy, since I use cygwin on the Windows machines, so I can assume a standard Unix-ish directory structure. But this gets more complicated when there are scripts or other code involved. What about application/platform-specific data? How do other people organize their data, anyway? Are there any useful standards? I'm hoping people will describe their approaches, and why they think they're (not) the best."
I have a fairly heterogenous network too, windows, linux (both PPC and x86) and OSX... I generally use VNC to access them remotely and scp to copy files back and forth between them
Gnutella is by far the best way to share your personal data.
Being able to sort and store email on a single server accessible from everywhere is a must...
What's the "F" and "J" stand for?
Without knowing more about the type of data you're storing, I would recommend putting it in a database. I like PostgreSQL 7.x myself.
For the software, I would organize it in a directory structure and use rsync+ssh to mirror it as needed.
For backup software, use Amanda.
For file sharing, use Samba.
'Nuff said.
I've been thinking of tackling this problem for awhile too. The best I can do is that you abstract the 'directory' (the list of what you have), for replication, accessibility (with convenience as the priority, especially). Then, when you need to do something with that data, your directory knows where it is and how to get at it. In this case, the convenience of accessibility isn't as crucial, and thus the need to transparently glue all these platforms and protocols, etc together isn't quite as important.
For me, I'd just like a top down, real time view with convenient access of what I have - getting it anywhere and anytime isn't quite as crucial for me.
Maybe you make a little daemon that can monitor your data respositories at several sources and 'merge' the data listings at a central source for publishing to multiple sources again?
"Old man yells at systemd"
Not enough info to answer the question. How much data total? How much needs to go out of the house? Do you want common accounts to various machines? What machines do you use most? What kind of data are you storing/want access to? What is your backup medium, what os is it linked to?
Jesus was all right but his disciples were thick and ordinary. -John Lennon
I put my porn collection on a couple of Jaz drives and just carry them around. I call them my "jiz disks".
I found this on google: Amoeba WWW Home Page
This seems to me to be a unique way of sharing data, since it isn't machine centric. Rather, it focuses on the user and the user's data. I have no experience with Amoeba, but on the face, it seems to answer this person's question.
My question is this: Why has interest for Amoeba dried up? (Or has it?) What with the proliferation of alternative OS'es over the past few years, why hasn't Amoeba caught on?
Put all the data on Exchange Server. Access it via VPN (unless or cable or ADSL supplier blocks incoming VPN, of course). If you need to access it from a box which doesn't have a suitable client use the web interface. You get moderately clever syncing with the message base on the lap top, which you can then use off line, and all the shared calendar stuff as well.
IBM has released Transarc's AFS as OpenAFS (http://www.openafs.org). Don't know if that is what you're looking for, but it is pretty nice. It's also portable, so it runs on various unices as well as Windows. Most can be found as binaries if you don't want to roll your own.
AFS is an NFS style implementation though, so you would have to save your files onto a special mount.
Wealth is the product of man's capacity to think. -Ayn Rand
nfs, vpn, and vnc i vpn to my home network and use vnc to control a computer at home, or rdp if i want to connect to my windows xp machine, i have access to all my data, it isn't nessicarily organized, but i know where everything is at, i'm more concerened about having my work with me at home than my home at work anyways
--fetch daddy's blue fright wig, i must be handsome when i release my rage
I'm hoping people will describe their approaches, and why they think they're (not) the best."
Why whould you think we're not the best? How self-degrading do you thing we are?
If you're religishitty, KILL YOURSELF!
XML!
Web Services!
Oh, sorry. Knee-jerk PHB response. Move along.
Samba, Shares, Windows' Briefcase.
And a notepad equipped primate for redundancy.
they are all homos at Slashdot; that's why the dot is slashed; didn't you know?! old news, buddy!
"kio_fish is a kioslave for KDE 2/3 that lets you view and manipulate your remote files using just a simple shell account and some standard unix commands on the remote machine. You get full filesystem access without setting up a server - no NFS, Samba,
It works through SSH, so everthing is encrypted.
I use this with the konqueror file browser, but all KDE apps can transparently access files on remote hosts using this amazing utility, which required no special setup on either end, at least on my systems.
Solved all my data sharing needs - and andromeda solved the rest :)
sig sig sputnik
What you need is something known as a "server." A server is where you can store all your files, and in some cases, account information.
With the right kind of server, it can do AppleShare, NFS, and SMB, allowing all your other machines to mount the shares and make them appear as local drives. This keeps all your data in one place, allowing for easy backups, and also makes it easy to get at the same files from any computer.
My personal preference is a Linux computer with several cheap IDE drives each on their own IDE controller (no slave drives). The drives are configured as software RAID 5 and ext3. Regular backups are setup through cron to a tape drive. Samba handles file sharing, printing, roaming profile, and PDC duties for Windoze. Netatalk 1.6cvs handles file sharing duties for pre-OSX systems. NFS is used for file sharing to *nix systems. The only thing I'm missing is a NetInfo daemon for Linux so it can act as a complete configuration server for NeXTSTEP, OPENSTEP, and MacOS X systems.
I'd say what you need is an internet-enabled file system. Some might say NFS, and that seems like a fine solution.
On the other hand, if you have a computer that is always on, that can run Apache, you can have your own personal WebDAV server instead. Simply install mod_dav, and access it through mod_ssl, and have a secure web-based filesystem.
Better than NFS, you can mount it on Windows (through web folders), Linux (through davfs) and Mac OSX (through the native DAV file system client that is designed to run with iDisk).
NOTE: I work for Xythos software, and we make an enterprise-level WebDAV server called the Xythos WebFile Server. It's significantly more expensive than free, and we run in-house copies of the product (y'know eat your own dogfood), so that's where I keep my shared data, but if I didn't, I'd have mod_dav running right now.
http://www.cvshome.org/
"but I'm beginning to wonder if some mirroring software (e.g., Unison) might be in order. It'd provide the function of backups, as well as guaranteeing availability"
No really it would provide the "function of backups" becuase the purpose of backups isn't just for hardware failure, but also for accidental deletions, incorrect changes to files etc. So in this way if something like that happened, it would simply be mirrored to the other machine. I'd hate to loose my data.
On no! The feds are coming! C:\>delete *.mp3
Ooops, it was just the cat wanting in. $%$#% mirror! I wish I had a backup.
Sounds like a ng question to me...
/. == news?
Como muitos povos, eu tenho um grupo das máquinas que eu me uso regularmente. Estes incluem máquinas de Linux, máquinas do DEB, uma máquina do OS X do mac, e uma máquina de Windows. Estas máquinas estão em um número de redes. Todos têm o connectivity do Internet. Alguns deles powered sempre sobre. Alguns deles não são. Obviamente, eu tenho um grupo dos clientes. E, vai sem dizer, mim tem um grupo dos dados. Que são as mais melhores aproximações a compartilhar de dados? Eu quiser poder alcançar firmemente meus dados home quando no trabalho, e de uma máquina a outra, etc.. Os terminais do ssh da abertura são a aproximação que eu me usei tradicional, mas eu estou começando a querer saber se algum software espelhando (por exemplo, Unison) puder estar em ordem. It'd fornece a função dos apoios, as.well.as garantir a disponibilidade. Seria mais sábio tunnel o NFS sobre o ssh? Ou há alguma opção melhor? Supondo eu começo realmente espelhar dados através das máquinas múltiplas, mim necessitarei organizá-la em um taxonomy portátil. Isto é quase fácil, desde que eu uso o cygwin nas máquinas de Windows, assim que eu posso supor uma estrutura padrão do diretório de Unix-ish-ish. Mas isto começa mais complicado quando há uns certificados ou uns outros código envolvido. Que sobre dados de application/platform-specific? Como os povos organizam seus dados, em todo o caso? Há algum padrão útil? Eu estou esperando que os povos descrevam suas aproximações, e porque pensam eles são (não) os mais melhores.
Interesting question. The correct answer is, of course, "it depends."
I like to put as much of my information on Web pages as I can. Most of it I use read-only, and some of it I can update through a form. That way, it is easy to get to from any browser. Secure with HTTPS and passwords as needed.
Does anybody know if NetBSD has been ported to this yet?
I keep a few things I want to be able to access from anywhere in my webmail. Perhaps for larger things you could use a version control system, I'm sure people will help recommend one which is usable across different platforms (I haven't found a CVS client for Windows that I like yet, but then again I may not have looked enough).
For email, I only use IMAP (no POP). It rocks. Unfortunately Earthlink sucks ... ahem I mean they don't support IMAP yet, but that account is a spam-trap anyhow.
Web bookmarks are more of a problem. Creating a webpage is one solution, but not optimal. So far I'm still looking.
As for data, I stay away from MS products whenever possible, Luckily I mostly use LaTeX.
I've wanted to setup a NAS of some sort, and mount the same home directory from all machines. MacOSX 10.5 should understand SMB much better, but I'd rather stay away from that. Still not sure best solution here either.
Hi. ? ??????????? ??? ? ?look at all the music
Dude, the community is hetero. The fags are the Slashdot writers, and what can we do about it. Hey, Cmdr Taco is the main gay here. I mean doesn't the name Cmdr Taco reminds you of the Village People?
I don't think you're going to get a one-size-fits-all solution for all your data, but the following might give you something to think about.
.rc files, scripts, etc. It's easy to set up a repository, and very easy to get at the data from anywhere, over ssh. Cygwin is the ssh/cvs client of choice on windows.
For a while now I have been using CVS to store my commonly-shared data like
The biggest effort I have found is in writing portable data that can be shared on all machines. And even then this problem can probably be avoided with branching or similar...
Food for thought anyway.
Damn it! You screwed up the display of the page! Grrrrr!!!
If you're religishitty, KILL YOURSELF!
Uniformize the platforms? Ditch some of the boxes? Laptop? A new box which will be your "server", and all other machines access it?
I think you can use secure nfs and the automounter to get your home dir to follow you around.
I do this regularly with normal nfs and automount on a lan, but I see no reason why this couldn't work in a secure fashion over a wider network.
Anyone?
I have twelve computers in my apartment and use all of them for something-or-other. Several are just test machines but even with those, I used to run into situations all the time where I saved something on one machine and forgot to do anything with it.
:)
My solution was to write a series of little scripts to copy data from common share points on each machine to a large, central data store, and into a "backed-up" directory on the workstations. Presently my central data store is 600GB of IDE disks in a RAID1 array (10 disks, total). If I lose the central fileserver, all my data, and the scripts needed to recreate the information in that 600GB is sitting out on my workstations
It's kind of a brute force approach, but it works OK. I'm not sure how well it would work for non-local systems, though.
I'm sure there are better ways to do what I do, too, but it's nice to have a single place to look for my MP3s or whatever, while knowing they're backed-up in multiple locations as well.
-- I wanna decide who lives and who dies - Crow T. Robot, MST3K
first of all, seperate your home life and work life. Then seperate the data. I understand that once in a while you need data from one place at the other, but avoid those situations.
At work: that is IS's problem. Store all work data on the work machines, and make IS do the backups. Use SSH, or other VPN when you want to work from home. Compile (or whatever) at work as much as possible. If you have data that you need on the road, get a laptop or PDA for work, and synchronize that when you are at work.
At home: set up a linux box (a 386 is enough, though you might want more) with a big disk, a UPS, and a network card. Put it in a closet or on a shelf. Install SAMBA, and Netatalk. with NFS built in (though there is better than NFS if you look, nfs is there) Use one loging for all machines.
Laptops are a problem, because you often want to use them where you can't get to the network. The first solution to that problem is 820.11. Use it at home, and look for open access on the road. With good VPN (ssh+nfs) you can get to your network server from many places. I manually synchronize only the files I need, but my laptop is rarely used outside of 802.11 areas, if you travel often, then you might need more. (CODA? AFS? )
I'm not sure if this is entirely applicable to your situation, but here's what I do, and it works reasonably well.
:P
...
I have a server on a public IP address that runs SAMBA, but only accepts connections from 'localhost'. From my Windows box and iBook (running OS X), I just do a bit of SSH tunneling, and I'm able to mount the machine from anywhere I happen to be.
As far as I can tell, it's reasonably secure, and it works just fine for general files.
I also have a CVS repository on the server for my development projects, but that doesn't work so well for binary files like images and Word documents.
One of my friends keeps his files synchronized via an htaccess protected website which allows him to download and upload files. If you're interested, I'll see what I can do to track down his PHP script
Exchange is one of the worst pieces of software I have ever had the misfortune of working with. The most laughable thing I ever saw was an Exchange Book that said: It is often preferable to backup and reinstall the server instead of trying to find a hidden setting that is causing your configuration problems. Unbelievable, only a Windows user would think this is an acceptable solution to the Troubleshooting section of a book on running Exchange Server. This facilitated my migration to Linux. Have learned a lot and saved a lot of cash in the process.
Now this is not totally fair, since it implies a pointy haired boss situation. All it really means is that that you would would have to have a better definition of the problem.
What it seems that you really need is an application, a database, that would constantly monitor in realtime the status and availablility of your various resources. This would tie into your other dataservices so that when you do a query on "XP sourcecode", or whatever, one of the result you get is from this resource monitor database saying that "the resource is offline" or "the data is available, but you don't have access rights", etc. depending of the resource status, and other realtime situations.
It occurs to me that clever design of the database may be able to do the resource availibily query in advance of the actual access of the data, so that you do not get a crash or whatever if a child record or whatever is unavailable.
Currently, I do not know of any tool that does this, although obviously this is not my area of expertise.
"It is a greater offense to steal men's labor, than their clothes"
I might be off base here, but..
Why not use Gnutella or a similar P2P system? There are clients for basically any OS out there, the files don't have to reside in a central location.
It works for the internet - Why not your own 'mini-internet'?
One modification you would want make is to get it to make a listing of all that you have.
Could you use SSH tunneling with a system like that?
Put a small raid 5 partition on your *nix machine. Store everything there, and use your access of choice to use the data on your other machines. I like ssh/scp for access, as it works mostly everywhere, and is encrypted, but then again most of my data needs accessed in a CVS manner. If you are constantly editing word docs and the such, samba would perhaps be a better option.
This way the data is in one spot, but it's much less vulnerable to hdd failure. Plus since it's on a *nix machine, you can export it to your clientelle.
Unless you want to share your data with lots of 'friends' you just haven't met yet.
NFS is used very often to mount home directories. But what is stopping someone from unplugging the workstation, plugging in a linux laptop with the IP of the legitimate workstation and mount the share, "su - user", and voila, you now have all the user's files.
That's just the simplest way. The problem is that most NFS implementations don't have *any* authentication except for IP authentication. So so other DNS attacks would work as well.
I am surpised that the most widely used network file system implementation for linux and most posix OSes has no real authentication. There *has* been authentication built in the protocol since version 3, but last time I checked, it was not supported on the linux. I was told by one guy working on the project that the problem was that there's no crypto in the kernel.
I used secure NFS on Solaris 8 for a while but I constantly lost the mounts. That but be fixed now, I don't know.
Use AFS, CVS, rsync, intermezzo, or something. But I would stay away from NFS.
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
Yep. Unified access to e-mail via IMAP is definitely the linchpin of a good arrangement.
I've been trying to deal with the same problems as you for several years. I have a Mac running Mac OS X, Windows PC, Linux server, and a NeXT around my desk. I have two large hard drives. One is in the Mac and that holds my home directory, and the Linux machine has all my MP3s. My home is exported via NFS and is mounted on the Linux box and on the NeXT so I always have live access to my files. The Windows box only does my TV program and Kazaa, so I'm content to simply have it use FTP to copy files back and forth (I haven't found a decent Windows NFS program.)
It all gets the job done, and it all works smoothly. Printing is done by IP printing to my big 'ol LaserJet. All the mail is kept either on my server at school, or on the cyrus server on the Linux box. It's a delight =)
There should be a moratorium on the use of the apostrophe.
Max V.
NeXTMail/MIME Mail welcome
There's this great standard for sharing files over the internet called the World Wide Web. Perhaps you've heard of it?
Seriously -- run a webserver + WebDAV on each of your machines. Then you can read/write from anywhere, and with any platform.
Systems like YouServ/uServ provide a webserver, access control, and mirroring/replication support in a single package. This way as long as only some of your machines are online, the data from every machine remains accessible. Unfortunately the system is not available for general public use, but the system may be in open source soon.
That's what it's for, right?
Seriously, I think it would be great if there was a P2P backup system. Private files could be encrypted, and everything could be uploaded to multiple peers. Obviously some sort of trust system would have to be worked out, but it could work. Even if I just connected to myself and two or three real life friends with DSL connections, it'd be great to have my files accessible everywhere, almost all the time.
It's NUMBER ONE!
CLIT. Are you a memb
I have two large drives sitting on a Linux box doing RAID mirroring. For remote access, I use ssh/scp. For local access from other Linux machines, I use NFS, and for Windows machines, SMB.
. music/singletracks/cache1/files.mp3.v ideos/ ..
The point is, everything is stored and "backed up" centrally, but accessed using a different mechanism depending on where I'm at when I need my data. Since I don't delete files accidentally, mirroring works fine for a backup - I'm really only concerned with drive failure.
I then structure the directories according to type of file. I've got a documents directory where I keep anything I create myself. Specific projects that require multiple files generally go under documents/projectname. I've got a music directory, and many subdirectories under it:
music/fullalbums/artistname/albumname/files.mp3
music/music
Etc. Then software. apps/isos. apps/windows. apps/linux. And so on, and so forth.
-Ryan, with the unoriginal sig
this goes along with the subject somewhat. Does anybody have experience sharing a bookmark file, let's say, being able to place bookmarks and view them across multiple computers and browsers? (I'm not sure if bookmark files have a uniform format so that may not be possible to do across browsers) I just have too many bookmarks, and when I go to look for one I can't remember if it was at home, work, laptop, or what?! Anyone have a good solution?
How about a Beowulf cluster? :p
Use the excellent rsync from Paul Makerras (of pppd fame) and Andrew Tridgell (samba team) in combination with OpenSSH and SSH for windows (both based on Tatu Ylonen's work; OpenSSH is maintained by and expert team including Markus Friedl and the recently monkey-cracked Dug Song, among others).
Set up your accounts to rsync-upload changes to whichever server is most secure when you log out, and use a cron job on that server to rsync-download to all the other servers nightly. You can make a tar backup part of the system also.
You will have to remember what's going on so you don't modify the same file differently on two different systems within 24 hours. If you want to overcome that shortcoming by making this work on an immediate sync basis rather than periodically, you'll need something like SGI's fam (included with recent linux distros) to trigger the updating processes.
You should already be 90% there if you have your ssh keys set up for passwordless login. Passwordless PKI logins are not significantly less secure than passworded logins in most situations (granted hostile system management can get you, but the BOFH can trojan your login anyway).
Lots of people use this technique to sync CVS trees over slow links. Rsync is very efficient for that kind of thing (large volume of files, low number of changed bytes).
3D volume holographic optical storage
introduced by Lucent and in development
by others will someday replace all memory.
http://www.colossalstorage.net
So this is not a troll, but have a look
at hier(7) on FreeBSD. Since bsd is
more than just a kernel, there's already
a great deal of thought put into
directory structures and layout. Have
a look at the bsd man page for hier(7)
and get some ideas on how to
approach this.
there are no heteroes who use linux, only homos
I am obcessed with OSes, and I have running Solaris 8, redhat 7.3, freebsd4.5, slackware8,win2k,win98,netbsd and unixware with each computer having both ethernet and tokenring cards. The FreeBSD machine has a directory tree where all files are stored with different ownerships and perms. They are served on different interfaces as http, nfs, samba, ftp, and netware volumes. security req is low so I dont use encryption, except for the www service to the internet where i use the ssl. These tools are all out there are pretty well documented. They all work well in a practical setting. What else do you want to know?
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
You have a requirement that isn't met by any existing software?
Start coding.
And screw Stallman. Release it under the modified BSD license.
What's this new 820.11 protocol? :-)
© 2004 The SCO Group, Inc. All Rights Reserved.
First, I try to adhere loosely to the FHS for ideas on overall organization. Even though it's mostly intended for POSIX systems, following their philosophy will really help you separate your data from your platform-dependent program files and libraries.
Most of my important stuff goes on the Linux server in /home (on an IDE software RAID1). However, I try to limit files in here to stuff that's absolutely essential to keep the size down. I occasionally mirror this offsite to my friends' servers with rsync (with the private stuff pgp encrypted). I try to make browser caches, etc. symlinks to dirs in /tmp . Try to keep only the stuff you created yourself in here.
I keep media and downloads on a plain partition under /home/ftp/pub (which is also symlinked from the http document root). That way, all my computers can easily get access to music and installers and junk.
Samba helps win32 boxes access the /home and /tmp directories.
NFS exports /home to the other UNIXen, as well as /usr for the other machines with the same CPU arch. It should be acceptable to export /usr/share to other UNIXen with different architectures.
I'd like to set up CODA, since it seems to support more different kinds clients than Intermezzo. These support disconnected operation and are good for laptops. For the meantime, I just use rsync to mirror home dirs onto my laptop, though (and just keep track of stuff that I change on the road manually :/ )
No thoughts on how to combine everything into a distributedFS so you could have parts of, say, a music archive living over several machines. There are several projects for Linux-only (PVFS) or Win32-only (more advanced network-neighborhoods). I'd say your best bet for convenience is just to make sure everything is visible from your one server and reexport it from there (invest in a switch so it doesn't deadlock your network). Until better DFSes exist, though, I think you'll get better performance and less confusion from running everything from one beefed-up server with a RAID (or two if you want failover).
Maybe we can extend this problem and start an OpenSource effort to develop a high quality implementation of the OceanStore concept.
Securely access your data from anywhere. fault tolerant, secure, replicated, synchronized, pervasive.
I recommend you to take a look at the page and read their papers/presentations. An interesting idea.
I am in no way associated with the OceanStore project. Just an interesting research project I came across.
(\(\
(^.^)
(")")
*beware the cute-bunny virus
Here's my situation: I have a dual-booting Linux/Win98 machine at home, a Win98 laptop, a Linux server sitting in some network in a galaxy far, far away; and a bunch of other computers around the world.
At one point, managing all my data (I would change a bit here, and a bit there, then try to copy and synchronize by hand) was manageable, but I got real tired of it real fast. I considered putting together a CVS server, and then synchronizing that way, but it's really overkill and not a very user-friendly solution anyway.
Enter Unison. Now I just have a few directories designated as shared, and they get synchronized by Unison automatically. At home, my data is on a FAT partition, which is accessible to both Linux and Win98.
The good thing about this is that since I synchronize with the laptop when I'm connected, I get to use my data even when I'm on the move - not so with NFS. And I get free backups as well - I do have roughly 2Gigs of data, which would be a hassle to backup any other way. Besides, if I took tape backups, I would have to manually carry them off-site in case of a fire; now Unison takes care of backups to and from my remote machines.
Depending on your exact requirements, Unison might be the thing you've been looking for. It was for me. A bit of SSH config'ing and you're basically in business. For one thing, Unison just doesn't screw your data up . Secondly, it only updates what needs to be updated. You don't need to explicitly add new files either. It also has a sufficiently powerful filtering mechanism so you can have it ignore temp files, whatever they may be. It can be used in a centralised manner (with proper backing up) but is equally happy when used as P2P for plain synch'ing. I use it for my external HDD back-up, my public repository, half-way round the globe (literally). The lot. [Thanks B, et al.]
At least you should give it a spin.
Having all your data accessible from all of your workstations is highly overrated. I think you should just delete all of your data and quit your job.
/. way too much.
You hang-out on
© 2004 The SCO Group, Inc. All Rights Reserved.
Dave is a CIFS (SMB) uberpackage for Mac OS and Mac OS X, by far the best CIFS implementation I have seen for Workgroups.
Dave+Samba+CIFS for filesharing.
ssh with RSA keys for terminal type stuff.
VNC or X for Desktop Sharing.
That's the way mine is set up.
The Crazy Finn
"You've got an invalid haircut" -Warren Zevon - Life'll Kill Ya
the connectivity and security are as versatile (or more so) as unix pipes; also you can write programs for it that run without change (really!) on any supported platform ('cos it provides an OS level view of everything rather than trying to shoehorn itself into the parent environment like java).
the security model is public-key based and because it's end to end, you don't need to worry at all about little things like 802.11 insecurities...
plus it's all small, clean and beautiful as befits something coming from CSRG at bell labs.
For those systems that are on all the time, select one system to be a common server, I personally recomend a Linux box, though xBSD or OSX may provide the features you need as well.
In your home directory, create a folder you are going to put your mount points in to mount the data stores you need.
On all the other systems, create a share that will contain the data you want to access "anywhere". On the central server Mount all of these shares in that sharesmount folder. This may be nfs or cifs as the architecture of the servers dictates.
As this is all mounted to your home directory, you can go to just about any system in the network and remotely mount all of your folders by Mounting your home folder from your primary server.
To remotely access this storage center, use either nfs over ssh, or build appropriate links into your web pages, and run a secure varient of apache.
I also recomend keeping your work data in a seprate storage area from your personal/home data. You may recall that Northwest Airlines successfully sued to get the personal computers of Flight Attendants who they believed co-operatively negotiated a sick-out strike. Keeping your personal data completely separate would reduce the likelyhood of loosing your entire computer setup if someone at work files a complaint that they believe you are doing something wrong.
There are other advantages to this kind of a setup. By centralizing your data storage tree, it is easier to perform backups, you will only need to backup the one server's home directory, tracing into the peripheral servers. If you wish to set up a thin client in a bedroom, or someplace where you don't want to have a lot of fans going, this gives you a platform ready made for your storage needs, as well as a reasonable terminal server. I think you get the idea.
-Rusty
You never know...
I don't think that the whole email thing is a particularly good solution to the original question, but why should he pay for microsoft exchange when he could do this with any IMAP email server?
I have a cyrus imap server with webmail install that does exactly that. Also Courier MTA comes with a the pieces needed to do this btw.
Shared folders, server-side mail storage, SSL/TLS security are all part of the IMAP/IMAPS protocol that many ( most? ) free email servers support very well?
So why should he pay the $10-$40* per user license for microsoft exchange? *( lost track of the price )
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
I had this problem. It was simple. I Set up one machine with a Large drive volume (7@9gigs and Raid5 with 64mbs cache). It is not real fast but awfull snappy. It is a Dual 133 with 256mb but it serves this purpose very well. It also has CVS for FreeBSD and Squid and DNS.
Solution. NFS and Samba.
NFS: Export you Home over NFS and have AMD or Hard Mounts set up for home's on the machines. This should fix OS-X, Linux, BSD and the like.
Samba: Share/Export the Very same Home. Have it set up to map from you 9x/NT/2K/XP systems.
I have also contemplated Linux with Oracle and the IFS service. You can NFS Mount/ SMB Mount/ FTP to it./ Email you files in.. pretty cool but never realy looked at it.. any one what to give ware stories on this so I can be tempted to change my setup?
Blur - Coffee + TV = slashdot
Karma: Good (despite my invention of the Karma: sig)
Goat
Keep your valuable files on a Linux machine (running on good hardware you can trust, a stable kernel, a journaling filesystem, and software RAID if you want to go that far). Do backups from there. Run NFS to serve all non-Windows clients. Run SAMBA to serve Windows clients.
/home/foo and on Windows it's H:\ - all the files are there.
Sharing data files is easy with the NFS/SAMBA combination - e.g. non-Windows machines mount my home directory as
Sharing software is less easy since none of the common UNIXy filesystem layouts really let you have binaries for multiple platforms available at once. There are unconventional layouts that do this, but you'll have to compile a lot of things yourself and mess with configure scripts a lot... I've given up on sharing binaries and libs; I just run Debian on as many of my systems as possible, and run a script now and then that ensures the same packages are installed on each machine.
For remote work I use SSH to set up a VPN. However, unless I'm on a very low-latency connection, I find it difficult to use a shell remotely, much less NFS. I usually end up manually rsync'ing the files I need.
I use CVS to organize stuff I care about sharing. The stuff I don't share is NFS mounted (or sambaed) from a server either at the office or at home. "Rsync" and "scp" to move stuff when desired.
I think LDAP+Kerberos+AFS looks interesting, but there is a learning curve there that I have not climbed yet.
This works well for me to keep about 30 accounts in sync, most of them just get a minimal checkout of my home directory (5 mb or so), while 3 or 4 get the whole home directory and rsynced files (5 gb). The CVS repository is about half a gigabyte in size these days.
Once something that allows proper file rename tracking, like subversion, comes along, I plan to stop using rsync alltogether, and just check all the files in.
As has been noted elsewhere in this thread, one of the key things is coming up with a consistent directory structure and sticking with it.
see shy jo
/porn/
/video/
/pics/
/gamez/
/app z/
/audio/
/rock/
/hiphop/
/jazz/
/warez/
/mp3/
oh filter, why must thee filter my comment
MARIJUANA, SHROOMS, X: ONLINE?! - E
http://sfs.fs.net
When I was still in school I tried to figure out some good way of being able to work on my research project at home and at school, or at least massage the code and the data in both places.
Being an engineer, I thought of a bunch of ways of setting up complicated distributed ways of doing this, but settled on just leaving the data in one place, and SSH'ing to that box.
The benefits of keeping it simple were:
1. No new work, which is good for the lazy^H^H^H^Hefficient among us.
2. Data coherency. If its only ever in one place its hard to mess up.
3. Backups are easy, since you're only backing up one data set.
4. Did I mention no new work?
As much as data sharing on a heterogenous network would have been nice (Linux box at home, Suns in the lab, Windows at my parent's place, iBook in my backpack), the marginal utility of that data sharing was low compared to the marginal cost of actually doing the work to make it happen.
My vote is for keeping the data in one place and remembering how much you love the terminal. Not a sexy solution, but it works.
Outside of a dog, a book is a man's best friend. Inside a dog, its too dark to read.
A while back (3+ yrs) I've toyed around with a pretty slick setup of a VPDN (Vir :) :) Joke aside, it was actually consisting of an 8-
tual Private Dial Network)
bit clean ssh tunnel over which I was running..PPP !. And, since that was doing
"proxy-arp", from I was capable to get my work setu
p, with NFS, NIS and other assorted local services, all accross non-cooperating
NATs and firewalls. I even manged to present it as VPDN project (so the above jo
ke is half serious).
There are some drawbacks:
- To get decent performance from NFS (back then we had a paranoidal-fine-tuned S
olaris NFSv3 server), in addition to a really fat pipe (I have a FastEthernet co
nnection from home to work) the most critical part is that you have to have real
ly short RTTs. Thankfully I am aprox 200 ms away from the server and it worked "
quite" decently (was capable even of streaming 320x240 MPEG over that baby, w/o
noticeable jitter!).
-Do NOT use a Linux machine at the server side. After extensive hacking (w/ prec
ious help of Paul Mackerras) it turned out that the Linux driver back then (2.
2.12ac3 IIRC) it simply sucked and it couldn't keep up w/ the bit rate. After co
mpiling pppd on Solaris it worked like a charm.
- Protocol overhead is NOT an issue. As crazy as "NFS over PPP over SSH" might s
ound, it wasn't that bad, given the circumstances. CPU overhead was, and I was u
sing a relatively light-weight stream cipher (blowfish IIRC).
The reason I had to give up this dandy setup was simply that it works reasonably
only in very special setups, namely fat pipe AND _short_ RTTs.
What I do nowadays is 1) VNC or 2) ssh + running programs remotely (X11 forwardi
ng). Otherwise my home and work data are best kept sepparate (as another poster
already suggest).
Back to your question: While I do not have any experience w/ Unison, I would per
sonally go for "rsync" (optionally over ssh if paranoia is a must).
This post should be nominated for largest front page post ever. Throw it on the HOF!
Okay, you *could* use some form of networked file system, but a) your laptop and other machines would need to be connected to use it, and b) I hope you are willing to fight to get a good implementation to work, and c) I hope you aren't playing with big files
I use rsync. I have ~/Makefile, 'make sync' works wonders. Here's the contents:
On the laptop:
Works like a charm
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
This is not a Fugazi
I don't know about IE, but netscape/mozilla supports roaming profiles using LDAP. That way, all your profile info gets saved in the LDAP server and you just configure your browser to pull the information from that server.
LDAP is also often used for other small pieces of information like certificates, etc.
Check out the OpenLDAP website for more info.
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
If you've got the extra cash, buy the low end Xserve from Apple, set up a local area network connecting all the computers, and store all your files on Xserve. You already have a Mac OS X computer, you can control the Xserve with that. And if Apple's Xserve is really as easy to use and administer as they claim it to be, it would probably be worth the money for you. My suggestion only applies if you don't know enough about setting up servers and you want the quick and easy way out of your dilema. Hope this helps.
This question (or ones like it) has come up many times. This isn't the first time something like this has been posted on Slashdot. I'm currently looking at doing something like this myself and I'm obviously not the only one. While that lays the ground for a good open source project (ie- a distro that is set up for something like this, or a project that easily combines several tools to do this kind of thing), what I think we really need is a good HOW-TO. Maybe there already is one or are several related HOW-TO about setting up this type of file access. There have already been a number of good suggestions posted here on Slashdot. We need to get these and others together and put into a HOWTO so that it's not a research project every time someone starts exploring this idea of distributed data and somehow consolidating the mess. (And no, I'm not volunteering yet since I haven't done this yet and currently don't have the resources. But if something doesn't happen in a while, maybe I will...). If you know of a HOWTO or other site that covers this info, you should post it somewhere here.
Who said Freedom was Fair?
Please don't call people names.
Instead, read this page:
http://www-personal.umich.edu/~srb/openafs/
...Linux is dying. ... I plan on installing Windows 2000 on each node ... [of] a 500 node beowolf cluster.
sounds to me like it's your wallet that's going to be doing the dying...
not sure why I'm feeding it... oh well...
Oh god, that woman is John Romero!
I would think about taking a look at E-Smith Server. It could buy you a few different things, though with the exception of automatic backups that synching will buy you. Internally you could have Samba and Appletalk running, externally, you could connect via their I-Bays for file storage. Also has an IMAP server if you wanted to store things that way. I guess you could cron copy from this machine to another for the backups.
Novell's iFolder solves your problems quite easily ;)
WebDAV is used practically by many Mac OS users in the form of the iDisk service on Apple's iTools network services. Being an open standard, there must be some commonality that makes it practical to set up WebDAV services on any or all boxes for basic file sharing, or even a common location. iDisk itself isn't the solution, of course, but it shows the practicality of a WebDAV solution.
Vos teneo officium eram periculosus ut vos recipero is.
This advice is likely to get a lot of linux geeks electrocuted when they take their computers into the shower before turning them off. Good start!
I'm not really sure what type of work your doing where you need access to your files... I can relate my knowledge on dealing with unison over the past year though.
I do a lot of back end web development. As such I usually like to copy the entire site down to a local machine, work on the system, upload to a test machine, test, and then move to a development machine. Unison has made my job a lot easier than it using a bunch of ssh scripts since unison automatically checks for changes and only copies over files with changes.
A sample script is as follows:
From my local file system $HOME/web/(website) I execute the following script
unison -auto -batch include ssh://user@somehost.com//www/(website)/include
unison -auto -batch www ssh://user@somehost.com//www/(website)/www
This script pulls all my programming work in include and the website accessable files www to my local system... I then work on the files and upload using the following script
unison -auto -batch include ssh://user@testhost.com//www/(website)/include
unison -auto -batch www ssh://user@testhost.com//www/(website)/www
I then check the coding and on the test host, when I get it to the point I want I upload it to the production machine...
If I have problems on the test host, I can go in and remove all files on my development system and pull a fresh copy of files from the live site...
Since I don't need to program and compile on different systems, just uploading the the test and production machines it works well.
Recently I took a trip and did not have access to my local system. I was able to borrow a windows system and after installing putting, winscp and unison I was up and running within 10-15 minutes at the remote site, which allowed me to get back to work.
The problem with using a remote mounting system is that you have to maintain network connectivity while working on files, not always an option, plus you are working with the live production files...
So basically I use unison just like a cp command except that it does not copy files that already are synced between systems and it automatically keeps my permissions sync'd as well.
Hope that helps
/* TODO: Spawn child process, interest child in technology, have child write a new sig */
Some of us "Mac users" have afs experience (not only using but admin from startup through general matience) as well as general admin skills and very good coding skills. If you actually knew what you where talking about you probably wouldn't make broad generalization but oh well your a true slashdot moron.
You might want to check out the CODA Filesystem. Its a secure network filesystem like AFS and it has transparent replication support for offline access. Also, there's AFS, but that doesn't provide offline access, but I believe it supports more operating systems than CODA is currently ported to.
In addition to the usual $HOME/Documents/(doctype|projectname) structure, you mention binaries; uou probably have a bundle of $HOME/bin and $HOME/lib files for each platform? In my case, I changed to $HOME/bin/$arch and $HOME/bin/share; replace $arch as appropriate, and set up your profile(s) to set PATH and LD_LIBRARY_PATH as needed. (Most all shells give you some idea, or you can resort to "uname=`uname -a`; if [ "$uname" = "..." ] then export arch="..."; else ... fi") For example, $HOME/bin/cygwin-win32; $HOME/bin/linux-i386; $HOME/bin/darwin-ppc; Perl, shell scripts in $HOME/bin/share and include that always.
For the Macs, there's also netatalk. Turn off exporting "hidden" files and samba is more polite with both Unix dotfiles and the plethora of netatalk Finder Info / Resource Fork files; but don't use samba/nfs to move a file without moving its resource fork as well! (a touch annoying, that) -- same applies for Finder Info files in OS X on BSD partitions. You can always mount a partition in hfs, then samba can move around files nicely (resource fork moves with data fork), but GNU+Unix clients get pishy about the permissions (or lack thereof); this only applies to files set up for "Classic" MacOS.
DAV sharing rocks for remote access, and Linux and OS X have it natively -- I think ? it's available for Win32. There are some DAV-like interfaces available for browsers. (Does OS X have DAV-over-SSL? I think so, but anyone confirm/deny?) Browser interface with an input type="file" uploader and the usual download set-up is pretty near a universal interface.
In extreme cases, you can share files over IMAP; pretty much limited to text, but I've used it more than once to dump shell scripts, useful recipes, and software-in-progress out onto a network. Limitations are in the client software, not the IMAP protocol; IMAP from maildirs is particularly cool, since you can just MIME-encode files and pump a "fake" header onto them with a pretty short shell script to push them in, no mbox-mangling required.
done.
this has probably already been said, but i'll say it anyway :)
for my file sharing needs i have one machine between my router and internal firewall that i can ssh(openssh.org) to and use the rsync or scp utils to get data off of there. My internal mp3 machine is WinXP so i use smbmount(samba.org) from the DMZ to mount the shared DIR if i need to get to an Mp3 from the outside, same goes for any other win/linux machines on my internal network, all internal *nix machines run samba
i've never had a problem or thought that this was inefficient, so it works for me.
People who have witty things here blow.
active directory in a pure M$ environment will lower your TOC and save you time and money
more drugs please
I keep all the porn in a seperate directory. That seems to work pretty well.
DataDirect Networks for the S2A 6200 or 3000, as much storage as you want with 8 (4 on a 3000) Fibre Channel ports going to hosts or switches.
Then SANergy for file system sharing software.
I agree with you. Your question though, was overly general.
.bashrc, environment, ssh directory, pgp keys, etc.
.bashrc .bashrc, put specific things in the separate files.
.cvsignore to make sure it only manages the files you want it to. .emacs macro, or shell prompt tweak? Edit one account, cvs commit, cvs update the rest.
There's really three (or more) different separate data issues that you have to deal with.
Like most, I have many accounts, and just manage them on the fly. My data is retrieved manually when I need it. SSH (and scp), VNC, etc. This usually does the job.
Not the easiest way to do it. Especially when I recently changed jobs and had to setup new data and profiles - I thought, there must be a better way to do it.
So, here's a breakdown of the problems, and suggested fixes.
Break it down into 3 separate sets of data:
1. Profile data - Your shell scripts,
2. Daily Documents - My Documents folder, data directory. Limit this to stuff you need in ALL locations (though you could have a personal and a work version...) and on a regular basis.
3. Archived files - Infrequently used, but you occasionally need to access them from various places.
Then, the problem becomes much simpler. Instead of a grand scheme to manage all three of these at once, you have three smaller, simpler problems.
Here's my suggestions:
1. Profile info - Wasn't originally my idea, but the best thing I've found is to use CVS to manage the files. You'll also have to setup your shell scripts to detect the OS / machine you are on and run OS / machine specific versions.
For example:
Detects OS, runs ~/.profile.d/linux, ~/.profile.d/win32, ~/.profile.d/macosx, etc.
Detects hostname, runs ~/.profile.d/hostname.
Put core stuff in the
The rest, usually doesn't change.
Add it all to CVS on a personal server. Then just checkout to each account you have. cvs update will keep it up to date if you change the master copy. You might need a special
Then, you have the same profile files on all of your machines. Got a new
2. Daily use Documents. This is a mix. Perhaps you could use a separate CVS repository. Or, use rsync and rdiff type backup sync programs. The key here is to keep this to a minimum. How much to you really need, and how much *must* be in sync between all your machines at all times. Again, this is fairly easy for a small number of documents, so don't let it get out of hand. If you don't use the file all the time, and don't need to maintain changes, then push it to archives.
This is the issue that most other posts address, so I won't get into too much detail. All those solutions are much easier with a small number of documents.
3. Archived files. This is probably what you were really asking about with regards to NFS and sharing files. These are the files you need every so often, stuff like your mp3 collection, downloaded software, extended (non category 2) documents, and the like.
For these, it depends on your setup and level of network access (the speed is important too). rsync might work if you need a locally cached copy, but this is much easier if you leave it in one place. Setup a gateway on your home network with IPSec or PPTP. Or, find WebDAV or some internet accessible filesystem you can use (NFS or SMB even, depends on your security needs). Then, connect to the central repository when you need these files.
This can be large, but keep it so that you don't need to synchronize frequently, and preferably only in one direction. You listen to your mp3's, but you don't change them frequently. Same with your downloaded tar/zip files of software you've collected. (Face it, having a single directory with cygwin, mozilla, etc - all the software you have installed at each location - is much easier than finding and downloading them all from their various sites each time.)
Or, for these files, if you really don't need them all the time, leave them on the central server, and scp them when you need them.
--
So, that pretty much covers it. I hope these suggestions are useful. There comes a time where managing it on the fly just gets too cumbersome. (You'll know that time - it usually happens right after you wipe out some vitally important data because you didn't synchronize the files.)
Beyond this, you can always add all kinds of stuff. Some examples: ACAP (a configuration file server, I use it with mulberry, my IMAP client. It lets me set preferences), Kerberos for common authentication, LDAP for an address book or netscape roaming profiles, the list goes on and on.
What would be nice is a set of scripts to help manage this.
Imagine, getting a new account and typing "pullprofile", and having your environment and data all retrieved, pulled from your central server. Then you could have login and logout scripts to synchronize the data, or just manually (possibly remotely if you forgot to sync before you left work) run them. A cron job to synchronize the big data store overnight.
I'll keep dreaming, and keep looking on freshmeat and sourceforge for a project like this. Maybe one day I'll get up the energy to start it myself, but don't count on it.
;-)
~Jonathan
Score: -1, WHORE.
Take your fucking spam somewhere else, buttcunt!
Not sure how far along this is, but does samba have the File Replication Service working? It is what windows domain controllers use to keep certain things in sync. Mainly used for the NETLOGON share in pre-Active Directory and SYSVOL in Active Directory. This is where things like login scripts are kept. You could setup a file server in each location, and run FRS to keep things in sync. Dunno though.
Politics, Life, and More on my Aspiring for the Future
sorry for the anon coward. I reg'ed but didn't wanna check my e-mail from the place I'am at for the password (have to make it go live when I get home). But was reading the thread and was thinking, you might wanna try what I been doing.. Set the linux up to have ftp as a service, and since most os's have some form of ftp, use that, but secure it down by setting up proxy'ing rules via ipchains so that only inhouse numbers or specific inhouse ip numbers can even get access to the site at all, then of course place in various accounts. I suppose it's prolly not the best way to do it, but it's fairly, quick, simple, and somewhat secure. Though if the data your transfering must absolutely remain secure and not sniffed, I'd suggest sticking with ssh. Just my thoughts anyhow. Hope it helps
-- StakFallT
Use AFS and kerberos. Works for mit.edu, Ericsson, kth.se and MANY others so it should work for you too.
http://www.openafs.org
http://www.pdc.kth.se/heimdal
This is not an ad.
-- ac at work
The real Seth Finkelstein has slashdot uid #90154
The name is also a subtle misspelling
My name is Seth Finkelstein, the troll is using the name is Seth Finkelstien
I did not post the above message in this thread. People may disagree with what I say, but I don't do totally off-topic posts such as the above.
I just tried this program, does it ever work well! It will even copy programs across the network as well, thanks! :D
for the server part of the answer - take e-smith (www.e-smith.org). it sets up in 15 minutes, has appletalk, samba and nfs, has all sorts of VPN included, mailserver even and a lot of ways to connect to the net - installation and administration (web-interface, but it's based on redhat, so administer it like you like) is such a breeze that i sometimes don't believe it myself.. the most underestimated linux distro ever..
PAT
SEO Test: TIGI und SEBASTIAN - Online Shop - V
Not necessarily. I think a better arrangement is to read your mail on a single Linux box using any of the excellent text-based mail clients (pine, Emacs's vm, etc.). To access your mail from other computers, connect securely via SSH to that Linux box. It's simple.
Advantages:
Disadvantages:
I've had this arrangement for years and love it.
My mom's office had the same types of problems so here's what I did:
:)
1. Set up samba on the reliable (linux) machine, with proper tape backup, etc.
2. Firewalled the segment (which included their desktops) with a WatchGuard SOHO router (about $500 for 25 user support, runs linux
3. Set up Mobile User VPN on the firewall, and any laptops that might travel out of the office.
Samba and SMB are not the world's fastest solutions, but it is nice to be able to have the directory browsing in winders and macos. Samba is easy to set up, my first install of a samba PDC only taking about 3-4 hours (and never touch it again). If you need real speed for transferring over large files, you can always use SSH and SCP (putty and pscp for windows, niftytelnet for mac). Just always attempt to maintain a central data server, back it up as needed, and you'll be successful in clearing the data clutter.
AntiChristX
Daring to remain below 5 karma indefinitely
This is what we do, and it seems to work very well.
We have an OS X Server machine. It provides home directories and a few other shared directories. The Linux and other Unix system(s) NFS mount the directories. The Windows systems connect to the shared and user directories via Samba. The Apple clients ( OS X and OS 9.2 ) of course just work.
There were a few growing pains as we learned to set up Netinfo and share Windows printers and work around various small issues, but it's all working very, very well now. No matter where I log in, things look pretty much the same.
As far as remote access, I don't do it ( I live 7 blocks from work ), but there are ways via ssh to tunnel yourself in and get an IP address and file services, I have a buddy who does that from time to time, it works great. What else do you need? Forget copying data from system to system, just set up an OS X file server and keep it all in one place... and use a good SQL database like PostgreSQL for stuff that belongs in a database, of course!
I prefer keeping all documents and such on a linux server with a dedicated internet connection and using ssh/scp and zip for updating files and directory trees.
But more dynamic data such as email, address books, bookmarks, etc. is another story. As already pointed out, IMAP is the way to go on email. I use UW-IMAP w/ SSL because it setup easily under debian and integrates well with shell mail. However, it seems like Cyrus or Courier might have been a more robust solution. Comments welcome.
I'd like to setup ldap access for use with mozilla for my bookmarks and address books. I know there's a HOWTO, but it seems far from straightforward. Again comments welcome.
This response is dead on. The original asker needs a file server that speaks multiple protocols. Once you have a server, it is much easier to create the necessary ssh or ssl tunnels that you need for total security.
Trying to maintain coherency of data via replication across multiple machines is begging for trouble -- this is a hard problem that to my knowledge has not been solved in a clean, cheap way.
If you want to use NetInfo for Mac OS X, create a new port from the Open Darwin sources. There's a port of an old NetInfo server module for Linux floating around, but it's not what I'd call up to date.
A better choice would be to use OpenLDAP, as Mac OS X is designed to pull directory service info from an LDAP data source. Windows systems can also pull from a LDAP, as can Linux and *BSD and Solaris and so on.
--Paul
All you need is to buy a laptop.
thi
i've been looking for a while, but haven't been able to find one. can someone point me to som documentation on how to install/configure openafs for linux?
-- john
I have a Solaris system with an inexpensive disk
array attached. I use SAMBA and NFS to share
the disk space to my FreeBSD, OpenBSD, Win2K,
Win98SE, x86 Linux and now PS2 Linux systems.
I don't currently own a Mac, but I believe OS X
has a native NFS client...
check out the openafs website - http://www.openafs.org - follow the documents link, and go for quickinstall for unix. use your head and go step by step.
Good luck. OpenAFS rocks - it's stable, performs well over a lan, has some excellent ACL coolness (the reason I chose it for our implementation) and I generally think it is the Mercedes of file systems.
--Martijn
Stuff at home that should never be down goes on the Linux Samba server, my wife's pictures go on the dual boot Linux/Win98 workstation. My son's games go on the Win2K machine.
At work I put all my documents on the Win2K server except for stuff I'm working on in solid modelling, 100Mb isn't fast enough then.
I use VPND to join the two networks, so Samba, ftp, etc all work over the VPN. VPND is nice because it's a router-to-router VPN and I don't have to worry about configuring client computers to use it.
For backups, at work I don't have to worry, IS backs up my documents if I put them on the server. At home I just dump my data to a CD.
Sig is on vacation
My supervisor swears by one of these things... He used to have a complete mess of redundant files all over the place and could never remember which was the most current. Now it's easy. The VST drive is the definitive version.
Of course, there is an outside chance that you could lose the drive or the data be destroyed, so make a habit of backing up (using rsync or something similar) on a weekly, or even nightly, basis to a more secure machine (a desktop, for example). You could probably set up a nightly cron job to run that would check to see if the drive is connected and backup if it is. That way, backups for you would be as simple as connecting the drive when you get home...
Your best approach is to use Samba with Dave/Xsamba/Sharity or the SMB client in MacOS X 10.1 (?) on the Mac. Dave is for old, pre-MacOS X MacOS-es. Smbclient/smbmount on the Unix machines (which seems kind of slow). Or you could install OpenAFS which is a distributed filesystem with caching, crypto authentification (something like Kerberos), disconnected operation, etc. The sad part is that it wasn't ported on any of the BSDs yet, except Darwin/MacOS X. They are working on ports though and the server has been ported to FreeBSD and OpenBSD and the cache manager is in the works I believe. It works fine on Linux, Windows NT/2k, Solaris and a buch of other platforms.
I would recommend using a Snap Server http://www.snapserver.com. From their web page: The Snap Server 1100/80 GB and the Snap Server 1100/40 GB both install in minutes, automatically recognize and support all major network clients and platforms, and require very little administration or support. I have one and it works well.
Google Search Appliance: These guys are fucking gods.
OK, now while Apple have recently decided to open source their Open Directory system, the current state it is in is a little misleading.
What you want to do is to setup Open LDAP and do your authentication via that. This is what Apple are moving towards with Open Directory, NetInfo will be relegated to just another plugin, mainly for local configuration info, and as an interface to the LDAP store.
Anyway, I think this is somewhat off track anyway, as all the poster of this question wanted was a file server, not a cross-platform authentication mechanism. If they're only dealing with one or two user accounts, then they may as well just do it all via file sharing, and not bother having centralised user accounts.
i don't read slashdot anymore.
How about looking at Freenet? To Quote the site, "Freenet is a large-scale peer-to-peer network which pools the power of member computers around the world to create a massive virtual information store open to anyone to freely publish or view information of all kinds." It appears to be available for Mac X Windows, and Linux.
Keep It Crunchy TTFN Yafiyogi
I'm still settling on my setup, but on my trusty P200 linux box I run samba and have 2 hard drives; one which gets accessed and the other that gets a copy of its brother every night. It's a great feeling knowing you're backed up.
I considered RAID, but remember that RAID doesn't help you if you accidently delete things.
Lastly, for storing pictures and text that I want access to, I run a Twiki server. It's http visible, searchable, fun, versions everything for you, and is accessible anywhere. I love it.
--matt
Does it hurt to hear them lying? Was this the only world you had?
Well, let's say your working on a unix system and it crashes or loses its configuration and the network underneath it gets reconfigured. I find the best solution for moving data and preserving permissions is a tar pipe through ssh or rsh. Cpio and other stuff might work better since tar has problems for deep directories. But here's what I use:
But AFS looks cool. Does anyone know how secure it is?
There are many approaches to this problem. First you can use samba on all machines, not the greatest though. ftp or scp and the like. You could use rsync or cvs. I don't have any mac os machines but what I do is have a central server where I *try* to store data, and then use it as a point to transfer stuff to and from. I run samba as well as nfs, so it works out nicely. yymv.
This is what Coda does. Coda supports disconnected operation, resyncronization, and distributed servers.
With coda, when you plug in your laptop (or whatever), you work in online mode, with a local cache. When you pull the (network) plug, you keep the local cache but make changes offline. When you next come online, you resync.
Now, if only it were stable enough... Coda (and all of Odyssey) could really use some nice testing and hacking. And probably porting.
Whether or not you use this much separate hardware, I strongly suggest you plan for the day when you don't work for your current employer(s). Draw clear lines between your data and the company's data. If you're going to keep personal data at work, make sure you label it in a way that will make it clear to someone else.
for help with the OSX thing:
d ir ectory/
http://developer.apple.com/darwin/projects/open
"Apple's Open Directory Server is NetInfo. NetInfo is included as part of Mac OS X Desktop and Server, and is also included with Darwin.
so this would mean you can run darwin instead of linux and have everything you need. now if it is open source and portable to linux, you'll have to do some research for that.
Kenny Sabarese
www.kennysabarese.com
XML was invented to meet cross-platform needs, but developers still AREN'T USING IT ENOUGH. OpenOffice is an example of good XML use. And you can transfer documents between Linux/Windows/Mac versions without problems.
But these things should also apply to address book programs, scheduling software, e-mail programs.. so that you can take your data wherever you want, and it have it run on any machine first time.
mogorific carpentry experiments
SFS (the Self-Certifying Filesystem, http://www.fs.net) is like NFS, but encrypted and more secure. I use it to mount /home from a server on another computer, and it seems to work (home directories in a SFS share are a little unreliable, though, so I'm looking for an alternative). I'll probably stick with it for my "files" (music, archived downloads, installers, etc.) directory though.
After spending several days on trying to get AFS working on OS X, countless emails back and forth on the AFS users list, I never managed to get the server working.
The Windows installer also barfed, hanging when it tried to start the server.
This after I meticulously followed IBM's extremely long "quick" start instructions.
While I haven't got the whole setup tweaked exactly the way I want it yet, it works pretty nicely for now. With this setup, I can work on stuff under one OS, save it, then continue work under another. I also have a tape drive on the fileserver for backup. I still need to get a setup (coda maybe) to do disconnected work on the iBook. I also don't serve anything through the firewall, but I have the fileserver setup with an MTA for the internal network for the OS's that don't automatically run their own (Mac OS, Windows).
As for something that might help you keep things synchronized, I know of something related that will at least help if you ever have to do a reinstall. It's called Gutinteg and was very widely used for reinstalling a machine from scratch with both Windows 95 and Linux dual boot at the NMT Computer Center.
Nathan's blog
The biggest problem I have with this is the scattered data from Windows apps. They all seem to want to store stuff deep under C:/Program_Files/[app_name]/
You have to hunt it down. Then sometimes the app can only use data from that dir, so moving it to a data partition breaks functionality.
I don't want to back up C:, I can rebuild that.
I want to group my data under a partition so I know what needs to be backed up.
Where I work, there are Gigs of data stored
in massive Oracle SQL databases.
Obviously, if you are asking this question,
you don't need such a high powered system
as this (we have a big-iron Sun machine that
does the serving).
However, buying a powerful Dell Server, and
running Access on Win2K would give you a
decent SQL system to work with.
Applets can be written for any platform
which will all use SQL and can then translate
the results of the query into native stuff
for the computer its on.
Furthermore, look into Macromedia ColdFusion.
CF can be used to quickly create web-based
systems which interface with an SQL database
rediculously easy. (My department does just
this.)
You can use a web app and a database to
retrieve data and upload data, perform
authentication, all sorts of great stuff.
Personally, the way I do it now is to use ALOT of file shares or to access the data via SSH.
I was thinking that it would be nice to do it with something else. What, I haven't run accross yet, but something else.
My idea is like the P2P file sharing works, execpt more on a personal level.
Mirriroing data is a pain, and poseses problems.
File shares is nice, but you have to remember what system you put what on.
Sometimes it is nice to SSH in and grab what you need.
What the project would include would be this. A daemon which would run on every machine which you want data shared from. and a client which accesses this data. Somehow the client is smart enough to create a 'virtual disk' or something which allows you to access the data in any program under any OS - or at least popular ones.
This solves the problem of mirroring, space is cheap but the more stuff you have, and the more duplicates of stuff you have, it becomes difficult to organize. This would allow you to provide organization which is platform independant, and avaiablity. It dosen't matter where it is stored on a network, or the internet, it is a layer higher than that. Of course you don't wnat other people accessing your data (or at least the important stuff, perhaps maybe MP3s and porn and things of that nature) so an anonymous login if you choose may be neat.
Technically I don't know how to do it, but I would imange it is possible. Making virtual disks is already done - on multipule platforms. Daemon Tools, VCD, etc.
File sharing, organization over a network is done, samba, NFS, etc.
The only thing would be to roll it all into one nice easy to use client/deamon.
Ideas, comments, is this good, is it done, or am I crazy?
If you have enough disks to make 600GB with the 100% overhead of RAID1, I hate to think howmuch space you're wasting that you'd have free if you used something smarter like RAID5.
RAID5's overhead is a fraction of RAID1's overhead, and as long as you don't have a lot of drives fail at once (which is rare, and RAID isn't a replacement for backups anyways), you're much better doing anyways.
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
The most wonderful thing about a nice Apache server on a *nix based platform is how it treats files. It won't alter your file information or harm your contents. With Apache 2, you have webdav, in addition to basically anything else you need like the ability to serve on a port other than 80, or nfs, samba, ftp. Plenty of network permission on everything. Perfect for internet, intranet deployment. I keep saying it, eventually somenone's going to listen.
"You never want a serious crisis to go to waste." - Rahm Emanuel
My solution is fairly cool...
;)
/users/name )
:)
;)
I have my main files in my home folder on my laptop...
So I have my portable environment...
When I want to use another machine I boot the laptop in target disk mode and connect it to the machine via firewire...
All the machines I use are set so my home directory on them, is actually on my laptop
ie (/vol/fwdrive/users/name instead of the more normal
So I carry my data around... but use whatever machine I want...
You could do the same with an iPod.
Problem of course is that it probably only works with Mac OS X... oh well
Its really very cool tho
---
Live Long & Prosper \\//_
CYA STUX =`B^) 'da Captain,
Jedi & Last *-fytr
One common thread I see among the comments posted already is the notion that all you need for a backup is the ability to access your files if something goes wrong on one computer/disk.
I don't know how many times (at work) I've been told "Why do you have to use all of these tapes when you already have the hard disks mirrored?" This line of questioning usually lasts until someone accidently deletes a file. Or makes a change they didn't want to. Or a file gets corrupted
With backups, there is also the need for some form of change history of the stored files (not just CVS-style, but for all sorts of binary data too).
If you simply maintain a synchronized storage of your files/data, you have only covered one of the backup scenarios - breakdowns (or fires, theft etc...) but that is FAR from all you will ever need.
However, for the home user, maintaining a proper backup procedure is probably just a little ridiculous.
None of us is in control of the technology we use.
I've long since given up reading the hardware specs for the processors I'm using and expecting to understand every wire on the circuit board and every byte of code in the PROM. (Yes, I used to do this.) It's just all too complicated, and one does wish to have some time left to use the stuff.
It all got too much for me when processors started caching stuff internally, so you could no longer see what they were doing by watching the data fetches with a logic analyser; it was at this point that you could no longer calculate how long a processor would take to do something, because the same instruction might take a different number of cycles depending on cache history; you had to just run the code several times and measure it.
So the fact that I don't have a copy of several million lines of source code that I have no desire at all to spend time reading doesn't bother me in the slightest.
I don't have the time to read every single post right now, but since I didn't see anyone write about my idea, after looking at quite a few posts, I'll tell you what I think. . . My idea is, I believe, an ideal solution, but it requires OSs to be much more multi-user friendly. If we're talking about spreading usage over the 'Net, then Internet speed and security (interfaces) must also improve. The problem: We have multiple boxes running different OSs. There's a pretty easy solution if the boxes are all in your LAN. Why not run a central server with a large number of drives? You could then simply use the different boxes as dumb terminals. For this idea to work, one has to assume that the particular OS being used allows for multiple logins at any given time. . . Have different volumes dedicated to one particular OS. Net-boot your machines with that OS. . . log-in with one account if that's your desire, or log-in with multiple accounts at once if you wish to work on multiple machines at one time. If we're using the Internet, the additional complications are, of course, speed and security. Once the Internet gets speedy enough, and once there are easy to use, graphical security tools akin to SSH, my vision for distributed work flows should be very possible.
Part of the hardcore faithful who believed in Apple long before it was cool again to do so
Machines:
Dev box (PC - W2K/Linux)
Server (PC - Linux)
Firewall (PC - Linux)
Laptop (PC - W2K/Linux)
GF's machine (old Mac)
PDA (Psion 5)
All my working data lives on the server and is available to the other machines via Samba, NFS or netatalk. Backup via DDS3 on the server using afbackup, "minimal restore information" encrypted and mailed to a free webmail account so I can get to it if, say, the server catches fire.
Laptop has a directory under my ~ called "mirrored" which contains my current working set of stuff from the server. This is synced using unison whenever I come back from / head off on a trip to the office (I work from home 3 days a week).
GF has a home dir on the server which is visible on the mac desktop and has been told "put stuff here, it gets backed up, put stuff anywhere else, it's your problem."
Dev box and laptop are dual-boot linux/W2K, with a VMWare install running inside linux set up to boot from the physical W2K install, which can see both ~ on the host machine and on the server (if connected).
PDA syncs with Outlook (no email, just calendar and tasks) on the dev box - VMWare or real hardware, works just the same and the data is visible in both as it's the same physical drive.
Everything works very smoothly, except for:
- Unison which dies if it tries to use more than 64M of RAM to do the sync. This has only happened to me once, when trying to sync about 40-50,000 files in one go. For normal day-to-day jobs, I've never had a problem with it.
- The W2K VMWare session on the dev box losing the serial port occasionally, which means I need to reinstall the port or boot into native W2K before I can sync the PDA. Not really a problem as it only happens very occasionally.
What would Lemmy do?
This is something i have extensive experience with. We have a bunch of servers all over the world, some of them on slow and unreliable links.
The short answer is that rsync out of cron works wonderfully. We sync up all the servers every day with rsync over ssh, so it's all very secure, and rsync only syncs the deltas of the file that actually changed. So if 100 bytes of a 100MB file changed it won't retrieve the whole 100MB file again, only the parts that changed (in reality it will sync a little more than that, but you get the idea). I someomes even use rsync for local copies and over fast links. The speedup due to only syncing the deltas are well worth it. Based on my experience make one server the "master" server. Let the master server log into easch machine periodically and sync the deltas. Then each machine should log into the master and get all the changes from all the other machines.
OS X does not have DAV over SSL natively. See my article over on Mac OSX Hints for how to set this up with stunnel.
Heh, I actually remember your posts to the OpenAFS lists. You're right, the server software doesn't quite work on OS X, and the Windows version is kinda dodgy. But, you only need the client software to actually access AFS-space, which works fine on both OS X and Windows. Put the server software on a couple of UNIX machines, and access the filespace from any OpenAFS-supported platform.
This is not a Fugazi
your a true slashdot moron
you're a moron.
HTH. HAND.
1) keep everything on one server
2) make use of GNU screen package to access over terminals
3) make use of VNC by AT&T -
http://www.uk.research.att.com/vnc/
That was more of a joke than a troll. I would say that it is a contender for
BEST. JOKE. EVER.
Karma: Good (despite my invention of the Karma: sig)
You got my hopes up. :-( No big deal, though, as it's not really a big issue, until, someone has the nerve to do that with every consecutive thread.
If you're religishitty, KILL YOURSELF!
I use a Linux server with two features that allows me to access my files as SMB mounts over the Internet. 1. VPN - MS PPTP This is a must because you don't want everyone to see your data through the IP packets. On the Linux SMB server, I run PopTop (pptpd). You can get the software at www.poptop.org. You can log in from Windows, Mac and Linux machines. I'm pretty sure that FreeBSD, Solaris and other UNIX' should have support. Then, its just a matter of running Samba to provide SMB file services. On Windows machines, you can just map the drive or through network neighborhood and get to the data. Double click on the file and you are using it. None of this upload and download, sync and other xxxx things. If you are sharing a Word document for example, Windows also takes care of locking it so others knows they should not write to the file. An added bonus is that you can set up IP Forwarding on the Linux VPN server and all traffic from your client cannot be observed. This protects your privacy if your are doing things like web browsing, webmail, IM, etc. This is not to imply you are doing anything bad, it just protects your privacy, say if you are consulting for 2 companies and reading webmail from one while working at the other's site. Make sure you set up pptpd to use encryption for both passwords and traffic. 2. VPD - A product from my company. This is optional and probably is needed only if your SMB server is not hosted on your premise. Since I have my server at a data center and it is backed up by them, they can see my data. Therefore, I need a superuser independent encryption mechanism. At the same time, I don't want to explicitly encrypt and decrypt each time I use a file. I'm not trying to sell anything. I just want to caution people about data on the Net. If someone else can get to your server, you need to encrypt your data. I use a product from my own company, VPDisk Pro. This software provides on-the-fly encryption. I just start smbd and nmbd with the proper keys each time I reboot my machine. Alternatively, you can automate this step, but it is less secure. Combining this with VPN, you get seamless secure SMB/CIFS file access over the Internet. Both the IP traffic and data on disk are encrypted.