Home Directory In CVS

shamir_k writes "Joey Hess has come up with an innovative solution to a problem we have all faced. He's put his whole home directory in CVS. Not only can he move between multiple computers easily, he also has automatic distributed backups."

Comments

Here's a piece of advice

[Rosco P. Coltrane]

Don't forget the -kb switch when you do "cvs add pr0n.avi", otherwise you'll be disappointed when check the file out again.

Re:Here's a piece of advice

[Anonymous Coward]

CVS will also be a great help when you download pr0n that have the same file names.

like: untitled.bmp and untitled.bmp - even though they're two different pictures! It'll save the time to rename whole directories!

And, you can hide all your pr0n from prying eyes too! OOooooo Ahhhhhh!

Unless, you've named your tree - jerkoff_material.

CVS takes files?

[cuppm]

Man, I have a hard enough time trying to get them to accept my insurance card for prescriptions. Hats off to him for getting them to take his files...

My favorite directory to put in CVS

[Beryllium Sphere(tm)]

/etc

Have a checkin comment for why a configuration change got made. Be able to roll back a failed experiment reliably. Find out when a change happened.

Re:My favorite directory to put in CVS

[Rosco P. Coltrane]

My favorite file to put in CVS : /dev/hda1. Takes a while to recover an old file though ...

Re:My favorite directory to put in CVS

[Espectr0]

/etc

Or you can just use Gentoo, which does this automatically when you update your system, pointing out the location and files that are different, all with diff output, and the ability to merge the changes, overwrite or ignore the change.

Re:My favorite directory to put in CVS

[inferno0069]

Where do you think Gentoo got this from?

Debian.

I wonder.....

[The One KEA]

Has anyone tried this with BitKeeper?

Re:I wonder.....

[Kourino]

You wouldn't do this in Bitkeeper if you were a privacy freak. Remember, the Bitkeeper liscense requires that you maintain open logging. Realistically, this means that info about what files you change get transmitted across the network; I don't know if it's encrypted or not. It's not that big of a deal, but I'm sure someone here would care.

That being said, doing it in BK would be a compelling alternative if you wanted to use the same /home repo across (say) three or more machines, since you could take advantage of its more complex merge operators. Arch or SVN also might be good ideas. (Don't have any experience with Subversion, though.)

Why just home?

[AuMatar]

I asked this on a local linux mailing group recently- what do people think about the idea of a version control file system? Disk space is cheap these days, we can afford it space wise. Think of all the problems it would solve.

*Made a mistake in your config file? Revert it
*User deleted the file? Revert it
*Want to see why you made a change to any given file? Check the comments (commenting would be optional, of course)
*Your system was exploited? Revert the entire system to before the exploit
*Upgraded an app and regret it? Revert the files

And so on. I'm not sure if CVS would be the best method (I'm not a SCM specialist), but I'd see this as an extremely useful feature who's time has come.

Re:Why just home?

[PhilipPeake]

Ever hear of VMS ?

It had a filestore with file versioning - about 30 years ago.

Re:Why just home?

[Jellybob]

I'll get modded down to oblivion for mentioning an MS product in a positive light, but Windows XP+2003 Server supports this already.

Users can rollback to previous revisions of files that they've saved to the 2k3 server, saving the sysadmins the time of restoring *another* accidently deleted file from the backup tapes.

Re:Why just home?

[pclminion]

The basic problem with versioning in the file system is deciding when to commit. Do you commit for every byte written? Of course not, that would be a massive waste of time and disk space. Do you commit once for every write() call? That suffers the same problem, because sometimes writes are very small, and you can never predict in advance how large a write will be.

You can't automatically commit after every xxx bytes of data written, because that means there will be a bunch of intermediate states on disk, most of which are probably bogus (this argument applies to the above two options as well).

The only thing that even remotely makes sense is to commit on close(), but that doesn't make sense for applications like text editors which keep the file open for the entire duration. You want each SAVE to be a commit, not each CLOSE.

What this boils down to is that there must be application level support for the commit operation (i.e., a new commit() syscall). The application has to specifically be coded to tell the operating system "Ok, I'm done with the revision, commit this now." And that means every application on the system will have to be tweaked to make the necessary commit calls, not to mention the thought that goes into deciding WHEN it is appropriate to commit. I would wager that a lot of maintainers wouldn't bother to make those changes, and as a result those applications wouldn't support versioning.

I agree that a versioned file system would kick ass, and there are even some out there already (Google for "versioning filesystems"). But they tend to be special purpose. I don't see how it could be cleanly and transparently integrated into a general purpose system such as Linux.

Note that I didn't bother to check CiteSeer to see if there is academic work on this before posting this comment. So if anyone knows of any work toward that end, I'm sure a lot of us would appreciate a pointer to it.

Re:Why just home?

[AuMatar]

To make this more legible- a text editor, for example, does not have the file open the entire time waiting for input. It opens the file, reads it, then closes it at startup. When the user hits save (through keyboard commands, mouse click, whatever), the editor opens the file again, this time in write mode, writes the data, and closes it. By this model, close could be the commit function, and open could be the checkout function.

Re:Why just home?

[bug-eyed monster]

It's been quite a while since I used VMS... IIRC the problems we had we the VMS versioning:

- It created a new version every time you saved, so just going through a few change/compile/fix cycles (for example) would create lots of versions clogging up the disk.

- The old versions were in the same place as the latest version, and if you wanted to delete a file, you'd just say "delete blah.blah.*" to wipe out all versions (and therefore all traces of the file)... then say "oops!"

It was useful in many cases, but in a different way from CVS. A very useful solution would be to have file-level journaling with the ability to throw in comments and create tags and branches.

Re:Why just home?

[bcrowell]

The basic problem with versioning in the file system is deciding when to commit.
I use Unison for this, and it works great. A commit happens whenever I choose to run Unison. I'm a human. Unlike a computer, I understand what I'm doing and why. I know when I'm done working on something, and when I'm ready to synchronize my files.

From the article:

• I get three major benefits from keeping my whole home directory in CVS: home directory replication, history and distributed backups.

I get all these benefits from Unison. Admittedly, the history only gives me snapshots at two different times (time 1=now, time 2=last time I committed), but that's always been good enough for me. Deleted the wrong file? No problem -- get it back from the other machine.

Unison is also cool because, unlike CVS, it (a) is easy to set up and maintain, (b) is quick and easy to run, and (c) works just as easily with binary files as with text files.

Re: an innovative solution

[Anonymous Coward]

... quick, patent it!

Re:I keep my life in a CVS repository

[Rosco P. Coltrane]

No. But you can do this right now:

cvs commit suicide

Sourceforge

[Smitty825]

I'll put my home directory on Sourceforge! Everyone can now help me maintain it!

Re:Sourceforge

[AvantLegion]

I patched your political science essay.

I fixed all the anti-Bush bugs.

How does he handle renaming and erasing in CVS?

[Anonymous Coward]

'cause CVS loses the history in these operations.

Re:How does he handle renaming and erasing in CVS?

[neonleonb]

Alias mv to execute some shell script:
mv $1 $2
cvs remove $1
cvs add $2
cvs commit -m "move $1 $2" $1 $2
You might want to add some script to deal with the case where $2 is a directory, but that shouldn't be too hard.

That way, information about the move and the previous file status is kept.

CVS, eh?

[Kourino]

I wonder why CVS, and not something more advanced, like Arch or Subversion? Especially since he outright complains about common limitations in CVS, like moving files and dealing with directories at all. If he's hoping, as he says, "for a better replacement some day", why not see what the present has to offer?

I mean, that's not to say that alternative systems are perfect, either. I'm going through the process of learning arch now. There's a learning curve, but not nearly as big as it's made out to be. Still, using something else (almost anything else) would probably help on things like the merging issues, especially since he mentions that sometimes it's a pain keeping things in sync between three of his machines.

Re:CVS, eh?

[self assembled struc]

well,

have you every tried to install and set up subversion?

plus, after years and years of development, it's still point-oh-something and specifically states they can only guarentee that your repository can be read with x number of versions after the version that created it.

as for arch, i've been looking into it, but it's still a hack onto cvs.

subversion gets some of it right, but at my old company we used perforce and honestly, i'm spoiled and i've been trying to find a free (just as in beer, free as in speech would be nice, but i just want to get my work done quickly and without hassle) source control client that can mimic the functionality and stability of perforce, and i've yet to find one.

too bad my new place won't shell out the $650/seat for the license...

Re:CVS, eh?

[vslashg]

I agree that Perforce is awesome. But here's something you may not know -- Perforce is completely free for the kind of thing this article is talking about doing.

Specifically, Perforce is available for download here. Without a license, it only supports two users and two clientspecs... not enough to manage a project shared among developers, but wonderful for managing your code in home projects. More good news: Perforce is free-as-in-beer for use in developing open source projects.

(This isn't meant as a holy war... I know that many of you might think that source control package Foo is better than Perforce. You may be right. I'm just pointing out some Perforce licensing facts for those who like Perforce.)

Re:CVS, eh?

[aardvarkjoe]

have you every tried to install and set up subversion?

Subversion is pretty easy to install and set up as long as you don't use the Apache interface. It's pretty much just a matter of downloading the subversion and libdb archives, and doing "./configure; make; make install". I've never had any problems with it, and it is rapidly approaching version 1.0.

One of the main things that they ought to make clear is that the http interface is a nightmare to set up -- with the exception of making publically-accessible repositories, using the svnserve method over ssh is a far better choice.

Re:Described in Linux Journal months ago

[kajoob]

I like ragging the editors as well, but come on, the blurb said 'innovative' and that is all. There's nothing about "new" mentioned anywhere in there, you pulled that out of your ass just so you could complain. Now quitcherbitchin and just read the kewlness that is home directies in cvs. ;)

auto-backups the plan9 way

[DrSkwid]

Plan9 has the mantra : "file creation is forever"

Automated incremental backups are a way of life.

With Venti one can even back up two windows/linux machines and *not* use up disk space for commonly used blocks, so backing up 100 machines wont use up the usual 1Gb each for the duplicate libs/windows directories.

The yesterday command give you the power to browse back through your life

Find what has changed in the C library since March 1:

yesterday -d -0301 /sys/src/libc/port/*.c

When did you say this guy did the innovation again?

Re:auto-backups the plan9 way

[Rosco P. Coltrane]

Right, so on plan9, you can recover all your previous file versions, for example, letter_to_boss.txt :

v1.0: "You stupid fuck, why don't you give me a raise some day?"

v1.1: "You tight-pocketed capitalist, isn't it high time you gave me a raise?"

v1.2: "Hey Boss, I really think I deserve a raise, I've been working on this project for so long."

v1.3: "Hello Boss, I respectfully request that you should consider giving me a raise, as I think have proven to be a reliable, hard-working employee. Please?"

v1.4-FINAL: "Dear Mr. Schmoe, I wish to apologize in advance for stopping work on our most important project for five minutes, but I would like to present an idea to you : you see, ever since I have started working at 6-pack computing, I've tried to be a model employee and ... ... ..."

Oh I like that

[The Munger]

That's a smart way of doing things. I've found version control to be worthwhile on even single user projects. Having the same kind of backup/restore and history tracking on every one of your files just makes sense. I'm suprised no-one has done this sooner.

In a slightly more abstract sense, it provides a 'working set' of documents on your computer. Comments on your version history adds meta-data to files that is time-based. Most systems at the moment add meta-data that is for the current file. Imagine, you check in some files with a comment like 'Project: holiday snaps 03'. Then later on, you use one of those files in a presentation 'Project: report for Bill 03'. With standardised formatting of such tags, the file keeps with it the idea that it has been used with multiple projects at different times. That's a powerful method of grouping.

Ever been in the situation where a file belongs with multiple projects? With your standard directory structures, you might put it in one directory and shortcut/alias/whatever it in the other (or maybe make a second copy). It's pretty ugly right? What if you could say, this file belongs to both of these projects, and you could even provide the old version that was used in another project. OK, so all of that would require some more automation - we can dream can't we?

That has a lot of possibilities.

Great for Mozilla settings!

[adamfranco]

I currently store parts of my .thunderbird and .phoenix directories in CVS and do the commit/update to sync work and home. In general it works pretty well, though not all my settings translate well between OS X (work) and Red Hat (home). For this reason in particular, extensions are not in my CVS and this makes keeping stuff custumized a bit of a pain still.

Pun Intended...

[Vaulter]

"Hold on a minute, I've got to check out this porn."

Re:Not Innovative

[TheZax]

What's next? Keeping system configuration in CVS?

Slow down, I can only patent so quickly...

Note to self:

1. Patent home dir in CVS

2. Patent system configs in CVS

3. Patent pr0n in CVS (note -kb)

unison is probably a better solution

[penguin7of9]

My recommendation: use the Unison file synchronizer to keep multiple copies of your home directory in sync.

It's not quite the same thing as CVS, but that's probably a good thing. Most importantly, it won't give you versioning. On the other hand, it is symmetric (meaning, none of the copies are distinguished) and it is much less hassle to use. Also, you can define custom merge methods to automate merging of things like mailboxes. Unison is great for keeping a home directory (or portions of it) in sync between different desktops, and between desktops and laptops.

Note that for live backups, rsync is probably still the best choice because you want something unidirectional.

Renaming a directory

[Henry Stern]

I shudder to think what happens when he tries to rename his files and directories.

Definitely the biggest problem with CVS. :(

Re:Or just RCS

[pHDNgell]

I have a script that does all of this for me:

http://bleu.west.spy.net/~dustin/soft/filemonitor

You point it at a dir and run it from cron nightly. It also gives you a handy nightly mail telling you what changed. Excellent for those late night changes to systems where you don't remember what you did...or if someone else made some late night changes that you'd like to undo.

Re:ssh private keys

[dedave]

This isn't insightful. He specifically states in the article that he has a .hide directory that _doesn't_ get "sown like seeds across a number of systems" just for this very reason. Bah.

kinda old article; update

[joey]

I'm not sure why something that I wrote in 2001 and that appeared in print media in 2002 is news.
This is the second time I've been slashdotted for something over 1 year old this year. Previously it was the pkg-comp page, which I wrote circa 1998.
Kinda makes you wonder.

Anyway..

I suppose I should mention that these days I keep most of my home directory in subversion. I have not gotten around to writing a successor to this article yet, but it works even better than cvs, and that's probably the most common question people ask me about this article these days.

Re:Described in Linux Journal months ago

[hondo77]

There's nothing about "new" mentioned anywhere in there, you pulled that out of your ass just so you could complain.

Merriam-Webster defines news as "a report of recent events". If Slashdot is "News for Nerds" then an article from over a year ago ain't news.

The comments say he uses subversion now

[Polo]

Did you see his followup reply at the bottom??

Re: CVS homedir (Score: 0)
by Anonymous on Tuesday, November 11, 2003

I suppose I should mention that these days I keep most of my home directory in subversion. I have not gotten around to writing a successor to this article yet, but it works even better than cvs, and that's probably the most common question people ask me about this article these days.

-- Joey Hess

Been doing that a *long* time

[Anonymous Coward]

I've been keeping my home directory in CVS since 1996. Prior to that, I kept it in RCS, since around 1985, and briefly in SCCS.

I haven't yet decided to go to Subversion, in part because I have a patched version of CVS that allows me to check into multiple CVS repositories - i.e. I can check in when on my laptop on an airplane, disconnected from the net, and can later check in to my main repository when I get connected. (Yeah, yeah, BitKeeper is a way to do that.)

As Joey's article discusses, there are minor issues with CVS'ing your home directory: use of modules, etc. I divide it into stuff that is owned by the company that I am currently working for, and stuff that I own.

When I get an account on a new machine, one of the first things that I do is create a new branch (or, a new version on some existing branch) to hold the dot files and other files
that were pre-installed in my home directory. Having saved them, I then blithely overwrite them with my standard home directory, and maybe do a quick check to see if there are any special features worth propagating to my standard home directory.

Oh, yeah: a bit of footwork to checkout
onto your home directory:

cd ~
mv * .* /tmp/glew/old-home-directory // actually, usually saved already
cvs -d MYSERVER co glew-home
mv glew-home/{*,.*} .
rmdir glew-home

I often find myself pasting together
modules from different repositories.
Sometimes I *want* a "cvs update"
in the root, e.g. in ~, to traverse
repository boundaries - no problem.
But sometimes I don't - e.g. I frequently
work in ~/hack, and check out stuff into there.

To do this, I have fallen into the habit
of creating a CVSBARRIER directory that is
not checked in, that prevents cvs update
from traversing.

Also, I find it useful to have a place
to put overall comments for a repository.
Typically, this is ~/README or ~/CVS-status.
The overall comments - such as "the tag
named FOO is my home directory at the time
I moved to university BAR and merged in
changes from my laptop BAZZ that had diverged"
get suck in the CVS log, via
cvs ci -f CVS-status.

Oner thing: when yiou have as much history as this, you notice when programs change their interface. E.g. nmh doesn't run my ~/.mhrc from 1994 :-(

--- Andy "Krazy" Glew
(I'm too lazy to create a slashdot account)

Rsync anyone?

[pico303]

Isn't this what rsync is for? CVS seems to be the wrong tool for this job.

Old news

[whereiswaldo]

This was featured in a Linux Journal article from September 2002:

http://linuxjournal.com/article.php?sid=5976

Same guy, too.

Check your assumptions

[MarkusQ]

Smitty825:

I'll put my home directory on Sourceforge!

Everyone can now help me maintain it!

--

Doh!

AvantLegion:

I patched your political science essay.

I fixed all the anti-Bush bugs.

--

Now hold on. Just because his sig is "Doh!" doesn't automatically mean he's anti-Bush.

It might just mean that he's pro-Homer Simpson.

-- MarkusQ

Re:Described in Linux Journal months ago

[zsau]

Perhaps it's stuff that matters?

rcsvi for easy sysadmining

[negyvenot]

rcsvi is a simple wrapper for vi that puts edited files under revision control. It does not support any vi flags. It only takes one file argument and an optional revision number for reverting to previous versions. A few examples:

# alias vi=rcsvi
# vi /etc/passwd
[ remove Agent Smith's account ]
:wq
/etc/passwd: 12 lines, 94 characters.
enter log message, terminated with single '.' or end of file:
>> Agent Smith's account removed.
.

Voila. You now have your passwd file under revision control. More examples:

# vi -r1.2 /etc/passwd
# rlog /etc/passwd
# rcsdiff -r1.2 -r1.3 /etc/passwd

To make a complete backup of your system configuration:

# tar fc /tmp/RCS.tar $(find / -type d -name RCS)

Now you may ask "ok i want to edit multiple files and/or do some other trickery". Don't. It's a simple tool, that i'm using for years now with great satisfaction.

Check it out here