About as slowly as it is possible for it to work, and still be faster than copying and pasting those changes by hand.
We use SVN 1.5... with -r and --ignore-ancestry. In other words, we use it exactly like SVN 1.4.
This also doesn't give you any less conflicts, it only saves you from looking up the versions. That's significant, but it's not enough. Most of us are running git-svn branches, now. A full conversion to Git seems inevitable.
The only thing distinguishing the commercial VCS tools now is their integration with the likes of project management and issue tracking tools
And it's my guess that Ditz will change all of that, when it's mature enough to actually replace the existing tools.
It already has one feature I haven't found anywhere else. Not that it doesn't exist, but that I doubt it could be done this well:
Since Ditz is stored as simple text files inside git (or bzr, or whatever, even SVN if you like), this means that changes to the Ditz repository follow development. Which means that your ticket workflow automatically matches your development workflow.
It means that, for example, it's possible for a ticket to be marked resolved, but only within a given branch. The statement claiming the bug is fixed would follow the fix itself.
Which means that you get (for free!) a comprehensive list of new features in a given release, a comprehensive list of known bugs, even a kind of poor-man's roadmap. And you get this information simply by tagging the release, no need to mess with whatever your issue tracker calls "milestones" or "versions" (separate concepts in Trac, for some reason).
We work on one branch per developer, typically. This was intended to help keep trunk clean, keep us from stepping on our toes, but avoid the dangerous habit of just not checking something in until it's ready.
Then, we got sick of the amount of time required -- both of us and of SVN -- to do merges. We pretty much decided we were going to move to Git -- but the time to do so wasn't available, so we instead switched to SVN 1.5.
It was nice, at first. It's really nice to be able to just fire off an "svn merge../some_branch" and let it do its thing. It was only slightly slower than in 1.4.
Then it got slower.
And slower.
Now, there's a good chance a merge will take a solid 20 minutes or so to even start, and several minutes after that to finish the merge -- during which time, you can't really touch that branch. But exactly the same number of merge conflicts as before.
Turns out, all SVN 1.5 does is track merge history. This makes the output of "svn blame" somewhat more useful, and (so far) only slightly slower. It also means it can automatically figure out the range of revisions which need to be merged from any given branch, by looking at the merge history to figure out which revisions have already been merged.
Which means it saves you having to look at the log and find out those revisions for yourself. It won't save you from a single merge conflict, no matter how stupidly easy such a conflict would be.
Now, since it's possible that:
- You could merge from trunk
- I could merge from trunk
- You could make a bunch of changes
- You then merge back into trunk
- I merge from you
- I then try to merge from trunk
In that example, since I merged from you earlier, I already have the changes that you merged from trunk.
If you think about this a bit, the implication is that an SVN 15 merge has to consider almost every merge that has ever happened. Even if the branch in question was deleted long ago, SVN will dig it up, just to make sure you don't have those commits.
And this is actually the best they can do. I believe it would take a significant restructuring of SVN to do better.
So, if you only have one or two branches, and merging is a rare (monthly) occurrence, you'll probably be fine. For awhile.
Eventually, you'll probably do what our team ended up doing -- record mergeinfo in the commit log, and merge manually, by specifying a range of revisions -- and, to make it go faster, specify --ignore-ancestry.
In other words, to make it go faster, we have to explicitly force SVN 1.5 to merge exactly the way SVN 1.4 does.
So, my suggestion is, don't bother. It's a smooth enough upgrade, but it offers very little of value.
You'll probably end up doing what we're doing now: Staying on trunk, mostly, or on a longer-lived feature branch, using git-svn to make it slightly less brain-dead.
SVN isn't better or worse than GIT, except perhaps when examining a given feature.
There are a very small number of features for which SVN is better than GIT.
Therefore, even when working in such a way that SVN would suit you fine, I'd go with Git for things like size and speed alone.
An SVN checkout of a single branch of my project (trunk, say) is a little over 160 megs. A git-svn checkout of the entire project -- all history it could find, all branches, all tags (and we tag once per deploy, which can be several times per week) -- is 140 megs.
Given that, why would I waste my time with SVN? What does it do better?
From what I can tell, there's partial checkouts -- which still have a fair chance of using more disk than a simple git clone -- and Tortoise. And there are things gitk does better than Tortoise, because of fundamental features missing from SVN.
If I'm keeping track of a bunch of branches in my head, then that limits the amount of other stuff I can keep track of.
Which means it should be doing something for you to be worth that cost -- like reducing what you have to keep track of.
I know I can keep track of a number of branches -- both in my head and by looking at current available branches -- much more easily than I can keep track of a number of features, which may or may not be implemented in my current branch.
The time I and the people on my team spend talking about the different branches.... If my ops guy or my product manager has to figure out what's where and what's elsewhere....
You're doing it wrong.
Branches in Git are so cheap, and so easy to manage, that you might well create a few branches just for yourself, that no one else even has to know about, unless you decide to commit that branch.
Once it's merged to Master, you don't have to think about them, because that branch is gone. Your ops guy and your project manager only really need to worry about Master (equivalent of Trunk in SVN).
about 90% of the time I see people doing lots of branching and merging, it's a response to screwed-up organizations or dysfunctional personal relationships.
Those are certainly situations which could force people to branch/merge more than they otherwise would. But it's by no means a measure of how useful branching/merging are.
Taking a simpler example: The case could be made that most people use version control at all as a response to a dysfunctional organization. After all, if people worked well together, you could simply keep your code on a shared folder, and make sure people only updated it when it was their turn.
A more blatant analogy: A cripple needs a car. That doesn't mean a car is useless for anyone but cripples. Just because something is useful to a cripple doesn't imply it's a crutch.
branching and merging adds a fair bit of overhead
...on SVN. On Git, branching and merging is less painful than checking out and committing is on SVN.
The main reason I can't do that is the amount of time I'd be sleep-deprived before it started working (can't lose a full week that way). And I don't really want the absolute inflexibility of pretty much instantly falling asleep at naptime.
I was referring to the full-blown application state preservation (a la GMail) as proper RESTfulness. Anything less is just stupidity on the part of developers.
Be clear, then: Exactly what are you claiming is not RESTful, and stupidity on the part of developers?
Certainly you're not claiming that an AJAX-free page cannot be RESTful.
Even when they were in the womb, my daughters (who have virtually identical genetic codes) showed different personalities--one would react to Jazz and Rock (by kicking), the other would react most strongly to Classical.
Firstly: You're mistaking a reaction to a personality. All you understand is that you did something (played rock, or played classical), and your infants reacted (kicked). Everything after that is an assumption, unless your daughters can actually tell you that they remember hearing one or the other.
To put another way: When I play Half-Life, the Marines react one way when I shoot them (by shooting back), and scientists react another way (by screaming and running away). Similarly, they react differently when I walk up and press the "use" key -- the Marine kicks at me (because I'm close enough that he can't shoot me), and the scientist starts following me -- or stops, if he was.
Yet these are machines. Less than that, they are programs. Their whole existence can be distilled down to a few lines of code.
What I am trying to say is: We don't know. No one can say for sure even whether adult human beings have a soul, much less a fetus. We cannot tell whether
What we do know is that aborting the pregnancy will save the mother a great deal of pain, and sometimes, a great deal of danger. (Keep in mind: McCain is unwilling to make an exception for the life and health of the mother.)
We kill for far less than that. We kill cows, chickens, turkeys, fish, and other things for no reason other than that we like the way we taste. (After all, we can survive just as well on a vegetarian diet.) Arguably, most of the things we kill for food have more personality than a first-trimester fetus.
Now they're 7, and they're near-opposites in everything but appearance.
Which again proves nothing -- this is actually quite common among twins.
every abortion kills a baby.
Go find out what a fetus looks like at various stages of development. I'll bet your daughters were at least in the second trimester when you felt them kick.
To say that every abortion kills a baby is a bit like being outraged that every period kills an egg.
Most userspace apps, like OpenOffice, orderly open a file, write to it, then close it. close() is a potentially perfect place for the required hooks that solve the problem.
That's a big assumption, which will fall apart horribly if the user happens to be using something like Amarok on SQLite, or whatever the hell Firefox does for its own storage.
It's quite irrelevant when the data hits persistent storage,
It is, however, relevant to when the filesystem decides to make a snapshot.
Concerning amount of space used for this feature - I think it should be configurable (e.g. maximum and minimum number of historical versions both per single file and globally per filesystem; maxmimum and minimum amount of used space - likewise).
Which means we can now consider how much reliability we then lose in consideration of space.
The reason I mention fsync, and transactions, and the whole concurrent write mess, is that solving this problem solves the other. If we could identify with certainty where a transaction is, that would be an ideal place to snapshot.
I think the biggest problem is that the filesystem would lose the ability to make intelligent decisions about where to allocate extents from and how to index them
And the extent layer would pick up that ability.
What additional information does the filesystem have, that it would make this choice with?
without the scheme devolving into a "allocate one extent the size of the disk and manage it locally" being the best performer.
Certainly, most database engines have devolved into "allocate one file the size of the data and manage it locally". I would guess that's mostly because existing filesystems don't provide a better option.
Most of the hard parts of reliably packing extents into a block device are done different ways by different file systems for different reasons.
Most are also done making large assumptions about the underlying media -- mostly that it is a single magnetic disk.
ext3 is probably already close enough to what you describe; it has a way of allocating extents and naming them (files) and using them for whatever you want.
I never said they would be named.
And the names introduces a whole new world of filesystem-specific performance implications (Trivial example: dir_index or not?) and actual limitations (UNIX file permissions? POSIX ACLs? Extended attributes? Separate streams?) that belong more to a filesystem (or database) than to a simple storage manager.
Rather, you would allocate an extent, and it is automatically assigned a number, which you memorize (somehow). So it's much closer to an inode -- but still not the same, as an inode stores metadata about the file besides its physical location on disk.
In a RAID on top of extents
No, the point is that the extent layer itself would be where you implement RAID. Not above it (because then it's filesystem-specific, and you've got another ZFS), and not below it (because then you lose information).
Trivial example: RAID-Z, from ZFS, will only bother reconstructing space which is actually allocated, somehow. Traditional block-level RAID can only reconstruct the entire disk, including unallocated (and thus worthless) space -- because with no knowledge of the filesystem, it can't know what's allocated or not.
All the more reason to make it pluggable. But hey, at least we could try.
It would have to track which file systems own each extent
I don't really see why. All it needs to keep track of is a unique id for the extent. Garbage collection could be left up to the filesystems -- or to another layer, depending on what people end up building.
All it would have to keep track of is at least one superblock. That would be enough for another system to bootstrap, and handle "ownership" of the rest -- but I still don't think it needs to be tracked beyond, say, a superblock per filesystem.
Another analogy: This is like Ring 0. Yes, any filesystem could accidentally trample all over another filesystem's extents, just as any kernel driver can trample over another kernel driver's memory. If you really need the protections of userspace, you build something on top of it.
That's not entirely a new idea -- LLVM handles the low-level details of linking, native machine code, etc, but does absolutely nothing for garbage collection. Elsewhere, some languages use garbage collection libraries, and some use simple reference counting -- but LLVM is designed to be able to host them all.
if you want to come close to matching RAID-Z you are going to need to be able to return very small extents (LVM defaults to 4MB, IIRC).
Which is why LVM is not appropriate for this. Nor is device-mapper.
I can think of ways to optimize that a bit, but I imagine it'll still have a significant space+performance impact.
I don't think it would have a significant space impact -- especially as it would utterly trivialize the process of adding transparent compression to any filesystem.
I don't know enough to really say whether it would have a significant performance impact. I don't think it would.
I also think that it would be worth a performance impact. I can sacrifice a few cycles if it means a better structured, more reliable system.
Thanks for your response -- it looks like most people thought I was just asking for LVM. I'm glad at least one person gets it.
My description was of an extent layer -- that is, when a filesystem needs an extent, it asks this layer. The idea is that, if a filesystem were to use this for individual files, it would allow the per-file striping/mirroring of ZFS, but would work with any filesystem implementing this layer.
I doubt either LVM or device-mapper would work for the number of extents I'm talking about.
It almost looks like they just regurgitated an article of Firefox 3.0.
For instance, Cntrl-tab has been added as a shortcut to quickly switch between tabs. Also, tabs can be dragged and dropped between Mozilla windows, a feature that works well and smoothly.
Both of these are present and working well in my Firefox 3.0.3 on Ubuntu.
Oh, and it's not Cntrl. It's Ctrl or control, depending on your keyboard, but I've never heard one call it Cntrl.
Well, it was another copy/paste troll. And I found quite a lot more wrong with it than the lynching -- which was downright comical. Black robes and hoods?!
Way up at the beginning, "ATHEIST KID" is probably meant to be a teenager. I can see where some people might have a problem with that -- especially a parent. If nothing else, there's STDs, and there's the working conditions of the hooker.
Marijuana is almost certainly not as bad as it's made out to be, and I would say the impact of outlawing it is worse than the drug itself. But from what my parents tell me of their own experience (so to speak), it absolutely has a negative effect on memory. You're going to tell me that atheists, who don't believe (or don't know) that they're going anywhere after they die, are going to risk the health of their one and only brain for a high?
Some would, I'm sure. Some Christians would (and do) as well. But I doubt there's a correlation, except in extreme cases -- like Rastafarians, who consider it a religious experience.
And, well, abortions are expensive, potentially dangerous, and time consuming. Even if you know nothing about atheists -- and hey, there's no reason an atheist can't be pro-life -- why would anyone choose a $500 abortion over a $1 condom?
Even if you assume atheists are inherently amoral (NOT true), why would you also assume they're complete morons?
Consider that POSIX provides no transactional guarantee other than sync and fsync.
How does the filesystem know when to create a new version? Should every byte ever written to the file be construed as a new version? If so, how does the admin figure out which precise version, out of the literally billions that would be created, is the right one?
And how do you reasonably prune that wasted space?
No, what you really want is version control software. Users too. At the very least, it ensures that each commit was deliberate, and represents a valid state.
ZFS duplicates a lot of functionality that belongs outside of a filesystem.
Very true.
It wouldn't be possible to duplicate RAID-Z with LVM.
Also true.
And the features which could be duplicated, couldn't be done nearly as well without a little more knowledge of the filesystem.
The real problem here is that we're finding out that generic block devices aren't enough to do everything we want to do outside the filesystem itself. Or, if they are, it's incredibly clumsy. Trivial example: If I want a copy-on-write snapshot, I have to set aside (ahead of time) some fixed amount of space that it can expand into. If I guess high, I waste space. If I guess low, I have to either expand it (somehow, if that's even possible) or lose my snapshot.
A filesystem which natively implemented COW could also trivially implement snapshots which take up exactly as much space as there are differences between the increments. But because of the way the Linux VFS is structured, this kind of functionality would have to be in a single filesystem, and would be duplicated across all filesystems. Best case, it'd be like ext3's JBD, as a kind of shared library.
A humble proposal: We need another layer, between the block layer and the filesystem layer -- call it an extent layer -- which is simply concerned with allocating some amount of space, and (perhaps) assigning it a unique ID. Filesystems could sit above this layer and implement whatever crazy optimizations or semantics they want -- linear vs btree vs whatever for directories, POSIX vs SQL, whatever.
The extent layer itself would only be concerned with allocating extents of some requested size, and actually storing the data. But this would be enough information to effectively handle mirroring, striping, snapshotting, copy-on-write, etc.
It wouldn't be universal -- I've said nothing about the on-disk format, and, indeed, some filesystems exist on Linux solely for that purpose -- vfat, ntfs, udf, etc. Those filesystems could be done pretty much exactly the way they're done now. After all, the existence of a block layer in no way implies that every filesystem must be tied to a block device (see proc, sys, fuse, etc.)
But I think it would work very well for filesystems which did choose to implement it. I think it would provide the best of ZFS and LVM.
I haven't actually been seriously following filesystem development for years, so maybe this is already done. Or maybe it's a bad idea. If not, hopefully some kernel developers are reading this.
REST should be the default (as encouraged by Rails).
First, burden's on you -- show me how hash links are not RESTful.
There are cases where what you're describing makes sense -- although we're still almost certainly talking about destroying cache coherency, as you now have to provide both the AJAX version of a resource (just a page fragment) and the non-AJAX version (the full layout).
In fact, I've had to intentionally destroy cache coherency, to avoid very weird bugs where the AJAXy response was cached for the middle click action, or vice versa.
And there are cases where what you're describing makes pretty much no sense -- where the resource you're linking to really doesn't make sense outside of the context of a larger application. In cases like that, I'd much rather be able to link to a specific point inside of an application than to only be able to link to entire "resources", at whatever granularity the page author decided to define a resource.
Slashdot Mirror
Yeah, it works.
About as slowly as it is possible for it to work, and still be faster than copying and pasting those changes by hand.
We use SVN 1.5... with -r and --ignore-ancestry. In other words, we use it exactly like SVN 1.4.
This also doesn't give you any less conflicts, it only saves you from looking up the versions. That's significant, but it's not enough. Most of us are running git-svn branches, now. A full conversion to Git seems inevitable.
The only thing distinguishing the commercial VCS tools now is their integration with the likes of project management and issue tracking tools
And it's my guess that Ditz will change all of that, when it's mature enough to actually replace the existing tools.
It already has one feature I haven't found anywhere else. Not that it doesn't exist, but that I doubt it could be done this well:
Since Ditz is stored as simple text files inside git (or bzr, or whatever, even SVN if you like), this means that changes to the Ditz repository follow development. Which means that your ticket workflow automatically matches your development workflow.
It means that, for example, it's possible for a ticket to be marked resolved, but only within a given branch. The statement claiming the bug is fixed would follow the fix itself.
Which means that you get (for free!) a comprehensive list of new features in a given release, a comprehensive list of known bugs, even a kind of poor-man's roadmap. And you get this information simply by tagging the release, no need to mess with whatever your issue tracker calls "milestones" or "versions" (separate concepts in Trac, for some reason).
Bazaar will let you work as if it were Subversion, for example, with a central server, lightweight local checkouts (with no history).
Typically, Git checkouts take up less disk space than SVN checkouts, even though Git checkouts have history, and SVN checkouts don't.
Most of the workflows mentioned are just as possible with Git. All bzr does is make them easier, and that's arguable.
Well, there is one big advantage to SVN: it's possible for mere mortals to understand how it works. To understand git you have to be a fucking genius.
In many ways, the Git model is simpler than SVN.
You're right, it's more moving parts, but once you've got the fundamentals, the rest of it (like rebase vs merge) seems obvious.
We work on one branch per developer, typically. This was intended to help keep trunk clean, keep us from stepping on our toes, but avoid the dangerous habit of just not checking something in until it's ready.
Then, we got sick of the amount of time required -- both of us and of SVN -- to do merges. We pretty much decided we were going to move to Git -- but the time to do so wasn't available, so we instead switched to SVN 1.5.
It was nice, at first. It's really nice to be able to just fire off an "svn merge ../some_branch" and let it do its thing. It was only slightly slower than in 1.4.
Then it got slower.
And slower.
Now, there's a good chance a merge will take a solid 20 minutes or so to even start, and several minutes after that to finish the merge -- during which time, you can't really touch that branch. But exactly the same number of merge conflicts as before.
Turns out, all SVN 1.5 does is track merge history. This makes the output of "svn blame" somewhat more useful, and (so far) only slightly slower. It also means it can automatically figure out the range of revisions which need to be merged from any given branch, by looking at the merge history to figure out which revisions have already been merged.
Which means it saves you having to look at the log and find out those revisions for yourself. It won't save you from a single merge conflict, no matter how stupidly easy such a conflict would be.
Now, since it's possible that:
- You could merge from trunk
- I could merge from trunk
- You could make a bunch of changes
- You then merge back into trunk
- I merge from you
- I then try to merge from trunk
In that example, since I merged from you earlier, I already have the changes that you merged from trunk.
If you think about this a bit, the implication is that an SVN 15 merge has to consider almost every merge that has ever happened. Even if the branch in question was deleted long ago, SVN will dig it up, just to make sure you don't have those commits.
And this is actually the best they can do. I believe it would take a significant restructuring of SVN to do better.
So, if you only have one or two branches, and merging is a rare (monthly) occurrence, you'll probably be fine. For awhile.
Eventually, you'll probably do what our team ended up doing -- record mergeinfo in the commit log, and merge manually, by specifying a range of revisions -- and, to make it go faster, specify --ignore-ancestry.
In other words, to make it go faster, we have to explicitly force SVN 1.5 to merge exactly the way SVN 1.4 does.
So, my suggestion is, don't bother. It's a smooth enough upgrade, but it offers very little of value.
You'll probably end up doing what we're doing now: Staying on trunk, mostly, or on a longer-lived feature branch, using git-svn to make it slightly less brain-dead.
Try to do it from a command line (you can, but you'll need a lot of 'blame' and 'log' commands).
Actually, on a decent SCM (like Git), or on SVN 1.5 or greater, you need exactly one 'blame' command.
SVN isn't better or worse than GIT, except perhaps when examining a given feature.
There are a very small number of features for which SVN is better than GIT.
Therefore, even when working in such a way that SVN would suit you fine, I'd go with Git for things like size and speed alone.
An SVN checkout of a single branch of my project (trunk, say) is a little over 160 megs. A git-svn checkout of the entire project -- all history it could find, all branches, all tags (and we tag once per deploy, which can be several times per week) -- is 140 megs.
Given that, why would I waste my time with SVN? What does it do better?
From what I can tell, there's partial checkouts -- which still have a fair chance of using more disk than a simple git clone -- and Tortoise. And there are things gitk does better than Tortoise, because of fundamental features missing from SVN.
If I'm keeping track of a bunch of branches in my head, then that limits the amount of other stuff I can keep track of.
Which means it should be doing something for you to be worth that cost -- like reducing what you have to keep track of.
I know I can keep track of a number of branches -- both in my head and by looking at current available branches -- much more easily than I can keep track of a number of features, which may or may not be implemented in my current branch.
The time I and the people on my team spend talking about the different branches.... If my ops guy or my product manager has to figure out what's where and what's elsewhere....
You're doing it wrong.
Branches in Git are so cheap, and so easy to manage, that you might well create a few branches just for yourself, that no one else even has to know about, unless you decide to commit that branch.
Once it's merged to Master, you don't have to think about them, because that branch is gone. Your ops guy and your project manager only really need to worry about Master (equivalent of Trunk in SVN).
about 90% of the time I see people doing lots of branching and merging, it's a response to screwed-up organizations or dysfunctional personal relationships.
Those are certainly situations which could force people to branch/merge more than they otherwise would. But it's by no means a measure of how useful branching/merging are.
Taking a simpler example: The case could be made that most people use version control at all as a response to a dysfunctional organization. After all, if people worked well together, you could simply keep your code on a shared folder, and make sure people only updated it when it was their turn.
A more blatant analogy: A cripple needs a car. That doesn't mean a car is useless for anyone but cripples. Just because something is useful to a cripple doesn't imply it's a crutch.
branching and merging adds a fair bit of overhead
...on SVN. On Git, branching and merging is less painful than checking out and committing is on SVN.
Sounds suspiciously like an Uberman schedule.
The main reason I can't do that is the amount of time I'd be sleep-deprived before it started working (can't lose a full week that way). And I don't really want the absolute inflexibility of pretty much instantly falling asleep at naptime.
I was referring to the full-blown application state preservation (a la GMail) as proper RESTfulness. Anything less is just stupidity on the part of developers.
Be clear, then: Exactly what are you claiming is not RESTful, and stupidity on the part of developers?
Certainly you're not claiming that an AJAX-free page cannot be RESTful.
Even when they were in the womb, my daughters (who have virtually identical genetic codes) showed different personalities--one would react to Jazz and Rock (by kicking), the other would react most strongly to Classical.
Firstly: You're mistaking a reaction to a personality. All you understand is that you did something (played rock, or played classical), and your infants reacted (kicked). Everything after that is an assumption, unless your daughters can actually tell you that they remember hearing one or the other.
To put another way: When I play Half-Life, the Marines react one way when I shoot them (by shooting back), and scientists react another way (by screaming and running away). Similarly, they react differently when I walk up and press the "use" key -- the Marine kicks at me (because I'm close enough that he can't shoot me), and the scientist starts following me -- or stops, if he was.
Yet these are machines. Less than that, they are programs. Their whole existence can be distilled down to a few lines of code.
What I am trying to say is: We don't know. No one can say for sure even whether adult human beings have a soul, much less a fetus. We cannot tell whether
What we do know is that aborting the pregnancy will save the mother a great deal of pain, and sometimes, a great deal of danger. (Keep in mind: McCain is unwilling to make an exception for the life and health of the mother.)
We kill for far less than that. We kill cows, chickens, turkeys, fish, and other things for no reason other than that we like the way we taste. (After all, we can survive just as well on a vegetarian diet.) Arguably, most of the things we kill for food have more personality than a first-trimester fetus.
Now they're 7, and they're near-opposites in everything but appearance.
Which again proves nothing -- this is actually quite common among twins.
every abortion kills a baby.
Go find out what a fetus looks like at various stages of development. I'll bet your daughters were at least in the second trimester when you felt them kick.
To say that every abortion kills a baby is a bit like being outraged that every period kills an egg.
Most userspace apps, like OpenOffice, orderly open a file, write to it, then close it. close() is a potentially perfect place for the required hooks that solve the problem.
That's a big assumption, which will fall apart horribly if the user happens to be using something like Amarok on SQLite, or whatever the hell Firefox does for its own storage.
It's quite irrelevant when the data hits persistent storage,
It is, however, relevant to when the filesystem decides to make a snapshot.
Concerning amount of space used for this feature - I think it should be configurable (e.g. maximum and minimum number of historical versions both per single file and globally per filesystem; maxmimum and minimum amount of used space - likewise).
Which means we can now consider how much reliability we then lose in consideration of space.
The reason I mention fsync, and transactions, and the whole concurrent write mess, is that solving this problem solves the other. If we could identify with certainty where a transaction is, that would be an ideal place to snapshot.
Cool!
I guess I can't claim originality, but at least that's some reassurance that I know what I'm talking about.
I think the biggest problem is that the filesystem would lose the ability to make intelligent decisions about where to allocate extents from and how to index them
And the extent layer would pick up that ability.
What additional information does the filesystem have, that it would make this choice with?
without the scheme devolving into a "allocate one extent the size of the disk and manage it locally" being the best performer.
Certainly, most database engines have devolved into "allocate one file the size of the data and manage it locally". I would guess that's mostly because existing filesystems don't provide a better option.
Most of the hard parts of reliably packing extents into a block device are done different ways by different file systems for different reasons.
Most are also done making large assumptions about the underlying media -- mostly that it is a single magnetic disk.
ext3 is probably already close enough to what you describe; it has a way of allocating extents and naming them (files) and using them for whatever you want.
I never said they would be named.
And the names introduces a whole new world of filesystem-specific performance implications (Trivial example: dir_index or not?) and actual limitations (UNIX file permissions? POSIX ACLs? Extended attributes? Separate streams?) that belong more to a filesystem (or database) than to a simple storage manager.
Rather, you would allocate an extent, and it is automatically assigned a number, which you memorize (somehow). So it's much closer to an inode -- but still not the same, as an inode stores metadata about the file besides its physical location on disk.
In a RAID on top of extents
No, the point is that the extent layer itself would be where you implement RAID. Not above it (because then it's filesystem-specific, and you've got another ZFS), and not below it (because then you lose information).
Trivial example: RAID-Z, from ZFS, will only bother reconstructing space which is actually allocated, somehow. Traditional block-level RAID can only reconstruct the entire disk, including unallocated (and thus worthless) space -- because with no knowledge of the filesystem, it can't know what's allocated or not.
We'd never be able to get it right
All the more reason to make it pluggable. But hey, at least we could try.
It would have to track which file systems own each extent
I don't really see why. All it needs to keep track of is a unique id for the extent. Garbage collection could be left up to the filesystems -- or to another layer, depending on what people end up building.
All it would have to keep track of is at least one superblock. That would be enough for another system to bootstrap, and handle "ownership" of the rest -- but I still don't think it needs to be tracked beyond, say, a superblock per filesystem.
Another analogy: This is like Ring 0. Yes, any filesystem could accidentally trample all over another filesystem's extents, just as any kernel driver can trample over another kernel driver's memory. If you really need the protections of userspace, you build something on top of it.
That's not entirely a new idea -- LLVM handles the low-level details of linking, native machine code, etc, but does absolutely nothing for garbage collection. Elsewhere, some languages use garbage collection libraries, and some use simple reference counting -- but LLVM is designed to be able to host them all.
if you want to come close to matching RAID-Z you are going to need to be able to return very small extents (LVM defaults to 4MB, IIRC).
Which is why LVM is not appropriate for this. Nor is device-mapper.
I can think of ways to optimize that a bit, but I imagine it'll still have a significant space+performance impact.
I don't think it would have a significant space impact -- especially as it would utterly trivialize the process of adding transparent compression to any filesystem.
I don't know enough to really say whether it would have a significant performance impact. I don't think it would.
I also think that it would be worth a performance impact. I can sacrifice a few cycles if it means a better structured, more reliable system.
Thanks for your response -- it looks like most people thought I was just asking for LVM. I'm glad at least one person gets it.
My description was of an extent layer -- that is, when a filesystem needs an extent, it asks this layer. The idea is that, if a filesystem were to use this for individual files, it would allow the per-file striping/mirroring of ZFS, but would work with any filesystem implementing this layer.
I doubt either LVM or device-mapper would work for the number of extents I'm talking about.
It almost looks like they just regurgitated an article of Firefox 3.0.
For instance, Cntrl-tab has been added as a shortcut to quickly switch between tabs. Also, tabs can be dragged and dropped between Mozilla windows, a feature that works well and smoothly.
Both of these are present and working well in my Firefox 3.0.3 on Ubuntu.
Oh, and it's not Cntrl. It's Ctrl or control, depending on your keyboard, but I've never heard one call it Cntrl.
Well, actually, no. Not unless you're suggesting that LVM is appropriate at the file level -- as in, creating a new device per-file.
Well, it was another copy/paste troll. And I found quite a lot more wrong with it than the lynching -- which was downright comical. Black robes and hoods?!
Way up at the beginning, "ATHEIST KID" is probably meant to be a teenager. I can see where some people might have a problem with that -- especially a parent. If nothing else, there's STDs, and there's the working conditions of the hooker.
Marijuana is almost certainly not as bad as it's made out to be, and I would say the impact of outlawing it is worse than the drug itself. But from what my parents tell me of their own experience (so to speak), it absolutely has a negative effect on memory. You're going to tell me that atheists, who don't believe (or don't know) that they're going anywhere after they die, are going to risk the health of their one and only brain for a high?
Some would, I'm sure. Some Christians would (and do) as well. But I doubt there's a correlation, except in extreme cases -- like Rastafarians, who consider it a religious experience.
And, well, abortions are expensive, potentially dangerous, and time consuming. Even if you know nothing about atheists -- and hey, there's no reason an atheist can't be pro-life -- why would anyone choose a $500 abortion over a $1 condom?
Even if you assume atheists are inherently amoral (NOT true), why would you also assume they're complete morons?
That seems overly difficult to maintain.
Just treat the hard drive as a gigantic cache. When you run out of space, automatically delete files which haven't been so much as read in six months.
To keep it sane, keep an explicit break between what you're using as cache, and what is to be kept forever.
I do agree on tag-based, though I'd take it a step further and do search-based. Come to think of it, Spotlight (among others) already does this.
This is very, very tricky to get right.
Consider that POSIX provides no transactional guarantee other than sync and fsync.
How does the filesystem know when to create a new version? Should every byte ever written to the file be construed as a new version? If so, how does the admin figure out which precise version, out of the literally billions that would be created, is the right one?
And how do you reasonably prune that wasted space?
No, what you really want is version control software. Users too. At the very least, it ensures that each commit was deliberate, and represents a valid state.
ZFS duplicates a lot of functionality that belongs outside of a filesystem.
Very true.
It wouldn't be possible to duplicate RAID-Z with LVM.
Also true.
And the features which could be duplicated, couldn't be done nearly as well without a little more knowledge of the filesystem.
The real problem here is that we're finding out that generic block devices aren't enough to do everything we want to do outside the filesystem itself. Or, if they are, it's incredibly clumsy. Trivial example: If I want a copy-on-write snapshot, I have to set aside (ahead of time) some fixed amount of space that it can expand into. If I guess high, I waste space. If I guess low, I have to either expand it (somehow, if that's even possible) or lose my snapshot.
A filesystem which natively implemented COW could also trivially implement snapshots which take up exactly as much space as there are differences between the increments. But because of the way the Linux VFS is structured, this kind of functionality would have to be in a single filesystem, and would be duplicated across all filesystems. Best case, it'd be like ext3's JBD, as a kind of shared library.
A humble proposal: We need another layer, between the block layer and the filesystem layer -- call it an extent layer -- which is simply concerned with allocating some amount of space, and (perhaps) assigning it a unique ID. Filesystems could sit above this layer and implement whatever crazy optimizations or semantics they want -- linear vs btree vs whatever for directories, POSIX vs SQL, whatever.
The extent layer itself would only be concerned with allocating extents of some requested size, and actually storing the data. But this would be enough information to effectively handle mirroring, striping, snapshotting, copy-on-write, etc.
It wouldn't be universal -- I've said nothing about the on-disk format, and, indeed, some filesystems exist on Linux solely for that purpose -- vfat, ntfs, udf, etc. Those filesystems could be done pretty much exactly the way they're done now. After all, the existence of a block layer in no way implies that every filesystem must be tied to a block device (see proc, sys, fuse, etc.)
But I think it would work very well for filesystems which did choose to implement it. I think it would provide the best of ZFS and LVM.
I haven't actually been seriously following filesystem development for years, so maybe this is already done. Or maybe it's a bad idea. If not, hopefully some kernel developers are reading this.
Then what would you recommend?
And can you show me where this joke has existed before -- especially in the context of Gears?
REST should be the default (as encouraged by Rails).
First, burden's on you -- show me how hash links are not RESTful.
There are cases where what you're describing makes sense -- although we're still almost certainly talking about destroying cache coherency, as you now have to provide both the AJAX version of a resource (just a page fragment) and the non-AJAX version (the full layout).
In fact, I've had to intentionally destroy cache coherency, to avoid very weird bugs where the AJAXy response was cached for the middle click action, or vice versa.
And there are cases where what you're describing makes pretty much no sense -- where the resource you're linking to really doesn't make sense outside of the context of a larger application. In cases like that, I'd much rather be able to link to a specific point inside of an application than to only be able to link to entire "resources", at whatever granularity the page author decided to define a resource.