That's not data, it's only a representation, can you get it?.
Sorry, the person not getting things here is not me. Copy-on-write is a technique of avoiding changing shared data in place. Let's not get all abstract and confused about that, ok? I mean, you can wank on about your data abstractions, but in the process you will also wank away the computer science and miss the point entirely.
The motivation for not changing the original, shared data in copy-on-write is, we might not know about all the previously existing references to that data, so there may be no practical way to find them all and change them. Microsoft uses a different technique in shadow copy, they do know about all the incoming references from previously snapshots, so they change them to point somewhere else, copy the original data there, then proceed to change the original data. That is not copy-on-write because it does not avoid changing the original, shared data, in place. (See, I spelled out the "in place" thing as a comprehension aid.)
Two very different techniques, with very different performance characteristics. Copy-on-write is O(1) in number of incoming references, while copy-before-write is O(N) in number of incoming references. Copy-before-write requires stalling a write operation for the duration of the copy and metadata update, while copy-on-write does not. Fundamentally different algorithms, as is apparent because of the different complexity characterics. Irrespective of the computer science involved, Microsoft ignores the subtleties and calls the second thing copy-on-write anyway. Opening up plenty of opportunity for wanking on Slashdot by the likes of you. Now, if you want to be precise about it, use the term redirect-on-write every time we actually mean the classic computer science concept copy-on-write (which is documented reasonably well on Wikipedia, except in a certain paragraph containing the word "Microsoft").
As for your "just a representation" argument, let's extend that. A sorted list is "just a representation" of an unsorted list, therefore sorting costs nothing. Oh wait, that's nonsense. Just as your "that's not data" argument is nonsense.
Finally, I hope you agree that copy-before-write performance sucks pretty badly compared to classic copy-on-write (redirect-on-write). And by extension, Microsoft's shadow copy feature sucks badly. Confusing the issue by confusing the terminology does help Microsoft avoid criticism over poor performance. But I would not go so far as ascribing to malice what can be more easily explained by incompetence.
You're not even talking about LVM, you're talking about making multiple databases operate together atomically. In the context of LVM, that's complete nonsense. It's not an LVM concept, it's a high level application concept. Why are you even wasting bandwidth conflating these issues? If you want to do the job with LVM, you concatenate the volumes and run a single filesystem, or single database on the aggregate volume. Conflating this with the application level consistency somebody dragged into the discussion is just idiotic. By the way, you should keep a lid on the hubris about who knows what.
Your frontal cortex is a "stunt". Nature has repurposed wiring originally evolved to filter 2D imagery. Now it makes complex associations and manipulates data in abstract ways that we are only beginning to decode. But it's still a stunt. I understand why you have difficulty comprehending that progress in AI research is in fact progress. It's because you have no comprehension of the long term implications of work that is being done. You should have a look into the relevant scientific literature yourself, and this time don't just let the words slide away like water off the back of a duck.
No, I mean I have been following the scientific progress in that area for about 2 1/2 decades now , and nobody has a clue. Sure, cretins like Marvin Minsky have been predicting AI for decades now, but that is all about grant money, not about any real results or insights.
Every time you talk your way through a telephone menu you benefit from that work. Just because a computer can't yet compose a symphony does not mean there has been no progress. You can hold out for the artificially intelligent poet in your dreamland if you want, while I watch with interest the progress towards creating an automaton as intelligent as an ant (25,000 neurons). Then a mouse (75 million neurons). Finally, as intelligent as you, then it can post rubbish to Slashdot in your place.
Which of course you can do that, but then you can't have the database LV and the log LV on different physical disks any more, which is what was asked. Can you post an example how you would concatenate two existing LVs, with existing file systems on them, mounted and being modified at the time. into a "new virtual block device" without even un-mounting them, and then make a consistent snapshot of them?
You're delusional, "without even unmounting them" appeared nowhere in the discussion above, nor did the concept of making separate filesystems work together atomically. Your assertion about "different physical disks" doesn't make any sense at all. Of course you can combine different physical disks into a single logical volume. You would then create a single filesystem on the logical volume. Look here for examples.
As nobody even has a rough idea how an AI could be made (hint: it is not a question of computing power), there is little change of anybody making an AI "that could improve itself".
You mean, you do not have even a rough idea. Meanwhile, progress marches on and more and more of the original goals of AI research have been achieved. Machine vision and voice recognition are now commonplace. Computers win at an increasing number of games. Machines walk, balancing realistically. It goes on and on. The only thing that has changed since the beginning is, it's not considered a summer project anymore, the difficulty of engineering at the required scale and with techniques that are discovered and improved only with painstaking work is now properly appreciated. The fact that you have no part in this work doesn't mean much. Researchers in the trenches have considerably more than a rough idea of where they are going. So what if we haven't seen artificial behavior as sophisticated as an ant yet? An ant has 250,000 neurons and we are quite sure that all its behavior is governed by those neurons. Granted, each neuron is an impressively complex system in itself, however there are only so many of them, and increasingly we are gaining the tools to fully reverse engineer them. An ant-scale intelligence is not far away, just by the pedestrian technique of reverse engineering. I would say that the researchers involved in that area have a lot more than a rough idea of where they are going and how to get there.
His synths are beyond decent, they are best in class. What's up with you trash talkers? Ray Kurzweil has made his place in history in an amazing variety of ways. But it's kind of sad he's reduced to taking employment with Google instead of running his own show.
Sounds like Btrfs envy. Question is, can they get to work reliably?
Here is an informative post that details why Microsoft's Refs sucks and you don't need to care about it. Even if it works reliably, which is not at all assured (see many reports on the net of issues) this filesystem is pathetically feature poor. What's the point.
You say: "Copy-on-write leaves the original data unchanged" and VSS leaves the original data unchanged.
the implementation details doesn't change the logical concept. COW says: "The fundamental idea is that if multiple callers ask for resources which are initially indistinguishable, they can all be given pointers to the same resource" but it looks like you don't get it.
I did not say that VSS leaves the original data unchanged, I said the opposite. And this is not an "implementation detail", it's a fundamental property of the operation. And could you please read the next sentence after the one you quoted from Wikipedia, it invalidates your argument. And could you please stop chewing on my toes and learn something about computer science.
Sure, Microsoft abuses the CoW terminology and Wikipedia documents that. More politely than necessary, IMHO.
Copy-on-write leaves the original data unchanged. Copy on write makes a private copy, leaving the orignal unchanged. Microsoft has a different definition, but then Microsoft has a lot of different definitions. Let's you and me be precise about it, and avoid the terminology that Microsoft has wantonly polluted in its ignorance. Copy-before-write or redirect-on-write.
Modifications in the middle of files are extremely rare. It's true, running a database on top of a snapshotted spinning disk is probably going to suck. For normal users, keeping regular files mostly linear, and files in the same directory nearby each other is what matters, and yes, Btrfs does a credible job of that.
I know why shadow copy works the way it does. 1) It's simple, therefore likely to work. 2) It's an easy answer to the "how do you control fragmentation" question. But the write performance issue is so bad that it's a poor solution no matter how you justify it. It's just an attempt to get away with being lazy for a largely uncritical audience that isn't big into benchmarking, or indeed, isn't used to good disk performance.
Btrfs, like ZFS and Netapp's WAFL, use a far more efficient copy-on-write strategy that avoids the write penalty.
WAFL doesn't do copy-on-write. Copy-on-write means a write to a block in a file requires the original block to be read, written elsewhere for the snapshot, then the new block written in the original location. That's exactly what WAFL doesn't do. WAFL writes all changed blocks for multiple files in big RAID stripes, updating pointers to current copies and leaving snapshot pointers pointing to old copies of the updated files. Very efficient for writes, but changes almost all reads, random or sequential (within a file) into random reads (within the filesystem) because file blocks get scattered according to write order, not location of the block within the file. That's why they want lots of spindles in an aggregate and they love RAM cache and flash cache.
But since you say that copy-on-write avoids the write penalty I think you know what is does but simply don't know that it isn't copy-on-write.
We both know what we're talking about, we just disagree on terminology. Properly, a "copy-on-write" doesn't modify the original destination. Nobody should ever use the term "copy-on-write" to describe the algorithm that is properly "copy-before-write". The strategy that leaves the original destination untouched and updates pointers to point at the modified copy is correctly called "copy-on-write", but because the terminology has been so commonly abused by the likes of Microsoft and their followers, it is better to be clear and call that "redirect-on-write".
Finally, Netapp gets massive read fragmentation because they suck, not because it can't be avoided.
LVM is kind of cool but if you have a single database spread across multiple LV's then you can't snapshot them all as an atomic operation so it becomes useless.
You're also wrong about that. You can concatenate multiple logical volumes as a single logical volume and snapshot that atomically.
If you keep all old file where in their original sectors and write changes in new places, your files get fragmented to hell.
Microsoft's "shadow copy" doesn't work at the file level, it works at the block level, so it doesn't know anything about files. Btrfs and its ilk try to leave some empty space distributed across the volume, so copy-on-write can leave the copies in fairly reasonable places. After the copy is committed, the original space can be freed, so the next update won't mess things up too badly either. Snapshots mess this up because the original space doesn't get freed. But then, snapshots are always messed up, there is no such thing as a perfect snapshot strategy with respect to disk seeking. Incidentally, with flash you don't care about that any more, there is no seek time.
Anyway, yes, with a crappy copy-on-write (like Netapp's) you get horrible read fragmentation. With an intelligent implementation, it isn't so bad. Note that Btrfs is turning in good benchmarks, including read performance in mixed read/write loads.
Think about what this means. It is not a "copy-on-write", it is a "copy-before-write". Gross abuse of terminology if anybody tries to call it a "copy-on-write", which has the very specific meaning of "don't modify the destination data". Instead, copy it, then modify the copy. OK, are we clear? VSS does not do copy-on-write, it does copy-before-write.
Now let's think about the implications of that. First, the write needs to be blocked until the copy-before-write completes, otherwise the copied data is not sure to be on stable storage. The copy-before-write needs to read the data from its original position, write it to some save area, then update some metadata to remember which data was saved where. How many disk seeks is that, if it's a spinning disk? If the save area is on the same spinning disk? If it's flash, how much write multiplication is that? When all of that is finally done, the original write can be unblocked and allowed to proceed. In total, how much slower is that than a simple, linear write? If you said "on the order of an order of magnitude" you would be in the ballpark. In face, it can get way worse than that if you are unlucky. In the best imaginable case, your write performance is going to take a hit by a factor of three. Usually, much much worse.
OK, did we get this straight? As a final exercise, see if you can figure out who was talking nonsense.
...Like the way the photoelectric effect and the ultraviolet paradox were the loose threads that lead to quantum mechanics...
Apparently, some time after being generally accepted, quantum mechanics, was applied to solve the ultraviolet "catastrophe" as opposed to the other way round, while the photoelectric effect played a prime role as you say (Einstein got a Nobel prize for it).
A store doesn't have to sell you multiple copies of something...
And the proper response to this is to taser her.
Cue Apple employees lining up to agree that it is right for the Apple police to tazer anybody who does not do exactly as Apple wishes. Kicking in the front doors of journalists is another technique that Apple employees agree Apple should use liberally.
And Apple likes to tazer people who ask questions about it.
a 64-bit output isn't really collision resistant anyway
Plenty good enough for a hashed directory key, which doesn't need to be crypticographically secure, just to have good distribution and random results affected as much as possible by all input bits. The size of the output is not the dominant factor, the quality of the input mixing is.
NTFS doesn't have snapshots. Instead it relies on volume shadow copies, with known severe performance artifacts caused by needing to move snapshotted data out of the way when new writes come in. Btrfs, like ZFS and Netapp's WAFL, use a far more efficient copy-on-write strategy that avoids the write penalty. The takeaway: I would not go so far as to claim Microsoft has an enterprise-worthy solution either. If you want something with industrial strength dedup, snapshots and fault tolerance, you won't be getting it from Micorosft.
Oh yes, like wagon wheels going backwards. I also pine for the days of scratches, dust spots and pubic hairs on the big screen. And nothing but nothing beats the exhilaration of watching the celluloid melt because the projector stalled.
Slashdot Mirror
That's not data, it's only a representation, can you get it?.
Sorry, the person not getting things here is not me. Copy-on-write is a technique of avoiding changing shared data in place. Let's not get all abstract and confused about that, ok? I mean, you can wank on about your data abstractions, but in the process you will also wank away the computer science and miss the point entirely.
The motivation for not changing the original, shared data in copy-on-write is, we might not know about all the previously existing references to that data, so there may be no practical way to find them all and change them. Microsoft uses a different technique in shadow copy, they do know about all the incoming references from previously snapshots, so they change them to point somewhere else, copy the original data there, then proceed to change the original data. That is not copy-on-write because it does not avoid changing the original, shared data, in place. (See, I spelled out the "in place" thing as a comprehension aid.)
Two very different techniques, with very different performance characteristics. Copy-on-write is O(1) in number of incoming references, while copy-before-write is O(N) in number of incoming references. Copy-before-write requires stalling a write operation for the duration of the copy and metadata update, while copy-on-write does not. Fundamentally different algorithms, as is apparent because of the different complexity characterics. Irrespective of the computer science involved, Microsoft ignores the subtleties and calls the second thing copy-on-write anyway. Opening up plenty of opportunity for wanking on Slashdot by the likes of you. Now, if you want to be precise about it, use the term redirect-on-write every time we actually mean the classic computer science concept copy-on-write (which is documented reasonably well on Wikipedia, except in a certain paragraph containing the word "Microsoft").
As for your "just a representation" argument, let's extend that. A sorted list is "just a representation" of an unsorted list, therefore sorting costs nothing. Oh wait, that's nonsense. Just as your "that's not data" argument is nonsense.
Finally, I hope you agree that copy-before-write performance sucks pretty badly compared to classic copy-on-write (redirect-on-write). And by extension, Microsoft's shadow copy feature sucks badly. Confusing the issue by confusing the terminology does help Microsoft avoid criticism over poor performance. But I would not go so far as ascribing to malice what can be more easily explained by incompetence.
You're not even talking about LVM, you're talking about making multiple databases operate together atomically. In the context of LVM, that's complete nonsense. It's not an LVM concept, it's a high level application concept. Why are you even wasting bandwidth conflating these issues? If you want to do the job with LVM, you concatenate the volumes and run a single filesystem, or single database on the aggregate volume. Conflating this with the application level consistency somebody dragged into the discussion is just idiotic. By the way, you should keep a lid on the hubris about who knows what.
Your frontal cortex is a "stunt". Nature has repurposed wiring originally evolved to filter 2D imagery. Now it makes complex associations and manipulates data in abstract ways that we are only beginning to decode. But it's still a stunt. I understand why you have difficulty comprehending that progress in AI research is in fact progress. It's because you have no comprehension of the long term implications of work that is being done. You should have a look into the relevant scientific literature yourself, and this time don't just let the words slide away like water off the back of a duck.
No, I mean I have been following the scientific progress in that area for about 2 1/2 decades now , and nobody has a clue. Sure, cretins like Marvin Minsky have been predicting AI for decades now, but that is all about grant money, not about any real results or insights.
Every time you talk your way through a telephone menu you benefit from that work. Just because a computer can't yet compose a symphony does not mean there has been no progress. You can hold out for the artificially intelligent poet in your dreamland if you want, while I watch with interest the progress towards creating an automaton as intelligent as an ant (25,000 neurons). Then a mouse (75 million neurons). Finally, as intelligent as you, then it can post rubbish to Slashdot in your place.
Which of course you can do that, but then you can't have the database LV and the log LV on different physical disks any more, which is what was asked. Can you post an example how you would concatenate two existing LVs, with existing file systems on them, mounted and being modified at the time. into a "new virtual block device" without even un-mounting them, and then make a consistent snapshot of them?
You're delusional, "without even unmounting them" appeared nowhere in the discussion above, nor did the concept of making separate filesystems work together atomically. Your assertion about "different physical disks" doesn't make any sense at all. Of course you can combine different physical disks into a single logical volume. You would then create a single filesystem on the logical volume. Look here for examples.
There are probably dozens of smarter grad students they could have hired over this crackpot.
Clearly, you're not one of them.
As nobody even has a rough idea how an AI could be made (hint: it is not a question of computing power), there is little change of anybody making an AI "that could improve itself".
You mean, you do not have even a rough idea. Meanwhile, progress marches on and more and more of the original goals of AI research have been achieved. Machine vision and voice recognition are now commonplace. Computers win at an increasing number of games. Machines walk, balancing realistically. It goes on and on. The only thing that has changed since the beginning is, it's not considered a summer project anymore, the difficulty of engineering at the required scale and with techniques that are discovered and improved only with painstaking work is now properly appreciated. The fact that you have no part in this work doesn't mean much. Researchers in the trenches have considerably more than a rough idea of where they are going. So what if we haven't seen artificial behavior as sophisticated as an ant yet? An ant has 250,000 neurons and we are quite sure that all its behavior is governed by those neurons. Granted, each neuron is an impressively complex system in itself, however there are only so many of them, and increasingly we are gaining the tools to fully reverse engineer them. An ant-scale intelligence is not far away, just by the pedestrian technique of reverse engineering. I would say that the researchers involved in that area have a lot more than a rough idea of where they are going and how to get there.
His synths were decent.
His synths are beyond decent, they are best in class. What's up with you trash talkers? Ray Kurzweil has made his place in history in an amazing variety of ways. But it's kind of sad he's reduced to taking employment with Google instead of running his own show.
Sounds like Btrfs envy. Question is, can they get to work reliably?
Here is an informative post that details why Microsoft's Refs sucks and you don't need to care about it. Even if it works reliably, which is not at all assured (see many reports on the net of issues) this filesystem is pathetically feature poor. What's the point.
You say: "Copy-on-write leaves the original data unchanged" and VSS leaves the original data unchanged.
the implementation details doesn't change the logical concept. COW says: "The fundamental idea is that if multiple callers ask for resources which are initially indistinguishable, they can all be given pointers to the same resource" but it looks like you don't get it.
I did not say that VSS leaves the original data unchanged, I said the opposite. And this is not an "implementation detail", it's a fundamental property of the operation. And could you please read the next sentence after the one you quoted from Wikipedia, it invalidates your argument. And could you please stop chewing on my toes and learn something about computer science.
Dear Microsoft spinmods: you don't change the fact that your volume snapshots suck by modding down my post.
Sure, Microsoft abuses the CoW terminology and Wikipedia documents that. More politely than necessary, IMHO.
Copy-on-write leaves the original data unchanged. Copy on write makes a private copy, leaving the orignal unchanged. Microsoft has a different definition, but then Microsoft has a lot of different definitions. Let's you and me be precise about it, and avoid the terminology that Microsoft has wantonly polluted in its ignorance. Copy-before-write or redirect-on-write.
Sounds like Btrfs envy. Question is, can they get to work reliably?
Modifications in the middle of files are extremely rare. It's true, running a database on top of a snapshotted spinning disk is probably going to suck. For normal users, keeping regular files mostly linear, and files in the same directory nearby each other is what matters, and yes, Btrfs does a credible job of that.
I know why shadow copy works the way it does. 1) It's simple, therefore likely to work. 2) It's an easy answer to the "how do you control fragmentation" question. But the write performance issue is so bad that it's a poor solution no matter how you justify it. It's just an attempt to get away with being lazy for a largely uncritical audience that isn't big into benchmarking, or indeed, isn't used to good disk performance.
lvm lets you concatenate any block devices into a virtual block device
Btrfs, like ZFS and Netapp's WAFL, use a far more efficient copy-on-write strategy that avoids the write penalty.
WAFL doesn't do copy-on-write. Copy-on-write means a write to a block in a file requires the original block to be read, written elsewhere for the snapshot, then the new block written in the original location. That's exactly what WAFL doesn't do. WAFL writes all changed blocks for multiple files in big RAID stripes, updating pointers to current copies and leaving snapshot pointers pointing to old copies of the updated files. Very efficient for writes, but changes almost all reads, random or sequential (within a file) into random reads (within the filesystem) because file blocks get scattered according to write order, not location of the block within the file. That's why they want lots of spindles in an aggregate and they love RAM cache and flash cache.
But since you say that copy-on-write avoids the write penalty I think you know what is does but simply don't know that it isn't copy-on-write.
We both know what we're talking about, we just disagree on terminology. Properly, a "copy-on-write" doesn't modify the original destination. Nobody should ever use the term "copy-on-write" to describe the algorithm that is properly "copy-before-write". The strategy that leaves the original destination untouched and updates pointers to point at the modified copy is correctly called "copy-on-write", but because the terminology has been so commonly abused by the likes of Microsoft and their followers, it is better to be clear and call that "redirect-on-write".
Finally, Netapp gets massive read fragmentation because they suck, not because it can't be avoided.
LVM is kind of cool but if you have a single database spread across multiple LV's then you can't snapshot them all as an atomic operation so it becomes useless.
You're also wrong about that. You can concatenate multiple logical volumes as a single logical volume and snapshot that atomically.
If you keep all old file where in their original sectors and write changes in new places, your files get fragmented to hell.
Microsoft's "shadow copy" doesn't work at the file level, it works at the block level, so it doesn't know anything about files. Btrfs and its ilk try to leave some empty space distributed across the volume, so copy-on-write can leave the copies in fairly reasonable places. After the copy is committed, the original space can be freed, so the next update won't mess things up too badly either. Snapshots mess this up because the original space doesn't get freed. But then, snapshots are always messed up, there is no such thing as a perfect snapshot strategy with respect to disk seeking. Incidentally, with flash you don't care about that any more, there is no seek time.
Anyway, yes, with a crappy copy-on-write (like Netapp's) you get horrible read fragmentation. With an intelligent implementation, it isn't so bad. Note that Btrfs is turning in good benchmarks, including read performance in mixed read/write loads.
VSS is the snapshot solution for NTFS, and of course it uses copy-on-write
Well. Maybe you better sit down in a comfortable chair and think about this a bit. From Microsoft's site: When a change to the original volume occurs, but before it is written to disk, the block about to be modified is read and then written to a “differences area”, which preserves a copy of the data block before it is overwritten with the change.
Think about what this means. It is not a "copy-on-write", it is a "copy-before-write". Gross abuse of terminology if anybody tries to call it a "copy-on-write", which has the very specific meaning of "don't modify the destination data". Instead, copy it, then modify the copy. OK, are we clear? VSS does not do copy-on-write, it does copy-before-write.
Now let's think about the implications of that. First, the write needs to be blocked until the copy-before-write completes, otherwise the copied data is not sure to be on stable storage. The copy-before-write needs to read the data from its original position, write it to some save area, then update some metadata to remember which data was saved where. How many disk seeks is that, if it's a spinning disk? If the save area is on the same spinning disk? If it's flash, how much write multiplication is that? When all of that is finally done, the original write can be unblocked and allowed to proceed. In total, how much slower is that than a simple, linear write? If you said "on the order of an order of magnitude" you would be in the ballpark. In face, it can get way worse than that if you are unlucky. In the best imaginable case, your write performance is going to take a hit by a factor of three. Usually, much much worse.
OK, did we get this straight? As a final exercise, see if you can figure out who was talking nonsense.
...Like the way the photoelectric effect and the ultraviolet paradox were the loose threads that lead to quantum mechanics...
Apparently, some time after being generally accepted, quantum mechanics, was applied to solve the ultraviolet "catastrophe" as opposed to the other way round, while the photoelectric effect played a prime role as you say (Einstein got a Nobel prize for it).
Turns out, the Higgs was never the God particle, it was the Rabbit particle.
True. If you didn't need to know that, why ask?
A store doesn't have to sell you multiple copies of something...
And the proper response to this is to taser her.
Cue Apple employees lining up to agree that it is right for the Apple police to tazer anybody who does not do exactly as Apple wishes. Kicking in the front doors of journalists is another technique that Apple employees agree Apple should use liberally.
And Apple likes to tazer people who ask questions about it.
a 64-bit output isn't really collision resistant anyway
Plenty good enough for a hashed directory key, which doesn't need to be crypticographically secure, just to have good distribution and random results affected as much as possible by all input bits. The size of the output is not the dominant factor, the quality of the input mixing is.
NTFS doesn't have snapshots. Instead it relies on volume shadow copies, with known severe performance artifacts caused by needing to move snapshotted data out of the way when new writes come in. Btrfs, like ZFS and Netapp's WAFL, use a far more efficient copy-on-write strategy that avoids the write penalty. The takeaway: I would not go so far as to claim Microsoft has an enterprise-worthy solution either. If you want something with industrial strength dedup, snapshots and fault tolerance, you won't be getting it from Micorosft.
A lot fo the magic of film was 24fps.
Oh yes, like wagon wheels going backwards. I also pine for the days of scratches, dust spots and pubic hairs on the big screen. And nothing but nothing beats the exhilaration of watching the celluloid melt because the projector stalled.