NT before 2000 was hardly a 'useful' OS. It was Windows, but with an extremely limited set of available software since most things that worked in 95 or 3.x that weren't extremely simple wouldn't work right in NT, if at all. It was buggy, crashed often, even without any third party software or drivers.
I used NT4 from early 1996 until Win2k was released, and it was fine - most Win32 and even Win16 software worked without problem on it (including games), unless they were trying to do something particularly low level like disk checking, or were really just a DOS program wrapped in a cheap suit.
Certainly all the "important" software like Office, Photoshop, etc, I ever tried worked without a hitch.
Right. Because the last 100 years of history show that government can be trusted. Oh wait. No it doesn't! It shows the various governments around the world have killed over 150 million of its own citizens.
Fear that "Big Bad Government" in contemporary Western democracies like the USA, UK, Australia, etc is going to suddenly go off an kill millions of people is, indeed, idiotic.
"Fear of government" should be considered a valid survival instinct just like any other.
In China or Iran, maybe. In the civilised world, not so much.
Regardless, you don't need an "easy fix". No qualified sysadmin uses the stock kernel that came with the distro in any critical server. If there's a patch, you'll just apply it to your sources and recompile. Only desktop users and not-critical services should rely on distro's updates. If you are relying on your distribution's updates for critical fixes on any service even remotely important, you are either fucking nuts or absolutely incompetent.
Or your servers are being used for more important things that running a torrent server in your mother's basement.
If hostile users have local access, you're pretty much boned anyway.
All a hostile user has to do is convince a non-hostile user to trust them. The vast, vast number of successful Windows "exploits" that rely on this process should hopefully provide sufficient evidence that it's not a particularly difficult feat to achieve.
I'm making this amplifying post because the parent post was moderated down in one second. It was born silenced. Obviously there were moderators prepared to prevent you from hearing my response to the question asked. Some of you might for this reason alone find my words above meaningful or intriguing.
Your post was a troll.
Further, you're clearly a passive-aggressive arsehole who revels in the suffering of others, which makes anything you have to say even less credible than it might have been otherwise.
With your penchant for vague, unsupported assertions and callous lack of empathy, you'd be excellent upper management - possibly even C-level - material. That, or a RIAA lawyer.
Nexenta is developed by the people behind the Illumous Foundation, who have created a 'spork' of OpenSolaris, which will continue to import code from each of the source dumps that Oracle has said they will do after each Solaris release, will fix bugs, and will replace the binary-only components of OpenSolaris with open ones.
Like I said, it might take a bit longer for the body to stop thrashing.
Basing your whole product almost entirely on the generosity of another company - especially one not know for being generous - is nuts, and is basically what Nexenta are doing. The only people crazier are the ones buying storage solutions from them.
I hope you see now how a block layer dedup scheme is quite dumb in this scenario, which can thoroughly be properly addressed at the file system layer since files are byte-streams.
The question is how common is how frequently that happens compared to the ones where file-level dedupe is basically worthless - virtual machines, email attachments, databases, in fact basically anything where duplicated data may appear within files (or files masquerading as block devices) but not as discrete files.
I am quite confident those scenarios are vastly more common than one of inserting bytes randomly into the middle of existing files.
Oh that's strange. I've edited a block (ie done a write into the middle of a file) and it failed because the device is now full...
The real problem here is that your disk is full, not that it's being deduped.
However, if you knew your systems monitoring was so tragically bad (/nonexistant), you'd simply have a space guarantee on the volume (or the file) - any system capable of overallocating should support that.
The problem (conveniently illustrated by the below) is that your Windows software, is not under your control. You're a sharecropper, and if the people making the software go away, you're shit out of luck.
For this to be a remotely convincing argument, you need to present a somewhat realistic scenario in which it might happen.
You say it works well, but without the internals, you think it works well-- until it blows up. You think it works well-- because, likely, you don't know enough computing to alter what's inside the black box and make it work better.
The same is exactly true of OSS software. To the end user it's a black box, and if they need help with the internals they have to pay someone else for that help.
Please be honest and serious - there were better implementations of mail transfer agents and email clients before either of those two existed (both are still flaky at times).
That's about a quarter of the functionality Outlook+Exchange offers.
Now, for a different scenario, consider an organization that is just getting started. There are only a few people there, and the whole IT infrastructure still has to be set up. This, I think, is a scenario where free software can be very successful. It's also an interesting scenario to think about. Suppose you wanted to set up the IT infrastructure for at least a few hundred users, most of whom would have jobs where they have to use computers, without necessarily having any affinity for computers themselves. Assume you would need some common infrastructure: e-mail for everyone, calendaring would be very useful, and at least some desks will have computers that any among a group of people will have to be able to log into and get to work with (i.e. they won't have their own desk and their own computer). How would you do it?
If you wanted to go Linux ? By contracting out to people who were very knowledgable and skilled about setting up green-field Linux desktop/backoffice environments. Which, given their rarity, are going to be much more expensive than their Windows or Mac counterparts, and therefore not result in any cost savings overall.
We just converted from Xiotech to NetApp, and the NetApp is crap. "High end" isn't how we would describe NetApp. And their sales people lied to us (er, said things that may technically be true but are about as honest as 'pigs CAN fly with sufficient initial velocity').
For example ?
They also claimed that de-dupe would save us 50% storage space. Lies.
The fly in the ointment for this all is btrfs. My feeling is that by the time we get a mature not slow as shit port of zfs to linux, btrfs will have matured to a point that it'll have all the features of zfs I like (which it at this point already has) such as snapshots, clones, etc, plus deduping.
So far as I know, btrfs currently doesn't support parity-based RAID schemes and dedupe. I'd estimate it's 3-5 years away from being "production ready".
If you are relying on getting a fast replacement disk from *ANY* vendor to assure that you don't suffer data loss from disk failures then you are doomed from the get go.
I'm not. It's called risk minimisation.
Your array needs to have a sufficient number of global hot spares to automatically replace any failed disk. At which point the four hour response time is not actually worth it (do you really want to get up a 03:00 on Christmas Day to replace a disk?) and next day is just fine.
Whether I want to not is not particularly relevant. I'm getting *paid* to do it because it's my job.
If you came to me and said a disk failed but we don't need to bother replacing it for a few days because there is (or was) a hot spare, you'd be lucky to walk away still employed.
No, filesystems should be doing this. Then you get the benefits regardless of which hardware you want to put on the botton.
If you make it filesystem-specific, the benefits can only be realised by systems that can use that filesystem. Further, it means the data under consideration for dedupe must be within that filesystem.
When you're talking about centralised storage - which we are - you want the dedupe happening low in the storage stack so the benefits are spread across the widest range of systems and datasets.
Isn't not just small to medium sized businesses; most tech companies, even really huge ones, don't buy this kind of enterprise equipment.
I think you'll find they do. Small businesses less so, but certainly medium and up will almost certainly have something from the usual suspects of NetApp, EMC, IBM, Equallogic, etc.
You won't find any of it at Google or Amazon, for example, even though they are quite large.
Not really a good example. Both companies are huge and therefore have (more than) sufficient staff to both engineer and support their own internally developed solutions. Further, they can probably derive a business benefit from those developments, something rarely true for the average company, especially when talking about generic, commodity functionality like storage.
For businesses in the meat of the bell curve, most technology is a means to an end, not an end in itself. This is true even for "technology-driven" businesses, since the bulk of their IT functions are neither unique nor interesting. A couple of hundred grand of CapEx for a mid-range NetApp filer is not a lot compared to the couple of hundred grand *per annum* OpEx you'll need to spend on dedicated staff to develop, support and maintain something equivalent in-house, especially when taking into consideration how vulnerable you become when they decide to jump ship to another employer, and the fact that investment is almost certainly not delivering any direct benefits to the business.
The "big multiple in price" is really accurate either. I quickly priced out a rough equivalent to our 3140 in a 7410 with 4x20-spindle shelves, 2 "read accelerators", some 10GbE cards and 4Gb HBAs and it came out at over $250k. Obviously that's before discounting, and it has more/faster CPUs, but it's certainly in the same _ballpark_ as the ~$175k we paid.
I also just realised that $250k is probably not including support, which is likely to be knocking on the door of 6 figures for 3 years of 24x7x4 support. Half of the price of our NetApp was the support contract.
Are they really that superior to the Sun storage products (the ones Sun invented ZFS for) to be worth the big multiple in price? I mean, Sun isn't stuff-you-put-together-at-Frys prices either, but it's a lot cheaper than EMC or NetApp.
The issues that will give most people pause are those of maturity and future. Sun's solution hasn't been around for very long, and some features like FCP target and dedupe are *very* new. There's also something of a question mark over where it's going in the future with Oracle's acquisition of Sun.
The "big multiple in price" is really accurate either. I quickly priced out a rough equivalent to our 3140 in a 7410 with 4x20-spindle shelves, 2 "read accelerators", some 10GbE cards and 4Gb HBAs and it came out at over $250k. Obviously that's before discounting, and it has more/faster CPUs, but it's certainly in the same _ballpark_ as the ~$175k we paid.
That's still a particular type of compression, isn't it?
Not really. Compression is taking a chunk of data and replacing it with a different, smaller chunk of data plus instructions (albeit in abbreviated form) about how to turn it back into the original chunk of data. Dedupe is taking a chunk of data and replacing it with a pointer to a "remote", identical chunk of data.
Compression is nearly always applied on a per-file basis, whereas dedupe is applied on a per-volume basis. Conceptually similar to the difference between compressing every individual file in a volume vs tar.gz-ing the whole volume.
Compression is generally much more limited over the "window" of data it can work with at any time (eg: if you have two similar bits of data at the start and finish of a file, but they're separated from each other by 10 gigabytes of other data, then they'll probably be compressed independently).
Consider also the difference in processing necessary to decompress data vs dereference a pointer on disk. Some compression schemes (eg: h.264) are quite CPU intensive to decompress, while the overhead for reading deduped data is basically zero.
Slashdot Mirror
NT before 2000 was hardly a 'useful' OS. It was Windows, but with an extremely limited set of available software since most things that worked in 95 or 3.x that weren't extremely simple wouldn't work right in NT, if at all. It was buggy, crashed often, even without any third party software or drivers.
I used NT4 from early 1996 until Win2k was released, and it was fine - most Win32 and even Win16 software worked without problem on it (including games), unless they were trying to do something particularly low level like disk checking, or were really just a DOS program wrapped in a cheap suit.
Certainly all the "important" software like Office, Photoshop, etc, I ever tried worked without a hitch.
Right. Because the last 100 years of history show that government can be trusted. Oh wait. No it doesn't! It shows the various governments around the world have killed over 150 million of its own citizens.
Fear that "Big Bad Government" in contemporary Western democracies like the USA, UK, Australia, etc is going to suddenly go off an kill millions of people is, indeed, idiotic.
"Fear of government" should be considered a valid survival instinct just like any other.
In China or Iran, maybe. In the civilised world, not so much.
The anger and fear that corporations will take over is silly.
Indeed. Much like worrying about whether your gate is secure is silly once all your cows are already wandering off down the road.
Regardless, you don't need an "easy fix". No qualified sysadmin uses the stock kernel that came with the distro in any critical server. If there's a patch, you'll just apply it to your sources and recompile. Only desktop users and not-critical services should rely on distro's updates. If you are relying on your distribution's updates for critical fixes on any service even remotely important, you are either fucking nuts or absolutely incompetent.
Or your servers are being used for more important things that running a torrent server in your mother's basement.
If hostile users have local access, you're pretty much boned anyway.
All a hostile user has to do is convince a non-hostile user to trust them. The vast, vast number of successful Windows "exploits" that rely on this process should hopefully provide sufficient evidence that it's not a particularly difficult feat to achieve.
I'm making this amplifying post because the parent post was moderated down in one second. It was born silenced. Obviously there were moderators prepared to prevent you from hearing my response to the question asked. Some of you might for this reason alone find my words above meaningful or intriguing.
Your post was a troll.
Further, you're clearly a passive-aggressive arsehole who revels in the suffering of others, which makes anything you have to say even less credible than it might have been otherwise.
With your penchant for vague, unsupported assertions and callous lack of empathy, you'd be excellent upper management - possibly even C-level - material. That, or a RIAA lawyer.
Why does the summary and articles read like a paid advertisement for Ksplice?
Because it's being read by someone who has a juvenile fascination with server uptime ?
Nexenta is developed by the people behind the Illumous Foundation, who have created a 'spork' of OpenSolaris, which will continue to import code from each of the source dumps that Oracle has said they will do after each Solaris release, will fix bugs, and will replace the binary-only components of OpenSolaris with open ones.
Like I said, it might take a bit longer for the body to stop thrashing.
Basing your whole product almost entirely on the generosity of another company - especially one not know for being generous - is nuts, and is basically what Nexenta are doing. The only people crazier are the ones buying storage solutions from them.
I hope you see now how a block layer dedup scheme is quite dumb in this scenario, which can thoroughly be properly addressed at the file system layer since files are byte-streams.
The question is how common is how frequently that happens compared to the ones where file-level dedupe is basically worthless - virtual machines, email attachments, databases, in fact basically anything where duplicated data may appear within files (or files masquerading as block devices) but not as discrete files.
I am quite confident those scenarios are vastly more common than one of inserting bytes randomly into the middle of existing files.
Oh that's strange. I've edited a block (ie done a write into the middle of a file) and it failed because the device is now full...
The real problem here is that your disk is full, not that it's being deduped.
However, if you knew your systems monitoring was so tragically bad (/nonexistant), you'd simply have a space guarantee on the volume (or the file) - any system capable of overallocating should support that.
The problem (conveniently illustrated by the below) is that your Windows software, is not under your control. You're a sharecropper, and if the people making the software go away, you're shit out of luck.
For this to be a remotely convincing argument, you need to present a somewhat realistic scenario in which it might happen.
You say it works well, but without the internals, you think it works well-- until it blows up. You think it works well-- because, likely, you don't know enough computing to alter what's inside the black box and make it work better.
The same is exactly true of OSS software. To the end user it's a black box, and if they need help with the internals they have to pay someone else for that help.
Please be honest and serious - there were better implementations of mail transfer agents and email clients before either of those two existed (both are still flaky at times).
That's about a quarter of the functionality Outlook+Exchange offers.
Now, for a different scenario, consider an organization that is just getting started. There are only a few people there, and the whole IT infrastructure still has to be set up. This, I think, is a scenario where free software can be very successful. It's also an interesting scenario to think about. Suppose you wanted to set up the IT infrastructure for at least a few hundred users, most of whom would have jobs where they have to use computers, without necessarily having any affinity for computers themselves. Assume you would need some common infrastructure: e-mail for everyone, calendaring would be very useful, and at least some desks will have computers that any among a group of people will have to be able to log into and get to work with (i.e. they won't have their own desk and their own computer). How would you do it?
If you wanted to go Linux ? By contracting out to people who were very knowledgable and skilled about setting up green-field Linux desktop/backoffice environments. Which, given their rarity, are going to be much more expensive than their Windows or Mac counterparts, and therefore not result in any cost savings overall.
They do the same thing at every upgrade [...]
No they don't.
5 year olds can use Linux/GNOME/OpenOffice. So what's the problem with adults?
5 year olds have nothing to lose by getting it wrong.
We just converted from Xiotech to NetApp, and the NetApp is crap. "High end" isn't how we would describe NetApp. And their sales people lied to us (er, said things that may technically be true but are about as honest as 'pigs CAN fly with sufficient initial velocity').
For example ?
They also claimed that de-dupe would save us 50% storage space. Lies.
On what sort of data ?
The fly in the ointment for this all is btrfs. My feeling is that by the time we get a mature not slow as shit port of zfs to linux, btrfs will have matured to a point that it'll have all the features of zfs I like (which it at this point already has) such as snapshots, clones, etc, plus deduping.
So far as I know, btrfs currently doesn't support parity-based RAID schemes and dedupe. I'd estimate it's 3-5 years away from being "production ready".
If you are relying on getting a fast replacement disk from *ANY* vendor to assure that you don't suffer data loss from disk failures then you are doomed from the get go.
I'm not. It's called risk minimisation.
Your array needs to have a sufficient number of global hot spares to automatically replace any failed disk. At which point the four hour response time is not actually worth it (do you really want to get up a 03:00 on Christmas Day to replace a disk?) and next day is just fine.
Whether I want to not is not particularly relevant. I'm getting *paid* to do it because it's my job.
If you came to me and said a disk failed but we don't need to bother replacing it for a few days because there is (or was) a hot spare, you'd be lucky to walk away still employed.
No, filesystems should be doing this. Then you get the benefits regardless of which hardware you want to put on the botton.
If you make it filesystem-specific, the benefits can only be realised by systems that can use that filesystem. Further, it means the data under consideration for dedupe must be within that filesystem.
When you're talking about centralised storage - which we are - you want the dedupe happening low in the storage stack so the benefits are spread across the widest range of systems and datasets.
Isn't not just small to medium sized businesses; most tech companies, even really huge ones, don't buy this kind of enterprise equipment.
I think you'll find they do. Small businesses less so, but certainly medium and up will almost certainly have something from the usual suspects of NetApp, EMC, IBM, Equallogic, etc.
You won't find any of it at Google or Amazon, for example, even though they are quite large.
Not really a good example. Both companies are huge and therefore have (more than) sufficient staff to both engineer and support their own internally developed solutions. Further, they can probably derive a business benefit from those developments, something rarely true for the average company, especially when talking about generic, commodity functionality like storage.
For businesses in the meat of the bell curve, most technology is a means to an end, not an end in itself. This is true even for "technology-driven" businesses, since the bulk of their IT functions are neither unique nor interesting. A couple of hundred grand of CapEx for a mid-range NetApp filer is not a lot compared to the couple of hundred grand *per annum* OpEx you'll need to spend on dedicated staff to develop, support and maintain something equivalent in-house, especially when taking into consideration how vulnerable you become when they decide to jump ship to another employer, and the fact that investment is almost certainly not delivering any direct benefits to the business.
The "big multiple in price" is really accurate either. I quickly priced out a rough equivalent to our 3140 in a 7410 with 4x20-spindle shelves, 2 "read accelerators", some 10GbE cards and 4Gb HBAs and it came out at over $250k. Obviously that's before discounting, and it has more/faster CPUs, but it's certainly in the same _ballpark_ as the ~$175k we paid.
I also just realised that $250k is probably not including support, which is likely to be knocking on the door of 6 figures for 3 years of 24x7x4 support. Half of the price of our NetApp was the support contract.
Are they really that superior to the Sun storage products (the ones Sun invented ZFS for) to be worth the big multiple in price? I mean, Sun isn't stuff-you-put-together-at-Frys prices either, but it's a lot cheaper than EMC or NetApp.
The issues that will give most people pause are those of maturity and future. Sun's solution hasn't been around for very long, and some features like FCP target and dedupe are *very* new. There's also something of a question mark over where it's going in the future with Oracle's acquisition of Sun.
The "big multiple in price" is really accurate either. I quickly priced out a rough equivalent to our 3140 in a 7410 with 4x20-spindle shelves, 2 "read accelerators", some 10GbE cards and 4Gb HBAs and it came out at over $250k. Obviously that's before discounting, and it has more/faster CPUs, but it's certainly in the same _ballpark_ as the ~$175k we paid.
That's still a particular type of compression, isn't it?
Not really. Compression is taking a chunk of data and replacing it with a different, smaller chunk of data plus instructions (albeit in abbreviated form) about how to turn it back into the original chunk of data. Dedupe is taking a chunk of data and replacing it with a pointer to a "remote", identical chunk of data.
Compression is nearly always applied on a per-file basis, whereas dedupe is applied on a per-volume basis. Conceptually similar to the difference between compressing every individual file in a volume vs tar.gz-ing the whole volume.
Compression is generally much more limited over the "window" of data it can work with at any time (eg: if you have two similar bits of data at the start and finish of a file, but they're separated from each other by 10 gigabytes of other data, then they'll probably be compressed independently).
Consider also the difference in processing necessary to decompress data vs dereference a pointer on disk. Some compression schemes (eg: h.264) are quite CPU intensive to decompress, while the overhead for reading deduped data is basically zero.
Of course, as deduplication is mainly a solution for enterprises that have been tricked into buying obscenely expensive storage, [...]
If we were "tricked", what is the cheaper, but equally capable alternative ?
Actually, this feature is a recent addition to ZFS, and it's the main reason I'm interested in putting ZFS on my file server.
You'll probably be disappointed. Dedupe savings for the kind of stuff you'd typically find on a home file server are miniscule.