"You want to save money by replacing an eight hundred dollar 300GB SCSI enterprise class hard drive with a FOUR THOUSAND DOLLAR 400GB (WITH COMPRESSION) Quantum LTO-3 tape drive?"
First of all, LTO-3 is 400GB Native/800GB with 2:1. They can do 80MB/sec sustained for uncompressed streams.
Second of all, you can get new LTO-3 tapes for less then $100 each. They're still new, so they're still a bit expensive. Of course, that compares very favorably to a used, low-end 300GB, OEM SCA SCSI disk at $500. Chances are you'll need to buy an approved disk with special housing or risk voiding your warranty - which easily doubles the price, or more. And you can't seriously buy used disks for production backups. And - I'd love to see a single hard disk stream 80GB/sec for the entire 300GB, especially at bargain basement prices. SATA drives will yield less performance, albeit for less money. But again, you'll be hard pressed to insert a SATA drive you get at CompUSA into most storage cabinets.
It's not the cost of the drive, but rather how much you can back up with them when they're utilized in libraries and longer term storage, the ease of swapping media, and the feasibility of off-site storage. You can't swap out hard drive spindles from a hard drive. And what? You get a free drive shelf with every 10 disks you buy or something? Those things are expensive, even for a lower cost, older tech SCSI enclosure.
Let's do a little simple math here:
Drive enclosure: $5000 (PV220S) per 14 slots, filled with 14 100GB drives @$800: $16,200 Total space: 4200MB (uncompressed)
LTO-3 10 slot library with one drive: Adic 10-slot library, $7000. 10 tapes: $900. Total: $7900. Total space: 4000MB (uncompressed)
You could buy 90 more tapes for the extra cost of the disk array, which is another 36TB (uncompressed.)
Disk backups are not cheaper then tapes, so don't even try to make that argument.
"Also, why do you think Quantum is making hard disk backup systems now? Because the market is demanding it, that's why."
You've already said that. And they don't replace tape backup systems, they compliment them. They don't make sense for long-term storage. Unless, such as in your case perhaps, you're backing up a single SBS server or something. For very small shops, you might get away with a Maxtor "single button" backup USB drive. But as you say, if you need full redundancy, you'll have to buy a few of them and the costs add up fast. You'd be better off with a lower end tape drive.
"hard drives ARE far more reliable than tape - especially if they are only being used for backup"
Are you sure about that? Do you know of a SCSI enclosure that spins down disks when they're not being actively used? Otherwise, it doesn't really matter. Hard drive failures are about equal in idle (running) drives versus actively used ones.
I agree that hard drives are less prone to failure then tapes as a general rule, but the risk is great enough in both mediums that the difference doesn't even matter. Usually, disk backup systems are in a raid configuration - which of course makes it all but stupid to try and take raid sets off site on a regular basis for off-site storage. Plus, they're more fragile in transit, heavier, and don't hold as much as tapes.
"What I said was that sound backup and recovery practice precludes using archiving and compression together for critical data - and if you do it, you'd better have redundancy"
Wait - so if you have critical data, but you don't use compression, you don't need redundancy? I know that's not what you mean, but I feel as though the risks are equal regardless of whether or not you compress; meaning, you need to take the same precautions either way.
"which means PAR files which means increased complexity and cost for the dubious benefits of archiving"
I thought we got past this. The conversation evolved into tape backup/disk backup systems. Neither of which would ZIP your files in
No, you tell ME, Mr. Drive Speed Expert, of one *single* drive that can perform at anywhere near what tape systems are capable of these days? You mentioned that a SAN could perform well, in a RAID CONFIGURATION. And only maybe. You mentioned pulling a single drive and moving it off site. How do you propose doing this in an enterprise enviornment?
You mention small business for a hard drive going off site - how much have I talked about small business? None!
"Drop a hard drive? Irrelevant. Drop a tape? Irrelevant. Acidents happen and aren't relevant to the discussion."
Suddenly, it's irrelevant. Then so is compression! Right? What about the weight? Do *YOU* want to carry 20 or 40 hard drives around?
"And I damn sure don't want you backing up MY enterprise with that kind of attitude about backup: save time, money and media - and lose the data."
Ohh, snap. Ouch. That hurts. Here's a NEWSFLASH: time, money, and media are hugely important to a company, no? So why not get all the benefits of a modern tape backup system? Because you're a fool. Somehow, running compression on a backup kills backups. I prefer a sound backup methodology over irrational fear.
"Comparing the probability of bad media to lightning and nuclear weapons is just stupid."
Yes, and I said it was if you actually bothered to fucking read it. Your argument against compression is equally baseless and bullshit.
You ignore points that you agree with, and you harp on the little things that you ignorantly don't. Over the past posts on this topic I've discussed the basics of any good backup practice. You can't see it because you haven't worked in the enterprise. Hey, fuck it. I'm arguing with dumb and dumber here.
Points you've made:
- Take entire SANS off site for safe keeping. - Don't use compression, it's evil. No better reason. - Tapes are the devil. Use USB drives instead. - Hard drives never go bad. - Forensic bit-recovery is better then a sound backup and recovery practice.
Unless you install a client piece on the customer computers, it would be pretty easy to thwart such bandwidth limiting, service limiting restrictions. You can cloak the client PC's with a linux box, and chances are good that there would be little linksys-like routers available to do the same for the less technically savvy. I wouldn't be surprised if it became a check-box on common for-home devices, and that it would be enabled by default.
Of course, they could also monitor traffice in and out of an IP and watch to see if there's spy/malware type things going on, which a cloak wouldn't mask. In which case, they should notify the end-users, not restrict them without doing so.
We'll see how this plays out. The trend is toward more speed, more speed, and I don't see that changing anytime soon. If a malware infected PC's user doesn't know he/she has it, and internet service becomes slower because the cable company reduces the speed, the user will just think the service sucks and switch to DSL or whatever else.
After the whole thing with Sony's music CDs, I wouldn't be surprised if they've had to change some aspects of the copy protection - namely removing stealth copy protection mechanisms. Because you know it was on their minds, and probably already coded.
No, I'm not saying that files should be archived first, and then compressed onto tapes. There were two points of discussion - one was compressed archives, and it moved on to tape backup compression.
Most tape drives these days employ hardware-based compression to ease the burden on the backup servers. It makes more sense as speeds increase on the drives faster then the CPU on the backup servers. This rings true more so when you stream multiple jobs onto multiple tapes simultaneously to maximize throughput. Each file or unit that's backed up to tape is compressed on the fly, individually.
I don't agree that the compression has to occur outside of the backup software. While it may be true that there could be a bug, the same is true wherever you position it. Rest assured, IBM, Legato and Veritas have pretty much worked through compression bugs a decade ago or more..
It's not inherently true that one error in an archive means the archive is blown. A simple recovery record in the archive, available in most archivers, takes care of this handily. Even if you didn't do that, however, a little corruption doesn't mean the entire archive is hosed. It does depend on the archiver to a degree, of course. And if you must do this before backing up to tape, you can include parity files to mitigate risk of corruption.
You're not asking for trouble as long as you take steps to mitigate risk. That's what good BURP (Back-Up and Recovery Practices) is all about.
Your post pretty much proves your inexperience in enterprise backup systems, or in IT systems in general.
"a standard tape backup isn't reliable and never has been. It's also too damn slow - which is why disk to disk backup is taking hold"
Tape backups are pretty damned reliable these days, and very fast. You have to take care to pay attention to tape lifespan and any drive problems that might arise from time to time. Small computer tape systems starting with AIT/DLT have set the bar for reliability - LTO and SDLT have taken it a step further with leaders that hardly ever break. LTO2 tape drives can perform at 40MB/sec sustained with no compression. With 2:1, you'll get 80MB/sec sustained transfer rates. LTO3 doubles these figures, and LTO5 doubles them again. There's even an LTO6 on the horizon that will be able to sustain 540MB/sec with 2:1 compression. Do you know how many SATA hard disk spindles you'd need to even come close to these figures?
Quite often, backups to tape are faster then disk based backups. Disk backup systems *DO NOT* replace tape systems. They supplement them. They provide faster access to backups - no moving tapes around a library and you get random access. There's definite advantages to disk backups - especially when you introduce a snapshot technology like EMC's BCV. I use disk backups where I work, and make copies to tape for longer-term storage and off-site storage.
You can't take hard drives off-site. With single tapes holding up to 800GB of data in a smaller and lighter form factor, it's a no-brainer. You can drop a tape and it'll be okay - and you can even repair some tape cartridge damage. Drop a hard drive and you're screwed. Plus, who wants to carry 20 hard drives around to bring them off-site? They're heavy! Not to mention I've never seen a hot-plug bar-coded hard drive library..
"I have no objection to using compression to send data over the Net or a network - the protocols take care of reliability there (although I suppose there is a small chance of corruption as well which should be taken into consideration.)"
So you're willing to accept that corruption could occur over the network, but compression is okay here. Sounds contradictory to your argument.
"actually have no problem with compression if the redundancy is sufficient. In many cases, a simple double backup will be enough"
In fact, in most cases a single backup is enough in itself - you're not overwriting your backups every day are you? Especially if you backup to disk first, which would be your first copy.
"And this begs the question, which is - what IS a proper backup system and procedures?"
Perhaps a little more experience with enterprise level backup and recovery would be more sufficient then a response from me here. I covered the very basics to get you started in my other posts on this thread. At the very least, your backup system should be well documented and periodically tested using stand-by systems for restoring data.
"As I keep reiterating, the point of backup is RECOVERY. If you can't recover, because of compression or archiving, none of the justifications for using them are relevant."
And if lightening strikes, and blows up the building, it's all for nothing too. Better move underground. And if terrorists set off a nuke, you better have a bunker under there. Actually, turn the computers off, unplug them, and submerge them in concrete. That way, you won't have to worry about corruption. You better not drive to work anymore either, because if you life is of any value to you, you'd avoid the risk of a backup issue that you would be too dead to fix. Better not trust Iron Mountain with your tapes either, because the truck could get into an accident. It's all stupid, I know, just like avoiding compression because forensic recovery of a file MIGHT be more difficult. The idea is to mitigate corruption problems to begin with and have contingency plans, not deal with forensic recovery of corruption when it hits you.
Hey, do your uncompressed backups if you need that level of paranoia - I've made my point and I don't see that it needs to continue being made.
Are you that ignorant of how backups work? Each file is compressed individually to tape or disk backup files.
Are you also ignorant that the internet isn't all 100Mbit? You need compression or you'll be wasting a lot of time. And if you need to ensure integrity, you can use a recovery record in the archive and parity files on the outside.
BILLIONS of compressed files are transferred over the internet every day without issue.
If you take PROPER PRECAUTIONS instead of relying on forensic recovery you won't put yourself in a position to require it.
Are you ignorant to risk mitigation? Are you ignorant to the fact that some risks are so insignificant that you shouldn't nerf your whole system because of it? How hard is that to understand? Disabling compression on backup or file transfer systems is a ridiculous idea.
I appreciate the fact that it's more difficult to reconstruct a compressed file over an uncompressed one, although not impossible in the hands of the right people.
What I'm saying is that to forgo compression - with it's numerous advantages and only one questionable benefit - is a real silly thing to do. You've eluded to the fact that compression in fact causes corruption - which is inherently false (although you've backed off this arguement a bit.) And you seem to ignore the point that speed and capacity are huge pros for compression and have any bearing on the arguement.
I continue to believe that with a proper backup system and procedures, you'll never encounter a time when you'll have to reconstruct a corrupt file. I'll qualify that statement with this: Unless the file was corrupted from the source media. If you backup a hosed database, it'll be hosed on tape, and compression didn't play any part of it.
You began this discussion with "I just can't fathom why a responsible admin would risk the possible data corruption that could come with compression" and I felt compelled to respond to this insulting statement. Compression isn't corruption - it's a staple of large data storage and it has been for decades. I just hope I never end up on the job with someone of the same opinion. I take data integrity very seriously, and your arguement against compression has little credibility.
Really: if you truly need that level of data paranoia (I can only think of maybe one or two institutions that might) a standard tape backup system just won't do for you anyways. There's other ways to ensure data integrity besides full-on backups, and they're designed with high availability and high integrity in mind. EMC's SRDF (in conjunction with a couple of Symms) and some of the Veritas replication tools help with these types of requirements. Of course, all of these systems employ some sort of compression to send the data over the wire.
You might be right about that - I'd not thought that it was a civil case. However, the courts seem generally receptive to requests for council in a case where it's Megacorp vs Jane Doe. You might not get a full PD, though.
Either way, it's all bass ackwards. We shouldn't need to have studied law for ten years to defend ourselves in court - the laws are so damned complicated. Lawyers are paid a lot because they've had to specialize in law for a good part of their lives. They have to attend school forever just to get a grip on it. I don't fault them for charging the high rates that come from such specialization. (Of course, there's a lot of bad ones that really abuse the system for large sums of money.)
Just because you might have a huge library of porn that you need to back up, doesn't mean all of us do too.
Explain to me why and/or how compression reduces data integrity? In fact, I'd argue that it's the other way around in many cases where compression is appropriate since you'll know if corruption occured due to errors in decompression. Otherwise, you might not have as much fair warning. You do realize that we're talking lossless compression, not lossy (like JPEG) compression?
I don't undertstand why you think compression automatically destroys the chance of recovery? And how encoding in ASCII is better? What's the thing about "sectors"? I never said using a compressed volume on a hard disk was a good idea. Compressed files can be recovered too, you know. If you have the forensic expertise to recover a corrupted non-compressed file, changes are you'd also be able to recover the data from a compressed one.
The only arguement for compression is not the cost of media - in fact I didn't mention media price at all. I did mention the library capacity, however - and getting an even bigger library is a lot more expensive of a prospect then the $.75 you quoted per GB. Did you read the whole part of my post about speeds? If I can restore that database in half the time because of compression, that means less down time and less money lost. (Although, the money-lost factor doesn't really apply at a government institution; we're not selling anything.)
"If you backup more than once UNCOMPRESSED, you can recover almost anything because it is VERY unlikely that a bad sector will occur in the exact same spot or even in the same file (assuming the one file does not take up most of the specific media.)"
Wouldn't this apply to a compressed backup, too? You're assuming here that the file was unchanged in between the two backups - thus it would apply to any data, compressed or not.
"Alternatively, use PAR files to recover - as long as you're willing to add the extra space and time - which sort of obviates the advantage of compression, doesn't it?"
No - it simply lowers the compression ratio a bit. If you're getting 2:1 compression and add 10% pars, you're still looking at a 1.8:1 compression ratio, but with recoverability.
----
Within every IT budget, you must balance out the speed, recoverability, and cost of your backup solution.
In your solution of never using compression (since no admin should do that, you mentioned) you lose a lot of speed in backups and restores. Speed of recovery is a key factor in many enviornments. It's often the top question asked when in discussion of new backup solutions. You talk about this as an important point yet excluding compression could double your restore times, or more. Not to mention backup speeds - if you can take your backups in half the time, you effectively double the number of servers you could backup in the same amount of time. Or, you reduce the amount of time servers are busy with backups.
Recoverability is big - you want your backups to be reliable. Most of the time, any corruption is unacceptable, be it in a compressed file or not. It's either good or you throw it out and go back to the previous backup. Many IT shops are doing multiple backups these days - backup to disk first, then to tape. Then take snapshots of those tapes and bring them off-site. Compressed or not, testing your backups and ensuring you have no problems with hardware is much more effective then using uncompressed backups and performing forensics on them if they're bad. Speaking of which, I don't see why compressed data would be less recoverable.
Finally, you have cost. Yes, even when data recoverability is a key factor, you still have to consider cost. So, what makes more sense? Using uncompressed backups that will backup and restore slower, cost a lot more for media and library capacity, and cause more personnel overhead for swapping tapes - or using compression and cutting all that in half? You'd rather lose all that in the off chance that MAYBE you could recovery more of your data, in the off chance that NONE of your other backups are good? I don't know any resposible IT manager that could agree with you.
A proper backup and recovery plan with periodic testing and multiple copies held on-site and off is a much more effective solution then betting on forensic recovering of uncompressed data.
Hey, I'm not claiming that compression is always right in every situation. That's far fro
I agree - I believe Microsoft is still a Monopoly on the desktop. And perhaps more so with Office. While yes, there's alternatives, there's always been alternatives. It's the viability of the alternatives.
Microsoft still has the power to crush everything in one fell swoop. They could raise prices 400%, and a LOT of people would have no choice but to pay it. They could completely change all of their document formats, and change them on open. They could change IIS and IE to work well togher, but not with anything else. You could switch to something else, but then all your documents become unusable. Or at least, many of them. You'd have a hard time online.
They don't do it because they could get really stomped on legally, but they still do it in small ways. They still try to hang on to their same monopolistic practices of locking people in, rather then providing a better product or service.
All I see is ads. I think I found a paragraph that looked like it may have been the article, but every other word was underlined with an ad-link so I didn't think that was it..
If you're familiar with Usenet, you've probably encountered PAR files from time to time. A PAR file is a parity file which can be used to reconstruct lost data. It works sort of like a RAID, but with files as the units instead of disks.
Let's say you have a 200MB file to send. You could just send the 200MB file, with no guarantees that it will reach the destination uncorrupted. Or, you could use a compression program and bring it down to 100MB. In this case, even if you lost the first transfer, you could transfer it a second time. Then we look at PAR. You compress the 200MB file into ten 10MB files. Then, you could include 10% parity - if any of your files is bad, you'd be able to reconstruct it with the parity file. With only 110MB of transfer. PAR2 goes even further by breaking down each file into smaller units.
Besides transfer times and correction for network transfers, compression can also increase speeds of transfer to mediums. If you have an LTO tape drive that can only write to tape at 20MB/sec, you'll only ever get 20MB/sec. Add compression to the drive, and you could theoretically get 40MB/sec to tape with 2:1 compression. That means faster backups, and faster restores. On-board compression in the drives takes all the load off the CPU - but even if you use the CPU for it, they're fast enough to handle it.
Not to mention, it takes a lot less tape to make compressed backups. I don't know what world you live in, but in mine, I don't have unlimited slots in the library and I don't want to swap tapes twice a day. Handling tapes is detremental to their lives; you really want to touch them as least as possible.
Data corruption isn't caused by compression. If it's going to happen, it'll happen regardless. While your point is true that it MAY be more difficult to recover from a corrupt file, that's not the right methodology. If your backups are that valuable, you'd make multiple copies - plain and simple.
I can't fathom why a responsible and well informed admin would avoid compression.
"While the figure represents existing salaries and equipment drawn together under one roof, it also represents some shift in emphasis by IBM from Linux back to its mature Unix operating system."
How do you figure that? If all it is is shifting the books around, I don't see how it's changing emphasis on anything. Sounds more like simply reorganizing in the name of effenciency.
I share much of the same philosophy- I'll take that bundle of cash even if the job sucks, and it will enable me to do whatever I want when I'm not there.
However, while earning lots of money offsets the will to quit a bad job, it doesn't make it any easier to put in the 40 hours. We spend a lot of our lives at work.
Sometimes it's not worth it. If you're at a shitty 80k/yr job, and you have the chance to get a much more enjoyable job that's closer to home for 70k/yr, a lot of people might take it.
I agree that the punishment needs to be enough of a deterrent to dissuade any would-be lawbreakers, but I also believe that the punishment should fit the crime. I don't believe prison is the answer for many crimes. There's other forms of punishment - it doesn't always have to be prison time. People can screw up, and I don't think they should pay for it in years of jail time and a thoroughly ruined life (for a time anyways.)
Of course, I'm still talking about non-violent crimes here.
It's the repeat offenders that need more serious punishments, but even then I just can't justify sending people away is always the right move for many non-violent offenders. However, I guess in some cases it might be what's needed to "correct" someone. I just don't think we should be handing out years in jail to people as easily as giving out candy to school kids. It shouldn't be taken so lightly.
The fact is, tougher sentences have had very marginal impact on crime rates when applied. What criminal plans on getting caught? If someone breaking the law even seriously considered how much of a pain in the ass all the legal would be, it would probably be enough. But they don't. Murder rates in states with the death penalty doesn't decline, DUI doesn't decline with harsher penalties, and even the 3rd strike rule in California hasn't made any big impact on the number of robberies.
I don't necessarily agree that without strict laws, crime would run rampant. I'm not saying that crime shouldn't be taken seriously, of course, nor that we don't need laws. What I'm against is the trend toward unnecessarily harsh punishments. I guess the issue is what the definition of that is- and for many violent crimes I don't feel as though the punishment is harsh enough. 5 years for a woman that kills her baby in a fit of rage? And 5 years for a guy that sells modded xboxes with some pirated games? Something doesn't seem right there.
Quantum was always good for me, so was Seagate. Unfortunately, every Maxtor drive I ever owned went to shit within the year. I've had WD drives die on me, but not as much as Maxtor. I must admit that they did improve once they purchased Quantum, however.
There's a radio station near me that's decidedly conservative. They were talking about this issue and taking calls. Not one person had any problem with the government spying on people without warrants. "If we have to give up rights to be safe, I'm willing."
It's easy to say. Because really, who'd want to listen in on my calls or watch what I do online? And if they did, I have nothing to hide. Right?
The problem is that these restrictions on liberty, if left unchecked, are easily abused. There's no checks and balances. What happens if you're the subject of an unwarranted investigation? What happens when your phone calls and browsing are played back to an audience? What if it's used for other things, not necessarily "terrorism?" Whatever happened to privacy?
Sometimes the costs to our freedom are too high. The constitution is there for a reason. Willingly giving up the rights given to us by it is an affront to the people that died creating and protecting it.
Slashdot Mirror
Yea, definately. I was responding sarcastically - because the grandparent poster said it was impractical. I don't think it's impractical at all.
"You want to save money by replacing an eight hundred dollar 300GB SCSI enterprise class hard drive with a FOUR THOUSAND DOLLAR 400GB (WITH COMPRESSION) Quantum LTO-3 tape drive?"
First of all, LTO-3 is 400GB Native/800GB with 2:1. They can do 80MB/sec sustained for uncompressed streams.
Second of all, you can get new LTO-3 tapes for less then $100 each. They're still new, so they're still a bit expensive. Of course, that compares very favorably to a used, low-end 300GB, OEM SCA SCSI disk at $500. Chances are you'll need to buy an approved disk with special housing or risk voiding your warranty - which easily doubles the price, or more. And you can't seriously buy used disks for production backups. And - I'd love to see a single hard disk stream 80GB/sec for the entire 300GB, especially at bargain basement prices. SATA drives will yield less performance, albeit for less money. But again, you'll be hard pressed to insert a SATA drive you get at CompUSA into most storage cabinets.
It's not the cost of the drive, but rather how much you can back up with them when they're utilized in libraries and longer term storage, the ease of swapping media, and the feasibility of off-site storage. You can't swap out hard drive spindles from a hard drive. And what? You get a free drive shelf with every 10 disks you buy or something? Those things are expensive, even for a lower cost, older tech SCSI enclosure.
Let's do a little simple math here:
Drive enclosure: $5000 (PV220S) per 14 slots, filled with 14 100GB drives @$800: $16,200
Total space: 4200MB (uncompressed)
LTO-3 10 slot library with one drive: Adic 10-slot library, $7000. 10 tapes: $900. Total: $7900.
Total space: 4000MB (uncompressed)
You could buy 90 more tapes for the extra cost of the disk array, which is another 36TB (uncompressed.)
Disk backups are not cheaper then tapes, so don't even try to make that argument.
"Also, why do you think Quantum is making hard disk backup systems now? Because the market is demanding it, that's why."
You've already said that. And they don't replace tape backup systems, they compliment them. They don't make sense for long-term storage. Unless, such as in your case perhaps, you're backing up a single SBS server or something. For very small shops, you might get away with a Maxtor "single button" backup USB drive. But as you say, if you need full redundancy, you'll have to buy a few of them and the costs add up fast. You'd be better off with a lower end tape drive.
"hard drives ARE far more reliable than tape - especially if they are only being used for backup"
Are you sure about that? Do you know of a SCSI enclosure that spins down disks when they're not being actively used? Otherwise, it doesn't really matter. Hard drive failures are about equal in idle (running) drives versus actively used ones.
I agree that hard drives are less prone to failure then tapes as a general rule, but the risk is great enough in both mediums that the difference doesn't even matter. Usually, disk backup systems are in a raid configuration - which of course makes it all but stupid to try and take raid sets off site on a regular basis for off-site storage. Plus, they're more fragile in transit, heavier, and don't hold as much as tapes.
"What I said was that sound backup and recovery practice precludes using archiving and compression together for critical data - and if you do it, you'd better have redundancy"
Wait - so if you have critical data, but you don't use compression, you don't need redundancy? I know that's not what you mean, but I feel as though the risks are equal regardless of whether or not you compress; meaning, you need to take the same precautions either way.
"which means PAR files which means increased complexity and cost for the dubious benefits of archiving"
I thought we got past this. The conversation evolved into tape backup/disk backup systems. Neither of which would ZIP your files in
No, you tell ME, Mr. Drive Speed Expert, of one *single* drive that can perform at anywhere near what tape systems are capable of these days? You mentioned that a SAN could perform well, in a RAID CONFIGURATION. And only maybe. You mentioned pulling a single drive and moving it off site. How do you propose doing this in an enterprise enviornment?
You mention small business for a hard drive going off site - how much have I talked about small business? None!
"Drop a hard drive? Irrelevant. Drop a tape? Irrelevant. Acidents happen and aren't relevant to the discussion."
Suddenly, it's irrelevant. Then so is compression! Right? What about the weight? Do *YOU* want to carry 20 or 40 hard drives around?
"And I damn sure don't want you backing up MY enterprise with that kind of attitude about backup: save time, money and media - and lose the data."
Ohh, snap. Ouch. That hurts. Here's a NEWSFLASH: time, money, and media are hugely important to a company, no? So why not get all the benefits of a modern tape backup system? Because you're a fool. Somehow, running compression on a backup kills backups. I prefer a sound backup methodology over irrational fear.
"Comparing the probability of bad media to lightning and nuclear weapons is just stupid."
Yes, and I said it was if you actually bothered to fucking read it. Your argument against compression is equally baseless and bullshit.
You ignore points that you agree with, and you harp on the little things that you ignorantly don't. Over the past posts on this topic I've discussed the basics of any good backup practice. You can't see it because you haven't worked in the enterprise. Hey, fuck it. I'm arguing with dumb and dumber here.
Points you've made:
- Take entire SANS off site for safe keeping.
- Don't use compression, it's evil. No better reason.
- Tapes are the devil. Use USB drives instead.
- Hard drives never go bad.
- Forensic bit-recovery is better then a sound backup and recovery practice.
Have fun with that.
Wait, you just said that the idea was impractical but then you agreed with it in your own words, I think.
How do you propose the ISP would be able to determine whether or not joe consumer needs to "unfuck" his computer?
Unless you install a client piece on the customer computers, it would be pretty easy to thwart such bandwidth limiting, service limiting restrictions. You can cloak the client PC's with a linux box, and chances are good that there would be little linksys-like routers available to do the same for the less technically savvy. I wouldn't be surprised if it became a check-box on common for-home devices, and that it would be enabled by default.
Of course, they could also monitor traffice in and out of an IP and watch to see if there's spy/malware type things going on, which a cloak wouldn't mask. In which case, they should notify the end-users, not restrict them without doing so.
We'll see how this plays out. The trend is toward more speed, more speed, and I don't see that changing anytime soon. If a malware infected PC's user doesn't know he/she has it, and internet service becomes slower because the cable company reduces the speed, the user will just think the service sucks and switch to DSL or whatever else.
In the context of sarcasm, it was absolutely correct. There's a world outside the digital, you should join us.
After the whole thing with Sony's music CDs, I wouldn't be surprised if they've had to change some aspects of the copy protection - namely removing stealth copy protection mechanisms. Because you know it was on their minds, and probably already coded.
I kind of like it. I enjoy Battlestar, and they way they run the shows you never have to wait a half a year to see more new episodes.
At least it's better the The Supranos, where nothing happens for 10 episodes and then you have to wait two years to see more.
No, I'm not saying that files should be archived first, and then compressed onto tapes. There were two points of discussion - one was compressed archives, and it moved on to tape backup compression.
Most tape drives these days employ hardware-based compression to ease the burden on the backup servers. It makes more sense as speeds increase on the drives faster then the CPU on the backup servers. This rings true more so when you stream multiple jobs onto multiple tapes simultaneously to maximize throughput. Each file or unit that's backed up to tape is compressed on the fly, individually.
I don't agree that the compression has to occur outside of the backup software. While it may be true that there could be a bug, the same is true wherever you position it. Rest assured, IBM, Legato and Veritas have pretty much worked through compression bugs a decade ago or more..
It's not inherently true that one error in an archive means the archive is blown. A simple recovery record in the archive, available in most archivers, takes care of this handily. Even if you didn't do that, however, a little corruption doesn't mean the entire archive is hosed. It does depend on the archiver to a degree, of course. And if you must do this before backing up to tape, you can include parity files to mitigate risk of corruption.
You're not asking for trouble as long as you take steps to mitigate risk. That's what good BURP (Back-Up and Recovery Practices) is all about.
Your post pretty much proves your inexperience in enterprise backup systems, or in IT systems in general.
"a standard tape backup isn't reliable and never has been. It's also too damn slow - which is why disk to disk backup is taking hold"
Tape backups are pretty damned reliable these days, and very fast. You have to take care to pay attention to tape lifespan and any drive problems that might arise from time to time. Small computer tape systems starting with AIT/DLT have set the bar for reliability - LTO and SDLT have taken it a step further with leaders that hardly ever break. LTO2 tape drives can perform at 40MB/sec sustained with no compression. With 2:1, you'll get 80MB/sec sustained transfer rates. LTO3 doubles these figures, and LTO5 doubles them again. There's even an LTO6 on the horizon that will be able to sustain 540MB/sec with 2:1 compression. Do you know how many SATA hard disk spindles you'd need to even come close to these figures?
Quite often, backups to tape are faster then disk based backups. Disk backup systems *DO NOT* replace tape systems. They supplement them. They provide faster access to backups - no moving tapes around a library and you get random access. There's definite advantages to disk backups - especially when you introduce a snapshot technology like EMC's BCV. I use disk backups where I work, and make copies to tape for longer-term storage and off-site storage.
You can't take hard drives off-site. With single tapes holding up to 800GB of data in a smaller and lighter form factor, it's a no-brainer. You can drop a tape and it'll be okay - and you can even repair some tape cartridge damage. Drop a hard drive and you're screwed. Plus, who wants to carry 20 hard drives around to bring them off-site? They're heavy! Not to mention I've never seen a hot-plug bar-coded hard drive library..
"I have no objection to using compression to send data over the Net or a network - the protocols take care of reliability there (although I suppose there is a small chance of corruption as well which should be taken into consideration.)"
So you're willing to accept that corruption could occur over the network, but compression is okay here. Sounds contradictory to your argument.
"actually have no problem with compression if the redundancy is sufficient. In many cases, a simple double backup will be enough"
In fact, in most cases a single backup is enough in itself - you're not overwriting your backups every day are you? Especially if you backup to disk first, which would be your first copy.
"And this begs the question, which is - what IS a proper backup system and procedures?"
Perhaps a little more experience with enterprise level backup and recovery would be more sufficient then a response from me here. I covered the very basics to get you started in my other posts on this thread. At the very least, your backup system should be well documented and periodically tested using stand-by systems for restoring data.
"As I keep reiterating, the point of backup is RECOVERY. If you can't recover, because of compression or archiving, none of the justifications for using them are relevant."
And if lightening strikes, and blows up the building, it's all for nothing too. Better move underground. And if terrorists set off a nuke, you better have a bunker under there. Actually, turn the computers off, unplug them, and submerge them in concrete. That way, you won't have to worry about corruption. You better not drive to work anymore either, because if you life is of any value to you, you'd avoid the risk of a backup issue that you would be too dead to fix. Better not trust Iron Mountain with your tapes either, because the truck could get into an accident. It's all stupid, I know, just like avoiding compression because forensic recovery of a file MIGHT be more difficult. The idea is to mitigate corruption problems to begin with and have contingency plans, not deal with forensic recovery of corruption when it hits you.
Hey, do your uncompressed backups if you need that level of paranoia - I've made my point and I don't see that it needs to continue being made.
Thanks, Mr. 10th Person To Mention That.
Are you that ignorant of how backups work? Each file is compressed individually to tape or disk backup files.
Are you also ignorant that the internet isn't all 100Mbit? You need compression or you'll be wasting a lot of time. And if you need to ensure integrity, you can use a recovery record in the archive and parity files on the outside.
BILLIONS of compressed files are transferred over the internet every day without issue.
If you take PROPER PRECAUTIONS instead of relying on forensic recovery you won't put yourself in a position to require it.
Are you ignorant to risk mitigation? Are you ignorant to the fact that some risks are so insignificant that you shouldn't nerf your whole system because of it? How hard is that to understand? Disabling compression on backup or file transfer systems is a ridiculous idea.
I appreciate the fact that it's more difficult to reconstruct a compressed file over an uncompressed one, although not impossible in the hands of the right people.
What I'm saying is that to forgo compression - with it's numerous advantages and only one questionable benefit - is a real silly thing to do. You've eluded to the fact that compression in fact causes corruption - which is inherently false (although you've backed off this arguement a bit.) And you seem to ignore the point that speed and capacity are huge pros for compression and have any bearing on the arguement.
I continue to believe that with a proper backup system and procedures, you'll never encounter a time when you'll have to reconstruct a corrupt file. I'll qualify that statement with this: Unless the file was corrupted from the source media. If you backup a hosed database, it'll be hosed on tape, and compression didn't play any part of it.
You began this discussion with "I just can't fathom why a responsible admin would risk the possible data corruption that could come with compression" and I felt compelled to respond to this insulting statement. Compression isn't corruption - it's a staple of large data storage and it has been for decades. I just hope I never end up on the job with someone of the same opinion. I take data integrity very seriously, and your arguement against compression has little credibility.
Really: if you truly need that level of data paranoia (I can only think of maybe one or two institutions that might) a standard tape backup system just won't do for you anyways. There's other ways to ensure data integrity besides full-on backups, and they're designed with high availability and high integrity in mind. EMC's SRDF (in conjunction with a couple of Symms) and some of the Veritas replication tools help with these types of requirements. Of course, all of these systems employ some sort of compression to send the data over the wire.
You might be right about that - I'd not thought that it was a civil case. However, the courts seem generally receptive to requests for council in a case where it's Megacorp vs Jane Doe. You might not get a full PD, though.
Either way, it's all bass ackwards. We shouldn't need to have studied law for ten years to defend ourselves in court - the laws are so damned complicated. Lawyers are paid a lot because they've had to specialize in law for a good part of their lives. They have to attend school forever just to get a grip on it. I don't fault them for charging the high rates that come from such specialization. (Of course, there's a lot of bad ones that really abuse the system for large sums of money.)
There are public defenders available to aid people that cannot obtain their own council. All you have to do is ask.
Not all PD's are bad. Some of them are really good.
Just because you might have a huge library of porn that you need to back up, doesn't mean all of us do too.
Explain to me why and/or how compression reduces data integrity? In fact, I'd argue that it's the other way around in many cases where compression is appropriate since you'll know if corruption occured due to errors in decompression. Otherwise, you might not have as much fair warning. You do realize that we're talking lossless compression, not lossy (like JPEG) compression?
Please. Explain.
I don't undertstand why you think compression automatically destroys the chance of recovery? And how encoding in ASCII is better? What's the thing about "sectors"? I never said using a compressed volume on a hard disk was a good idea. Compressed files can be recovered too, you know. If you have the forensic expertise to recover a corrupted non-compressed file, changes are you'd also be able to recover the data from a compressed one.
The only arguement for compression is not the cost of media - in fact I didn't mention media price at all. I did mention the library capacity, however - and getting an even bigger library is a lot more expensive of a prospect then the $.75 you quoted per GB. Did you read the whole part of my post about speeds? If I can restore that database in half the time because of compression, that means less down time and less money lost. (Although, the money-lost factor doesn't really apply at a government institution; we're not selling anything.)
"If you backup more than once UNCOMPRESSED, you can recover almost anything because it is VERY unlikely that a bad sector will occur in the exact same spot or even in the same file (assuming the one file does not take up most of the specific media.)"
Wouldn't this apply to a compressed backup, too? You're assuming here that the file was unchanged in between the two backups - thus it would apply to any data, compressed or not.
"Alternatively, use PAR files to recover - as long as you're willing to add the extra space and time - which sort of obviates the advantage of compression, doesn't it?"
No - it simply lowers the compression ratio a bit. If you're getting 2:1 compression and add 10% pars, you're still looking at a 1.8:1 compression ratio, but with recoverability.
----
Within every IT budget, you must balance out the speed, recoverability, and cost of your backup solution.
In your solution of never using compression (since no admin should do that, you mentioned) you lose a lot of speed in backups and restores. Speed of recovery is a key factor in many enviornments. It's often the top question asked when in discussion of new backup solutions. You talk about this as an important point yet excluding compression could double your restore times, or more. Not to mention backup speeds - if you can take your backups in half the time, you effectively double the number of servers you could backup in the same amount of time. Or, you reduce the amount of time servers are busy with backups.
Recoverability is big - you want your backups to be reliable. Most of the time, any corruption is unacceptable, be it in a compressed file or not. It's either good or you throw it out and go back to the previous backup. Many IT shops are doing multiple backups these days - backup to disk first, then to tape. Then take snapshots of those tapes and bring them off-site. Compressed or not, testing your backups and ensuring you have no problems with hardware is much more effective then using uncompressed backups and performing forensics on them if they're bad. Speaking of which, I don't see why compressed data would be less recoverable.
Finally, you have cost. Yes, even when data recoverability is a key factor, you still have to consider cost. So, what makes more sense? Using uncompressed backups that will backup and restore slower, cost a lot more for media and library capacity, and cause more personnel overhead for swapping tapes - or using compression and cutting all that in half? You'd rather lose all that in the off chance that MAYBE you could recovery more of your data, in the off chance that NONE of your other backups are good? I don't know any resposible IT manager that could agree with you.
A proper backup and recovery plan with periodic testing and multiple copies held on-site and off is a much more effective solution then betting on forensic recovering of uncompressed data.
Hey, I'm not claiming that compression is always right in every situation. That's far fro
I agree - I believe Microsoft is still a Monopoly on the desktop. And perhaps more so with Office. While yes, there's alternatives, there's always been alternatives. It's the viability of the alternatives.
Microsoft still has the power to crush everything in one fell swoop. They could raise prices 400%, and a LOT of people would have no choice but to pay it. They could completely change all of their document formats, and change them on open. They could change IIS and IE to work well togher, but not with anything else. You could switch to something else, but then all your documents become unusable. Or at least, many of them. You'd have a hard time online.
They don't do it because they could get really stomped on legally, but they still do it in small ways. They still try to hang on to their same monopolistic practices of locking people in, rather then providing a better product or service.
All I see is ads. I think I found a paragraph that looked like it may have been the article, but every other word was underlined with an ad-link so I didn't think that was it..
If you're familiar with Usenet, you've probably encountered PAR files from time to time. A PAR file is a parity file which can be used to reconstruct lost data. It works sort of like a RAID, but with files as the units instead of disks.
Let's say you have a 200MB file to send. You could just send the 200MB file, with no guarantees that it will reach the destination uncorrupted. Or, you could use a compression program and bring it down to 100MB. In this case, even if you lost the first transfer, you could transfer it a second time. Then we look at PAR. You compress the 200MB file into ten 10MB files. Then, you could include 10% parity - if any of your files is bad, you'd be able to reconstruct it with the parity file. With only 110MB of transfer. PAR2 goes even further by breaking down each file into smaller units.
Besides transfer times and correction for network transfers, compression can also increase speeds of transfer to mediums. If you have an LTO tape drive that can only write to tape at 20MB/sec, you'll only ever get 20MB/sec. Add compression to the drive, and you could theoretically get 40MB/sec to tape with 2:1 compression. That means faster backups, and faster restores. On-board compression in the drives takes all the load off the CPU - but even if you use the CPU for it, they're fast enough to handle it.
Not to mention, it takes a lot less tape to make compressed backups. I don't know what world you live in, but in mine, I don't have unlimited slots in the library and I don't want to swap tapes twice a day. Handling tapes is detremental to their lives; you really want to touch them as least as possible.
Data corruption isn't caused by compression. If it's going to happen, it'll happen regardless. While your point is true that it MAY be more difficult to recover from a corrupt file, that's not the right methodology. If your backups are that valuable, you'd make multiple copies - plain and simple.
I can't fathom why a responsible and well informed admin would avoid compression.
"While the figure represents existing salaries and equipment drawn together under one roof, it also represents some shift in emphasis by IBM from Linux back to its mature Unix operating system."
How do you figure that? If all it is is shifting the books around, I don't see how it's changing emphasis on anything. Sounds more like simply reorganizing in the name of effenciency.
I share much of the same philosophy- I'll take that bundle of cash even if the job sucks, and it will enable me to do whatever I want when I'm not there.
However, while earning lots of money offsets the will to quit a bad job, it doesn't make it any easier to put in the 40 hours. We spend a lot of our lives at work.
Sometimes it's not worth it. If you're at a shitty 80k/yr job, and you have the chance to get a much more enjoyable job that's closer to home for 70k/yr, a lot of people might take it.
I agree that the punishment needs to be enough of a deterrent to dissuade any would-be lawbreakers, but I also believe that the punishment should fit the crime. I don't believe prison is the answer for many crimes. There's other forms of punishment - it doesn't always have to be prison time. People can screw up, and I don't think they should pay for it in years of jail time and a thoroughly ruined life (for a time anyways.)
Of course, I'm still talking about non-violent crimes here.
It's the repeat offenders that need more serious punishments, but even then I just can't justify sending people away is always the right move for many non-violent offenders. However, I guess in some cases it might be what's needed to "correct" someone. I just don't think we should be handing out years in jail to people as easily as giving out candy to school kids. It shouldn't be taken so lightly.
The fact is, tougher sentences have had very marginal impact on crime rates when applied. What criminal plans on getting caught? If someone breaking the law even seriously considered how much of a pain in the ass all the legal would be, it would probably be enough. But they don't. Murder rates in states with the death penalty doesn't decline, DUI doesn't decline with harsher penalties, and even the 3rd strike rule in California hasn't made any big impact on the number of robberies.
I don't necessarily agree that without strict laws, crime would run rampant. I'm not saying that crime shouldn't be taken seriously, of course, nor that we don't need laws. What I'm against is the trend toward unnecessarily harsh punishments. I guess the issue is what the definition of that is- and for many violent crimes I don't feel as though the punishment is harsh enough. 5 years for a woman that kills her baby in a fit of rage? And 5 years for a guy that sells modded xboxes with some pirated games? Something doesn't seem right there.
Quantum was always good for me, so was Seagate. Unfortunately, every Maxtor drive I ever owned went to shit within the year. I've had WD drives die on me, but not as much as Maxtor. I must admit that they did improve once they purchased Quantum, however.
There's a radio station near me that's decidedly conservative. They were talking about this issue and taking calls. Not one person had any problem with the government spying on people without warrants. "If we have to give up rights to be safe, I'm willing."
It's easy to say. Because really, who'd want to listen in on my calls or watch what I do online? And if they did, I have nothing to hide. Right?
The problem is that these restrictions on liberty, if left unchecked, are easily abused. There's no checks and balances. What happens if you're the subject of an unwarranted investigation? What happens when your phone calls and browsing are played back to an audience? What if it's used for other things, not necessarily "terrorism?" Whatever happened to privacy?
Sometimes the costs to our freedom are too high. The constitution is there for a reason. Willingly giving up the rights given to us by it is an affront to the people that died creating and protecting it.