It's long been clear that long-term digital archival is a serious problem. If dying media doesn't get you, technical obsolesence of the medium eventually will. Long-term digital archival requires periodically converting data to new media and even new media formats.
What's not long been clear is that long-term digital archival is a serious problem for the average Joe Blow consumer. Only in recent history have non-techie people begun to rely on digital backups for precious data such as family photos and video, tax returns, etc.
What we need is affordable long-term archival backup services for average consumers. The pickings are incredibly slim, despite the large number of companies offering "online backup". There are two problems here.
First, most people offering online backup services do not seriously stand behind their reliability, even in the short term. The fine print usually reveals that they absolve themselves from nearly all liability for completely losing your data.
Second, there's little reason to believe most of these online backup companies will even be around in 50 years when your grandkids want to retrieve your decades-old backed up digital photos (preferably in whatever bitmap format is in vogue in 2054). Certainly, most of the online backup companies clamoring for attention at the height of the dot-com boom are long gone, along with all the data they purported to keep safe for customers.
What's needed is a large, stable company with some long-term credibility (such as IBM) to offer long-term archival services that are affordable to ordinary consumers, and that are believably robust and long-lasting. If a company is willing to post a bond that promises me $100,000 if they are ever unable to retrieve my data, then I begin to at least believe that they are seriously safeguarding it. Whether they will stay in business for the next 50 years is much more difficult to demonstrate with confidence.
My first question would be whether this will make the problem of light->heat buildup->dye degradation more likely for "R" discs. Likewise, if the disc suffers exposure to humidity, will the extra layer make it both more resistant and harder to "dry out"?
See this text to get up to speed on some of the best ways to ruin some discs.
It's a little late, due to the deadlines for registering (which have already gone by in some little states like, oh, California).
And before the usual posts bemoaning the lack of turnout, please keep in mind that the U.S. voting system was always designed to prevent people from voting (the non-wealthy, women, non-whites, and now anyone-not-likely-to-vote-for-my-side). The registration process ain't about preventing fraud -- it's about preventing voting.
For some lessons learned from management simulations (years ago) that you can use right this minute, try reading "The Logic of Failure", by Dietrich Dorner. The simulations discussed included trying to keep meat from spoiling in a freezer with a simulated broken part (but you didn't know what part it was), as well as much more complex simulations in which you have to manage a small African ecosystem.
These simulations clearly expose general situations that humans are stunningly bad at unless they are trained to recognize them and behave against their natural inclination.
For example, the freezer simulation showed that humans have great trouble grasping any situation in which there is a delayed response to their actions (the temperature of the freezer responds to your changing the thermostat, but only after the fact, and it may overshoot). How does that apply to your world? I bet if your company has 100 people and needs to reduce the headcount to 90 people, they would lay off 10 people. The problem? The delayed effect that layoffs have in causing people who aren't layed off to look for work elsewhere. If you want to get rid of 10% of your people, you probably better only lay off perhaps 7% or 8%.
In recent years, I watched a local company go through no fewer than seven layoffs. Every single layoff was followed within a matter of weeks by hirebacks, as additional people departed in response to the layoffs and the company had to hire to fill essential positions. After seven iterations, the managers still had not grasped they were overcontrolling a system that had a response delay built into it.
It's hard to believe that such incompetence persists in the software business, where managers receive a level of thorough and professional training that... oh.
Small, independent developers, however, are recognising this is a serious problem and are generally stumped by what to do about it.
One of the reasons I joined the Association of Shareware Programmers was to see what real indie developers are really doing about piracy. Let's put it this way: they ain't stumped.
Piracy is kind of like spam, in that it's a cost of using the internet. Just as there are a bunch of things you can do to reduce spam, but no silver bullet for eliminating it, people in the business have a laundry list of steps for reducing piracy to acceptable levels. Some of them include:
Use well-known "locking" software. After a few weeks on the ASP newsgroup, you quickly discover the handful of tools for serial-number locking that generally offer the best results for software products.
Maintain and exchange blocking lists. A lot of cracks result in a sudden burst of download activity from "warez" sites. When these crop up, ASP members block the appropriate IP addresses, and often offer them to others (sounds more like spam, don't it?)
Get the warez site shut down. This is something ASP members have a lot of experience with. Being able to have helpful friends in some of the far-off countries that some warez sites live in is sometimes the key to getting their plug pulled.
Offer value that requires a website connection. That free serial number ain't so cool if a significant part of the value comes a feature that involves connecting back to the publisher's website, and you find your serial number was already "used up". Some members are also experimenting with schemes that allow remote revocation of serial numbers.
Just like spam, there's no silver bullet for software piracy. However, just like spam, little guys continue to band together to create and exchange schemes for effectively reducing the cost of the problem.
During the long competition to replace the floppy (with ZIP, LS-120, etc.), no one predicted the actual outcome -- that there would be no clear winner.
ZIP drives, well, not cheap and not small, and not widely built-in by box builders and (some think) not all that reliable.
CD-RW, well, not small, and the software was not built-in until Windows XP, and even that software is "one big burn" and doesn't let you copy/delete individual files one at a time so you can use it "like a floppy" and (some think) not all that reliable.
Then we come to USB disk-on-key. Small, software already mostly built-in, random access, can be used "like a floppy". Not real fast, but probably works pretty good for many floppy-like applications. But will it work for data backup? Most people aren't aware that the technology there tolerates a quite limited number of rewrites. Will people be happy when they discover their $50 USB dongle fails after less than a year of daily backups?
When it comes to making casual backups, the battle to replace the floppy is still ongoing. Maybe there'll never be a clear winner, or maybe it's going to be one of these technologies.
You can actually read (elsewhere) in realtime panicky messages being posted by website operators who just discovered their hosting service is located in Florida and is going to be going down for at least a few days. People think they've got their bases covered when they have a backup copy of the data files comprising their website, but they often get a nasty surprise like this.
It doesn't take a hurricane to teach website operators about backup problems, though. Worms that infect and destroy hosting service servers, or router attacks that effectively shut a hosting service down for days produce the same kind of collection of panicked webmasters.
Just backing up website data files is only a part of a website backup plan. You really want at least two independent (that means both geographically independent, and not run by the same company -- don't forget the "FBI shuts down hosting company" scenario) DNS servers listed as authoritative for your domain. Very few websites meet even that lone requirement.
You don't have good data backup if you can't demonstrate that you can recover from disaster, and the same is pretty much true for website backup. If you can't show that you can, within at least a matter of hours, have your website running on a machine it's never lived on before and serving "real" requests from the outside world, then you shouldn't really bother reading the fine print about whether your hosting company claims it offers 99.9% uptime or 99.99% uptime.
While some software packages can be successfully sold using online channels exclusively, these are the exceptions
That pretty much assures me the author does not know what he's talking about. The vast majority of software packages are sold exclusively via the web. They are mostly Windows software, mostly small companies (<10 people, skewed towards the 1-man band), and mostly make such a modest amount of money that the author should perhaps be forgiven for not noticing where the bulk of the software market iceberg lies.
If you want to really learn about selling software, join the ASP and talk to the little guys who (cumulatively) are making most of the software that gets sold in the world today.
Disclaimer: I'm a member, but I (alas) make no money for telling people to join:-).
If you have to keyframe every half second, then you can pretty much already get this effect using Hash's Animation:Master (www.hash.com).
Oddly enough, the author of A:M relates that Microsoft actually owned the source code to Animation:Master some years back. Apparently, he got pissed off, quit, and filed a lawsuit to get the code back. Microsoft strange story #298.
There was a brief moment in time when I thought that hard disks had finally gotten way bigger than I would ever need. With my 60GB, striped and mirrored, I only managed to use about 10GB. Even getting into digital photography, I could see that I was not going to run out of disk space this decade, even if I didn't drop all the totally useless photos from my collection.
Backups? Well, I was already striped and mirrored, but with disk space so far outstripping my data needs, I could also easily afford to do intelligent backups of just important data to other machines, even offsite. Life was good.
Alas, then I discovered digital video. In an instant, I went from viewing 250GB as unnecessarily huge, to barely big enough comfortably to edit one full-length movie. I'm not talking MPEG here -- when you're heavy into editing, you want to use full-size, uncompressed video, so each minute of video chews up the disk like it was going out of style.
This leaves me once again in backup hell. Do you know what digital video people use mainly to backup? Mini-DV tapes, pumping the video right back out to magnetic tape. This takes a while, to say the least, and also helps wear out the tape transport on your probably-much-more-expensive-than-a-250GB-HD digital video camera.
People into digital video see the cutting edge of where backup problems are headed. They see that the time required to just pump the data off the disk is becoming prohibitive. If disk transfer speeds don't suffer a quantum improvement soon, disks will become, from the perspective of full backups anyway, essentially like serial devices, were you have to focus on restricting access to the device because it's just so slow.
The real news is just how many years Microsoft will require to finally support any form of rewritable disc. From the first day that recordable CDs appeared, people have dreamed of using them "like a floppy" -- being able to add a file, delete a file, etc., instead of having to use the "one big burn" approach that requires first constructing on hard disk an image of everything that is to appear on the recordable disc. It's like they just don't place much importance on people being able to use discs for backing up data.
Windows XP appears to support rewritable CDs (couldn't get their rewritable DVD act together in time for XP, I guess), but it's really a bundled Roxio "one big burn" solution under the covers.
You can get third-party software that lets you treat DVD+/-RW and CD-RW "like a floppy", but so far reliability has remained suspect, and interoperability between different vendor solutions has been provably A Big Mess.
The re-recordable media problem is especially crucial to Windows backups. With dual-layer re-recordable DVDs coming, DVD could be a plausible Windows backup media for many situations -- if direct O/S support solved the compatibility mess.
I can't believe how many years will pass before Windows has any direct, API-level support for any form of re-writable disc. I also fear that when Longhorn finally ships, it will somehow manage to be one or more steps behind the current technology of that time, leading up to another umpteen-year delay to have decent backup solutions for the casual user.
Will they get Mt. Rainier support in, as promised? Seems like dual-layer RW will be a reality before Longhorn is -- will it support that? I wait with bated breath!
Absolute improvements in both disk and tape technologies are not as important as the fact that the rate of change of improvement for disk is exceeding that of tape. I can find no reason to believe this difference in rate of change will be inverted in the forseeable future.
As storage sizes and needs relentlessly increase, the serial nature of tape becomes an ever-greater penalty. Large, expensive hardware solutions can overcome this with parallelism. However, it's the cheap, commodity solutions that drive innovation and ultimately dominate the market, and that's hard disk backup solutions.
One of the primary barriers to the dominance of disk-to-disk backup is simply form factor. Large, cheap hard disks generally come with their electronic guts hanging out, and without an easy to plug/unplug connection. This is in the midst of change, however. Tiny form factor disks that were designed from scratch to be robust and carryable are increasing their capacities at a rapid pace. USB 2.0 and Fireware offer enough speed to be plausible connectors to backup drives.
Another factor to consider is that the rate of change of disk capacity increase is much greater than the rate of change of disk data transfer rate. Thus, image backups of disks are becoming impractical in many situations, and restoring from tape becomes even more tedious, as you try to wind to the desired version of a file in a vast body of backup data.
The fact that people who have a tape solution that works have no immediate need to switch to disk does not mean tape is not being displaced as a backup solution. We still have lots of mainframes being used; that does not mean that PCs did not usurp mainframes as the primary means of computing -- they certainly did.
No one with an effective tape backup solution today should drop it immediately. Tape backups will be around for years to come. Tape backups are being displaced by disk-to-disk backups. All three things are true and not at all contradictory.
Hack your firmware for increased storage? Maybe to make backup copies of your kid's Disney movies, but is that really the foundation you want for backing up your company's financial data?
Right now, we're in a window of transition towards finding a dominant medium for backing up computer hard disks. Most likely, the winner will fairly soon be: not DVD, not tape, but simply other hard disks. Right now, the minimum "nice" size for a backup device is 40GB, and by the time a recordable dye-based disc can reach that, the bar will have been raised to 80GB.
Cost is getting close to being eliminated as a factor in using hard disk as a backup media. The remaining hurdle is really a generic protective package (e.g., a little shock protection and don't expose any electronics) and the ability to walk up and plug it into any PC.
IoMega has issued their hard-disk backup medium solution, but it's slightly pricey, and strongly proprietary. Perhaps someone will start wrapping small form-factor disks in plastic with a connector that can plug into (and be powered by) either USB 2.0 or FireWire.
Within a few years, only the most low-end and casual forms of computer backup will be on DVD instead of hard disk.
While it's true that we've always lived in a world with lots of bacteria and viruses, there's a case to be made for the fact that Americans live in a dirtier world than some decades ago.
Instead of wood or linoleum floors whose dirt can be attached through relatively primitive means (water and cloth), we largely live on carpet. I can run the steam cleaner over my living room carpet apparently indefinitely without it ever failing to yield up more filth.
Instead of baths in porcelain tubs that get scrubbed at least weekly (to remove rings and, as a good side effect, germs), we're taking showers standing next filthy curtains, and neatly aerosoling germs straight to the lungs via the shower steam (if your house is an a high radon area and you take a lot of showers, might as well take up smoking too!).
Just general house cleaning has become both less common and less easy. Rember the phrase "spring cleaning"? Ever participated in one? Before anti-biotics made scratches and other small wounds of no account, keeping your local environment clean was a survival instinct as much as a social nicety.
At least in the case of shower curtains, however, there is a simple solution. Get a washable shower curtain (google for shower curtain washable cotton duck) and wash it in hot water once a week. One germ collector eliminated, and it's nicer brushing up against wet cotton than wet chemically treated plastic. You still need to Comet that tub once a week, though:-).
I've used nothing but RedHat in the past, but it's just become too much of a feeling that my business (yeah, I buy the CD's right from them) isn't terribly appreciated. So, yesterday I ordered the new release from Gentoo and will try it out on one machine. At least I can tell that Gentoo is in no position to sneer at my measly $20:-).
Other motivations include a growing desire for a slightly more cutting edge distro. I wanted the 2.6 kernel some weeks ago, and have an interest in playing with the new security models as well. Fingers crossed that I'll be a happy Gentoo camper from now on.
Unlike milter, the policy server interface is only processing SMTP commands -- the policy server does not get fed mail headers or body, for example (OTOH, this restriction makes it less likely you will write a policy server that slows mail processing to a crawl). The policy server can, however, insert mail headers into the target message, as that is one of the actions supported.
If you really want to wack on the mail headers and/or body, the Postfix 2.1 way is to essentially insert your own SMTP proxy into the stream. Not nearly as easy as writing a policy server, and (like most methods of reprocessing the entire mail message stream external to the MTA itself) likely to significantly reduce your mail server's peak processing ability.
Postfix's new policy server API
on
Postfix 2.1 Released
·
· Score: 5, Interesting
One of the geek-cool things about this release of Postfix is that it finally provides a way to add your own code to the SMTP conversation without having to understand or patch Postfix at all.
The new policy server interface is a simple sockets-based API for getting a chance to participate in the SMTP conversation as it is happening. The basic idea is:
tell your Postfix config file (main.cf) that you've written a "policy server" that listens on a particular Unix socket or TCP address/port.
You can have the policy server get "called" at any of the points in the SMTP conversation where Postfix may make a decision about how to dispose of the message (HELO, RCPT, etc.).
write your policy server. It listens for connections, and each connection sends you one or more requests. Each request contains a small set of information about the mail message being transmitted (client name/address, HELO text, etc.) Your server responds with one of a broad set of actions that Postfix supports (reject, accept, defer, perform other custom checks, etc.).
The protocol for talking to your server is a simple text-based protocol with newlines, much like the form of HTTP. I coded an initial policy server in good ol' C in about an hour.
In particular, this new API is a great place to implement greylisting. Your server can maintain its database of whitelisted and greylisted from/to/IP triplets all on its own and just respond to Postfix requests. And, once you've coded up your policy server, you don't have to revise it with every Postfix patch that comes down the pike. As long as the API remains backwardly compatible, your policy server code should survive any Postfix upgrades.
I recently did my first little test of greylisting + spamtrap RBL. This is where you tempfail any email you have any suspicions of for about, say, an hour, to see if the (suspected) spammer will, in that length of time, transmit something to a known spamtrap mail address. For my test, I accepted all mail so I could look through each one to check for false positives.
The result was: only about 2% of the spam would have gotten through. I think I can improve that rate by increasing my local spamtrap database to augment the larger one at cbl.absuseat.org. But even if I can't: 98% of spam eliminated in a 100% automated fashion, no tuning and tweaking and training. Completely automated spam removal, totally driven by the spammers themselves (they tell us what IP addresses they are using today by using them to send spam to a spamtrap address).
Greylisting + spamtrap RBL has some niggling problems, such as dealing with mailing lists that use a different sender address (and maybe even IP address) when they retry a tempfailed message. However, these problems seem manageable compared with solutions such as teaching every user to train a Bayesian filter.
To defeat greylisting + spamtrap RBL, spammers will have to locate all the spamtrap addresses in their databases and remove them. Good luck!
Greylisting + spamtrap RBL may not be a silver bullet, but it sure acts like one on my system.
Let's go to the actual study: http://www.psychology.iastate.edu/faculty/dgentile/MMVRC_Jan_20_MediaVersion.pdf
It's kind of a slide-show study report, so it's hard to get at all the details. But, there's room for skepticism...
Residents outweigh attending physicians 2-to-1 in this study. Wouldn't residents be more likely to be younger? Aren't younger people much more likely to have significant video game experience? I can find no place in the report that shows they controlled for age. Might the study simply be showing that "younger people have better eye-hand coordination than older people?"
Aren't most new kinds of video games and equipment (I would suppose, including laparoscopic equipment) built by young people with young eyes? Don't most older (>45) people develop farsightedness? Might the study merely be showing that "laparoscopic equipment needs to be improved for surgeons who have older eyes"?
In "Methods and Materials", I saw a quote that made me think "skill" was partially calculated by how fast the operation was performed. Might not residents who have only performed 2 actual surgeries be more likely to risk going faster, unlike experienced physicians who, with many more actual surgeries under their belt, might be more inclined towards caution? Do I really want the speed demon operating on me, or the guy who goes "slow and steady"?
Don't many video games essentially teach "it's better to be fast than right, better to keep moving than stop and think"? Is that the mindset I want in a surgeon?
It was hard to determine whether the simulator being used was closer to a video game or closer to real surgery. Might the study merely be showing that "people who are better at video games are better at surgery video games"?
This study, or at least this description of it, failed to convince me that I want a Doogie Doctor doing my next surgery. I think I'll go with the guy who has had a couple hundred successful operations over the guy who smoked him on Mortal Kombat.
Yeah, I laid out only the basics. There are multiple small warts to deal with, but I haven't found any of great inconvenience.
However, to be clear on one point: you say "you don't want to 'greylist' earthlink or AOL". To be accurate, you don't want to *blacklist* the IP address of a major MTA that serves many customers. It's still fine to greylist mail that came out of (for example) earthlink, because greylisting applies to a triplet (IP/sender/recipient). It will temporarily stop that same spam from being re-sent, but has no effect on other email coming from the same IP address, but whose sender/recipient pair differs.
And, FWIW, in my own mail logs, I have not seen any spam coming from a real AOL MTA in recent memory. Lots of spam with a forged address at AOL, some spam with an IP address that belongs to AOL, but none that would pass an SPF check (which should be applied before most anything else, as it's relatively cheap).
In fact, I should amend the title of that post to SPF + greylisting + honeypot RBL = works real good!
While there's unlikely to be a silver bullet for spam, greylisting combined with a honeypot rbl is likely as close as you can get. People usually criticize greylisting without grasping that it's only one-half of what's needed for effective and completely automatic spam elimination, with 0 rejection of legitimate mail (the 0 assumes no legitimate sender uses an MTA that can't retry, but that's close to true).
Step 1: Salt the spammer's email databases with guaranteed bogus email addresses that no legitimate email sender has ever seen. This is currently trivially implemented as follows. In your website's robots.txt file, list several files that robots must not examine -- these are your honeypot. Then, fill those files with HTML that contains your bogus email addresses. Spammers will, quite reliably, disobey the robots.txt file, use it to discover HTML files that are not linked to from anywhere else in the world, and add your bogus mail addresses to their database.
Step 2: Implement greylisting + honeypot-based RBL. When email arrives that is not whitelisted, see if it comes from an IP address that is "temporarily" blacklisted in your RBL. If it is, you can reject it right now. Otherwise, see if the target address is in your honeypot database. If it is, add the sender's IP address to your RBL and fail immediately. Otherwise, engage the now-classic greylisting algorithm (see http://www.greylisting.org/) to "tempfail" the email. The point of the temporary failure is to give the spammer time to use the same IP address to send the same spam to an address that *is* in your honeypot database, so you can then proceed to reject the retry of the spam to a legitimate email address).
requires no per-user work, such as "training" of filters.
requires no changes to any software, except MTAs (and only a handful of them handle most of the world's software). no new laws.
no false positives. to get blacklisted you *must* have transmitted email to an address that could only have been obtained by illegally harvesting a website.
even compromised home systems are not terribly harmed. if a spammer takes over your home computer and uses it, well, the IP blacklist need not be permanent, just long enough to cover a single spam run -- a few days is probably plenty. if the spammer is blasting out runs from your home computer continously, well then you have worse problems than finding yourself unable to send email to GrandMa.
not easy to defeat. right now, anti-spammers must work very hard to locate the "real" email amidst all that spam -- and never, ever mistakenly reject a "real" email. greylisting plus honeypot RBL inverts the equation. the spammer must make sure that not a single "bogus" email address is anywhere in his database! spammers are ingenious, but developing absolutely perfect lists of legitimate email databases is something they have no experience with so far.
no restriction of free speech. total whacko strangers who aren't spammers can still send you email -- it may just get delayed for an hour or so (a fact which is totally true already).
nobody makes any money off it. you don't have to pay anybody, except for the effort involved in setup and maintenance (a fraction of the total time wastes on spam currently).
computationally cheap. most MTAs are already looking up IP addresses and target addresses in databases. cost of this scheme should not greatly slow down most MTAs. especially compared to content-examination schemes such as Bayesian filters.
no judgement calls in blacklisting. no third party has to decide what is spam and what is not. the rbl in this scheme is totally generated from absolutely bogus email addresses -- the only way you can get in the rbl is to flat-out declare yourself a scumbag by sending to one of those illegally obtained addresses.
No scheme is perfect, but greylisting combined with an RBL that is derived solely from bogus email addresses is pretty damn good.
Or use SWREG for $1 + 4% and not have to worry about having your bank account frozen. Last I researched it, they seemed like about the cheapest reputable shareware cc proxy.
Hey, I already got Wi-Fi enabled with my crappy $100 Radio Shack unit that lets me broadcast the downstairs ReplayTV to the upstairs TV. Sure enough, the microwave makes loud and annoying static for it.
However, if the microwave simply supported an IR remote, then I could reprogram one of my hackable Radio Shack remote's buttons to pause the ReplayTV and mute the TV everytime I pushed the "cook" button. (Hmmm, I think I would use Shift-Freeze to be the "cook" button.:-)
Couldn't tell if you were being facetious or not. DVD-ROM is the physical format used by DVD-Video discs.
They pretty much all read DVD-ROM discs, except possibly for some DVD-RAM drives (which few are buying), so that would not be a good example of why one should wait for a single standard.
If you really are concerned about being able to buy a drive that reads your old discs in 10 years, then pay the high dollar for magneto-optical (MO). Pledged backward and forward support is its main claim to fame at this point.
Re:DVD-Rs go 8x
on
DVD-Rs go 8x
·
· Score: 5, Insightful
price: the cost per GB of DVD-R crossed hard disk prices recently, though they are still very close.
durability: DVD is not as susceptible to physical shock and magnetic fields as a hard drive.
movability: more PCs can read a DVD-R than have a slot for inserting a removable IDE drive.
size: when what you want to store fits fine in 4.7GB, a DVD is a much nicer form factor than an IDE hard drive (so far). (e.g., daily incremental backups extending back for a full month.)
movies: I can't create a movie on a hard drive and then stick it in my consumer DVD player (so far).
However, DVD+RW and DVD-RW would certainly be more attractive for general data use if the operating system actually supported them as random access devices. Don't know about *nix, but Windows does not support such access until the next version (XP supports drag and drop, but simply copies files to a temp area, and then waits for you to tell it to do the One Big Burn).
Slashdot Mirror
What's not long been clear is that long-term digital archival is a serious problem for the average Joe Blow consumer. Only in recent history have non-techie people begun to rely on digital backups for precious data such as family photos and video, tax returns, etc.
What we need is affordable long-term archival backup services for average consumers. The pickings are incredibly slim, despite the large number of companies offering "online backup". There are two problems here.
First, most people offering online backup services do not seriously stand behind their reliability, even in the short term. The fine print usually reveals that they absolve themselves from nearly all liability for completely losing your data.
Second, there's little reason to believe most of these online backup companies will even be around in 50 years when your grandkids want to retrieve your decades-old backed up digital photos (preferably in whatever bitmap format is in vogue in 2054). Certainly, most of the online backup companies clamoring for attention at the height of the dot-com boom are long gone, along with all the data they purported to keep safe for customers.
What's needed is a large, stable company with some long-term credibility (such as IBM) to offer long-term archival services that are affordable to ordinary consumers, and that are believably robust and long-lasting. If a company is willing to post a bond that promises me $100,000 if they are ever unable to retrieve my data, then I begin to at least believe that they are seriously safeguarding it. Whether they will stay in business for the next 50 years is much more difficult to demonstrate with confidence.
See this text to get up to speed on some of the best ways to ruin some discs.
And before the usual posts bemoaning the lack of turnout, please keep in mind that the U.S. voting system was always designed to prevent people from voting (the non-wealthy, women, non-whites, and now anyone-not-likely-to-vote-for-my-side). The registration process ain't about preventing fraud -- it's about preventing voting.
These simulations clearly expose general situations that humans are stunningly bad at unless they are trained to recognize them and behave against their natural inclination.
For example, the freezer simulation showed that humans have great trouble grasping any situation in which there is a delayed response to their actions (the temperature of the freezer responds to your changing the thermostat, but only after the fact, and it may overshoot). How does that apply to your world? I bet if your company has 100 people and needs to reduce the headcount to 90 people, they would lay off 10 people. The problem? The delayed effect that layoffs have in causing people who aren't layed off to look for work elsewhere. If you want to get rid of 10% of your people, you probably better only lay off perhaps 7% or 8%.
In recent years, I watched a local company go through no fewer than seven layoffs. Every single layoff was followed within a matter of weeks by hirebacks, as additional people departed in response to the layoffs and the company had to hire to fill essential positions. After seven iterations, the managers still had not grasped they were overcontrolling a system that had a response delay built into it.
It's hard to believe that such incompetence persists in the software business, where managers receive a level of thorough and professional training that... oh.
Piracy is kind of like spam, in that it's a cost of using the internet. Just as there are a bunch of things you can do to reduce spam, but no silver bullet for eliminating it, people in the business have a laundry list of steps for reducing piracy to acceptable levels. Some of them include:
- Use well-known "locking" software. After a few weeks on the ASP newsgroup, you quickly discover the handful of tools for serial-number locking that generally offer the best results for software products.
- Maintain and exchange blocking lists. A lot of cracks result in a sudden burst of download activity from "warez" sites. When these crop up, ASP members block the appropriate IP addresses, and often offer them to others (sounds more like spam, don't it?)
- Get the warez site shut down. This is something ASP members have a lot of experience with. Being able to have helpful friends in some of the far-off countries that some warez sites live in is sometimes the key to getting their plug pulled.
- Offer value that requires a website connection. That free serial number ain't so cool if a significant part of the value comes a feature that involves connecting back to the publisher's website, and you find your serial number was already "used up". Some members are also experimenting with schemes that allow remote revocation of serial numbers.
Just like spam, there's no silver bullet for software piracy. However, just like spam, little guys continue to band together to create and exchange schemes for effectively reducing the cost of the problem.ZIP drives, well, not cheap and not small, and not widely built-in by box builders and (some think) not all that reliable.
CD-RW, well, not small, and the software was not built-in until Windows XP, and even that software is "one big burn" and doesn't let you copy/delete individual files one at a time so you can use it "like a floppy" and (some think) not all that reliable.
Then we come to USB disk-on-key. Small, software already mostly built-in, random access, can be used "like a floppy". Not real fast, but probably works pretty good for many floppy-like applications. But will it work for data backup? Most people aren't aware that the technology there tolerates a quite limited number of rewrites. Will people be happy when they discover their $50 USB dongle fails after less than a year of daily backups?
When it comes to making casual backups, the battle to replace the floppy is still ongoing. Maybe there'll never be a clear winner, or maybe it's going to be one of these technologies.
It doesn't take a hurricane to teach website operators about backup problems, though. Worms that infect and destroy hosting service servers, or router attacks that effectively shut a hosting service down for days produce the same kind of collection of panicked webmasters.
Just backing up website data files is only a part of a website backup plan. You really want at least two independent (that means both geographically independent, and not run by the same company -- don't forget the "FBI shuts down hosting company" scenario) DNS servers listed as authoritative for your domain. Very few websites meet even that lone requirement.
You don't have good data backup if you can't demonstrate that you can recover from disaster, and the same is pretty much true for website backup. If you can't show that you can, within at least a matter of hours, have your website running on a machine it's never lived on before and serving "real" requests from the outside world, then you shouldn't really bother reading the fine print about whether your hosting company claims it offers 99.9% uptime or 99.99% uptime.
That pretty much assures me the author does not know what he's talking about. The vast majority of software packages are sold exclusively via the web. They are mostly Windows software, mostly small companies (<10 people, skewed towards the 1-man band), and mostly make such a modest amount of money that the author should perhaps be forgiven for not noticing where the bulk of the software market iceberg lies.
If you want to really learn about selling software, join the ASP and talk to the little guys who (cumulatively) are making most of the software that gets sold in the world today.
Disclaimer: I'm a member, but I (alas) make no money for telling people to join :-).
Oddly enough, the author of A:M relates that Microsoft actually owned the source code to Animation:Master some years back. Apparently, he got pissed off, quit, and filed a lawsuit to get the code back. Microsoft strange story #298.
Backups? Well, I was already striped and mirrored, but with disk space so far outstripping my data needs, I could also easily afford to do intelligent backups of just important data to other machines, even offsite. Life was good.
Alas, then I discovered digital video. In an instant, I went from viewing 250GB as unnecessarily huge, to barely big enough comfortably to edit one full-length movie. I'm not talking MPEG here -- when you're heavy into editing, you want to use full-size, uncompressed video, so each minute of video chews up the disk like it was going out of style.
This leaves me once again in backup hell. Do you know what digital video people use mainly to backup? Mini-DV tapes, pumping the video right back out to magnetic tape. This takes a while, to say the least, and also helps wear out the tape transport on your probably-much-more-expensive-than-a-250GB-HD digital video camera.
People into digital video see the cutting edge of where backup problems are headed. They see that the time required to just pump the data off the disk is becoming prohibitive. If disk transfer speeds don't suffer a quantum improvement soon, disks will become, from the perspective of full backups anyway, essentially like serial devices, were you have to focus on restricting access to the device because it's just so slow.
Windows XP appears to support rewritable CDs (couldn't get their rewritable DVD act together in time for XP, I guess), but it's really a bundled Roxio "one big burn" solution under the covers.
You can get third-party software that lets you treat DVD+/-RW and CD-RW "like a floppy", but so far reliability has remained suspect, and interoperability between different vendor solutions has been provably A Big Mess.
The re-recordable media problem is especially crucial to Windows backups. With dual-layer re-recordable DVDs coming, DVD could be a plausible Windows backup media for many situations -- if direct O/S support solved the compatibility mess.
I can't believe how many years will pass before Windows has any direct, API-level support for any form of re-writable disc. I also fear that when Longhorn finally ships, it will somehow manage to be one or more steps behind the current technology of that time, leading up to another umpteen-year delay to have decent backup solutions for the casual user.
Will they get Mt. Rainier support in, as promised? Seems like dual-layer RW will be a reality before Longhorn is -- will it support that? I wait with bated breath!
As storage sizes and needs relentlessly increase, the serial nature of tape becomes an ever-greater penalty. Large, expensive hardware solutions can overcome this with parallelism. However, it's the cheap, commodity solutions that drive innovation and ultimately dominate the market, and that's hard disk backup solutions.
One of the primary barriers to the dominance of disk-to-disk backup is simply form factor. Large, cheap hard disks generally come with their electronic guts hanging out, and without an easy to plug/unplug connection. This is in the midst of change, however. Tiny form factor disks that were designed from scratch to be robust and carryable are increasing their capacities at a rapid pace. USB 2.0 and Fireware offer enough speed to be plausible connectors to backup drives.
Another factor to consider is that the rate of change of disk capacity increase is much greater than the rate of change of disk data transfer rate. Thus, image backups of disks are becoming impractical in many situations, and restoring from tape becomes even more tedious, as you try to wind to the desired version of a file in a vast body of backup data.
The fact that people who have a tape solution that works have no immediate need to switch to disk does not mean tape is not being displaced as a backup solution. We still have lots of mainframes being used; that does not mean that PCs did not usurp mainframes as the primary means of computing -- they certainly did.
No one with an effective tape backup solution today should drop it immediately. Tape backups will be around for years to come. Tape backups are being displaced by disk-to-disk backups. All three things are true and not at all contradictory.
Right now, we're in a window of transition towards finding a dominant medium for backing up computer hard disks. Most likely, the winner will fairly soon be: not DVD, not tape, but simply other hard disks. Right now, the minimum "nice" size for a backup device is 40GB, and by the time a recordable dye-based disc can reach that, the bar will have been raised to 80GB.
Cost is getting close to being eliminated as a factor in using hard disk as a backup media. The remaining hurdle is really a generic protective package (e.g., a little shock protection and don't expose any electronics) and the ability to walk up and plug it into any PC.
IoMega has issued their hard-disk backup medium solution, but it's slightly pricey, and strongly proprietary. Perhaps someone will start wrapping small form-factor disks in plastic with a connector that can plug into (and be powered by) either USB 2.0 or FireWire.
Within a few years, only the most low-end and casual forms of computer backup will be on DVD instead of hard disk.
Instead of wood or linoleum floors whose dirt can be attached through relatively primitive means (water and cloth), we largely live on carpet. I can run the steam cleaner over my living room carpet apparently indefinitely without it ever failing to yield up more filth.
Instead of baths in porcelain tubs that get scrubbed at least weekly (to remove rings and, as a good side effect, germs), we're taking showers standing next filthy curtains, and neatly aerosoling germs straight to the lungs via the shower steam (if your house is an a high radon area and you take a lot of showers, might as well take up smoking too!).
Just general house cleaning has become both less common and less easy. Rember the phrase "spring cleaning"? Ever participated in one? Before anti-biotics made scratches and other small wounds of no account, keeping your local environment clean was a survival instinct as much as a social nicety.
At least in the case of shower curtains, however, there is a simple solution. Get a washable shower curtain (google for shower curtain washable cotton duck) and wash it in hot water once a week. One germ collector eliminated, and it's nicer brushing up against wet cotton than wet chemically treated plastic. You still need to Comet that tub once a week, though :-).
Other motivations include a growing desire for a slightly more cutting edge distro. I wanted the 2.6 kernel some weeks ago, and have an interest in playing with the new security models as well. Fingers crossed that I'll be a happy Gentoo camper from now on.
If you really want to wack on the mail headers and/or body, the Postfix 2.1 way is to essentially insert your own SMTP proxy into the stream. Not nearly as easy as writing a policy server, and (like most methods of reprocessing the entire mail message stream external to the MTA itself) likely to significantly reduce your mail server's peak processing ability.
The new policy server interface is a simple sockets-based API for getting a chance to participate in the SMTP conversation as it is happening. The basic idea is:
- tell your Postfix config file (main.cf) that you've written a "policy server" that listens on a particular Unix socket or TCP address/port.
You can have the policy server get "called" at any of the points in the SMTP conversation where Postfix may make a decision about how to dispose of the message (HELO, RCPT, etc.).
- write your policy server. It listens for connections, and each connection sends you one or more requests. Each request contains a small set of information about the mail message being transmitted (client name/address, HELO text, etc.) Your server responds with one of a broad set of actions that Postfix supports (reject, accept, defer, perform other custom checks, etc.).
- The protocol for talking to your server is a simple text-based protocol with newlines, much like the form of HTTP. I coded an initial policy server in good ol' C in about an hour.
In particular, this new API is a great place to implement greylisting. Your server can maintain its database of whitelisted and greylisted from/to/IP triplets all on its own and just respond to Postfix requests. And, once you've coded up your policy server, you don't have to revise it with every Postfix patch that comes down the pike. As long as the API remains backwardly compatible, your policy server code should survive any Postfix upgrades.Kudos to the new policy server API!
The result was: only about 2% of the spam would have gotten through. I think I can improve that rate by increasing my local spamtrap database to augment the larger one at cbl.absuseat.org. But even if I can't: 98% of spam eliminated in a 100% automated fashion, no tuning and tweaking and training. Completely automated spam removal, totally driven by the spammers themselves (they tell us what IP addresses they are using today by using them to send spam to a spamtrap address).
Greylisting + spamtrap RBL has some niggling problems, such as dealing with mailing lists that use a different sender address (and maybe even IP address) when they retry a tempfailed message. However, these problems seem manageable compared with solutions such as teaching every user to train a Bayesian filter.
To defeat greylisting + spamtrap RBL, spammers will have to locate all the spamtrap addresses in their databases and remove them. Good luck!
Greylisting + spamtrap RBL may not be a silver bullet, but it sure acts like one on my system.
It's kind of a slide-show study report, so it's hard to get at all the details. But, there's room for skepticism...
Residents outweigh attending physicians 2-to-1 in this study. Wouldn't residents be more likely to be younger? Aren't younger people much more likely to have significant video game experience? I can find no place in the report that shows they controlled for age. Might the study simply be showing that "younger people have better eye-hand coordination than older people?"
Aren't most new kinds of video games and equipment (I would suppose, including laparoscopic equipment) built by young people with young eyes? Don't most older (>45) people develop farsightedness? Might the study merely be showing that "laparoscopic equipment needs to be improved for surgeons who have older eyes"?
In "Methods and Materials", I saw a quote that made me think "skill" was partially calculated by how fast the operation was performed. Might not residents who have only performed 2 actual surgeries be more likely to risk going faster, unlike experienced physicians who, with many more actual surgeries under their belt, might be more inclined towards caution? Do I really want the speed demon operating on me, or the guy who goes "slow and steady"?
Don't many video games essentially teach "it's better to be fast than right, better to keep moving than stop and think"? Is that the mindset I want in a surgeon?
It was hard to determine whether the simulator being used was closer to a video game or closer to real surgery. Might the study merely be showing that "people who are better at video games are better at surgery video games"?
This study, or at least this description of it, failed to convince me that I want a Doogie Doctor doing my next surgery. I think I'll go with the guy who has had a couple hundred successful operations over the guy who smoked him on Mortal Kombat.
However, to be clear on one point: you say "you don't want to 'greylist' earthlink or AOL". To be accurate, you don't want to *blacklist* the IP address of a major MTA that serves many customers. It's still fine to greylist mail that came out of (for example) earthlink, because greylisting applies to a triplet (IP/sender/recipient). It will temporarily stop that same spam from being re-sent, but has no effect on other email coming from the same IP address, but whose sender/recipient pair differs.
And, FWIW, in my own mail logs, I have not seen any spam coming from a real AOL MTA in recent memory. Lots of spam with a forged address at AOL, some spam with an IP address that belongs to AOL, but none that would pass an SPF check (which should be applied before most anything else, as it's relatively cheap).
In fact, I should amend the title of that post to SPF + greylisting + honeypot RBL = works real good!
Step 1: Salt the spammer's email databases with guaranteed bogus email addresses that no legitimate email sender has ever seen. This is currently trivially implemented as follows. In your website's robots.txt file, list several files that robots must not examine -- these are your honeypot. Then, fill those files with HTML that contains your bogus email addresses. Spammers will, quite reliably, disobey the robots.txt file, use it to discover HTML files that are not linked to from anywhere else in the world, and add your bogus mail addresses to their database.
Step 2: Implement greylisting + honeypot-based RBL. When email arrives that is not whitelisted, see if it comes from an IP address that is "temporarily" blacklisted in your RBL. If it is, you can reject it right now. Otherwise, see if the target address is in your honeypot database. If it is, add the sender's IP address to your RBL and fail immediately. Otherwise, engage the now-classic greylisting algorithm (see http://www.greylisting.org/) to "tempfail" the email. The point of the temporary failure is to give the spammer time to use the same IP address to send the same spam to an address that *is* in your honeypot database, so you can then proceed to reject the retry of the spam to a legitimate email address).
- requires no per-user work, such as "training" of filters.
- requires no changes to any software, except MTAs (and only a handful of them handle most of the world's software). no new laws.
- no false positives. to get blacklisted you *must* have transmitted email to an address that could only have been obtained by illegally harvesting a website.
- even compromised home systems are not terribly harmed. if a spammer takes over your home computer and uses it, well, the IP blacklist need not be permanent, just long enough to cover a single spam run -- a few days is probably plenty. if the spammer is blasting out runs from your home computer continously, well then you have worse problems than finding yourself unable to send email to GrandMa.
- not easy to defeat. right now, anti-spammers must work very hard to locate the "real" email amidst all that spam -- and never, ever mistakenly reject a "real" email. greylisting plus honeypot RBL inverts the equation. the spammer must make sure that not a single "bogus" email address is anywhere in his database! spammers are ingenious, but developing absolutely perfect lists of legitimate email databases is something they have no experience with so far.
- no restriction of free speech. total whacko strangers who aren't spammers can still send you email -- it may just get delayed for an hour or so (a fact which is totally true already).
- nobody makes any money off it. you don't have to pay anybody, except for the effort involved in setup and maintenance (a fraction of the total time wastes on spam currently).
- computationally cheap. most MTAs are already looking up IP addresses and target addresses in databases. cost of this scheme should not greatly slow down most MTAs. especially compared to content-examination schemes such as Bayesian filters.
- no judgement calls in blacklisting. no third party has to decide what is spam and what is not. the rbl in this scheme is totally generated from absolutely bogus email addresses -- the only way you can get in the rbl is to flat-out declare yourself a scumbag by sending to one of those illegally obtained addresses.
No scheme is perfect, but greylisting combined with an RBL that is derived solely from bogus email addresses is pretty damn good.Or use SWREG for $1 + 4% and not have to worry about having your bank account frozen. Last I researched it, they seemed like about the cheapest reputable shareware cc proxy.
However, if the microwave simply supported an IR remote, then I could reprogram one of my hackable Radio Shack remote's buttons to pause the ReplayTV and mute the TV everytime I pushed the "cook" button. (Hmmm, I think I would use Shift-Freeze to be the "cook" button. :-)
They pretty much all read DVD-ROM discs, except possibly for some DVD-RAM drives (which few are buying), so that would not be a good example of why one should wait for a single standard.
If you really are concerned about being able to buy a drive that reads your old discs in 10 years, then pay the high dollar for magneto-optical (MO). Pledged backward and forward support is its main claim to fame at this point.
- price: the cost per GB of DVD-R crossed hard disk prices recently, though they are still very close.
- durability: DVD is not as susceptible to physical shock and magnetic fields as a hard drive.
- movability: more PCs can read a DVD-R than have a slot for inserting a removable IDE drive.
- size: when what you want to store fits fine in 4.7GB, a DVD is a much nicer form factor than an IDE hard drive (so far). (e.g., daily incremental backups extending back for a full month.)
- movies: I can't create a movie on a hard drive and then stick it in my consumer DVD player (so far).
However, DVD+RW and DVD-RW would certainly be more attractive for general data use if the operating system actually supported them as random access devices. Don't know about *nix, but Windows does not support such access until the next version (XP supports drag and drop, but simply copies files to a temp area, and then waits for you to tell it to do the One Big Burn).