Amazon's New Storage Service

dlaur writes to tell us that today Amazon announced their Simple Storage Service (S3) allowing users to store unlimited amounts of data at $0.15 per GB paid monthly. From the article: "S3 was purportedly built to support both Amazon's own internal applications and the external users of the Amazon Web Services platform. That should be proper motivation to build a service that's fast and robust enough for mission critical use, yet flexible enough to support any storage task thrown at it."

Comments

Encryption

[grahamsz]

I wonder how long it'll take to build a backup solution that encrypts your data locally with a private key before sending it off to amazon. That way they wouldn't be able to look through it, and at 15c/month/gig it'd be pretty affordable for home backups

Re:Encryption

[this great guy]

Current hard disks sell for 40c/gig. If you plan to keep your data for >3 months, it is more economical to use you harddisks. Though your solution has other advantages (data accessible from anywhere, no need to administer yourself your servers, etc).

Re:Encryption

[grahamsz]

But hard disks present their own challenges.

You should really use some sort of redundant arrangement to make sure that a failure of your backup device wont result in data loss.

You then need to either offsite the drives or keep them in a firesafe, in which case you probably need two sets of them so that you can keep one live and the other somewhere safe.

And of course the amazon solution leaves your data accessiblwe from anywhere.

Re:Encryption

[The Hobo]

Upload truecrypt files

Open source, cross-platform, creates a strongly encrypted file that the program can mount as a real HD, you can mount it on any platform, does transparent encryption (for example in WinXP, it mounts itself with a drive letter, you can throw stuff in directly just as if it were a real drive, and it encrypts as it goes in)

http://www.truecrypt.org/

You can do it in say N meg chunks or something, I guess you'd have to create a new truecrypt partition every time, but I don't really know much about it, just tried it out and it seems neat

Re:Encryption

[eric76]

First, hard drives are a very poor choice for backups. In addition to media failure, you also have serious problems with failure of the electronic hardware. There are just too many single points of failure. Tape is far superior.

Second, unless you have an extraordinary fire-proof safe, off site if far superior. Fire-proof safes are really only fire resistant -- it is merely a question of how long it takes for the internal contents to get warm enough to destroy the contents.

Re:Encryption

[grahamsz]

I'm of the belief that hard drives can be built into an excellent backup system. So long as you expect that they'll all fail and design around that then you can do fairly well.

I'm not really sure about the fire-safe issue. I know my old univerirsty (Edinburgh) had a catastrophic fire a few yaers ago and they pulled the backup tapes from the firesafe the following day and they were back up and running in no time.

The other problem with tape is that you need discipline. You need to actually go in each day/week and make a backup and do something with the tape. I'd prefer something that didn't require my interaction.

What sort of tape system would make good sense for a home user?

Re:Encryption

[misleb]

Except that the upload speeds for most home users is pretty bad. You'd spending a lot of time pushing files depending on exactly what you chose to backup.

-matthew

Re:Encryption

[jcr]

First, hard drives are a very poor choice for backups

I beg to differ. Hard drives are the fastest, easiest to use, and most convenient option available today. Tapes simply haven't kept up with the growth of hard drive capacity.

Tape is far superior.

Nope. If you're doing anything other than a complete dump or restore, tapes are a major pain to deal with. It's far more likely that I'll be recovering a single file than an entire volume, and I can be waiting for a LONG time if that file happens to be near the end of the tape.

-jcr

Re:Encryption

[jcr]

I wonder how long it'll take to build a backup solution that encrypts your data locally with a private key before sending it off to amazon.

If you're using a Mac, then it's basically already done. Create an encrypted disk image in the Disk Utility program, mount it, use it like any other volume, and send the .dmg file to Amazon anytime you want it backed up.

-jcr

Re:Encryption

[Master of Transhuman]

Here we go again.

Tape has fewer points of failure than a hard drive?

Oh, please...

Explain to me why the entire industry is moving to disk-to-disk recovery backup with tape relegated to archival backup.

Explain to me why most data is kept on hard drives for day to day use if they are so failure prone.

Optimum backup requires disk-to-disk for quick and absolutely reliable recovery. For security, disk-to-disk over the network to an offsite location allows for fully automated reliable offsite backup, but it is expensive in bandwidth even if you only transmit deltas. For offsite storage where netword bandwidth is not available or too expensive, for long-term archival storage, tape is useful - provided the tapes are maintained properly, stored properly and not overused.

Modern tape systems can be very fast and very large, and can cost less than equivalent capacity disk drives, but the fact of the matter is that industry studies show more problems with tape backups than with disk backups. Between equipment failure and operator error, tape backup is problematical for recovery purposes.

Re:Encryption

[Anonymous Coward]

Under Linux, does this have some benefit over cryptoloop, which is already included in the kernel?

Re:Encryption

[Xenna]

The way you describe it makes it sound quite useless. We need full as well as incremental backups (who wants to upload 10 GB every night???) that can be automated to run nightly. Manual backups are worse than useless.

A good Open Source encrypted back up solution that makes use of this in an intelligent way would be great. I really want to automatically store my photo's and important document outside my house. Houses do burn occaionally, computers get stolen by burglars. No need to loose all your important data in the process.

X.

Re:Encryption

[LordLucless]

Tape has fewer points of failure than a hard drive?

It does, because, unlike hard-drives, the media and the reading mechanism separate components. If your read head drives on your hard-drive, it is difficult and expensive (but not impossible) to retrieve your data.

Explain to me why the entire industry is moving to disk-to-disk recovery backup with tape relegated to archival backup.

Convenience

Explain to me why most data is kept on hard drives for day to day use if they are so failure prone.

Convenience

Hard drives are more prone to failures than tape drives, but that can be alleviated through stuff like RAID. Hard drives are more convenient than tape for all but the most fundamental backup needs (full backup, full restore).I prefer to use hard drives too; but they are more prone to failure than tape. If I had to choose to trust all my data to a single tape or a single hard drive, I'd go tape every time. If I had the capacity to create a redundant array of hard drives, I'd go with hard drives. If I needed offsite storage on a budget, I'd go with tape - it's easier to transport and store than a hard drive array. If I had the money for it, or my needs were simple enough that the solution wasn't that expensive, I'd go for a local hard drive array backup, and a remote network backup.

That last one is, in fact, the backup system I use at home. I have a cheap RAID array, and a script that encrypts my most important files and FTPs them to a friend's computer once a week. My important files are mostly source code and documents I've written myself - it doesn't chew through much bandwidth or storage space.

Re:Encryption

[Anonymous Coward]

No need to loose all your important data in the process.

"lose".

Re:Encryption

[MORTAR_COMBAT!]

with truecrypt, your linux partition could be on a USB key (or copied, or on a shared partition, etc) that could be shared with a windows machine. truecrypt is really a nice piece of software, even though i have absolutely no "real" use for it other than storing passwords and the odd bit of personally sensitive backup like tax returns. the danger with all encryption is when people use it for illegal purposes (terrorism, copyright infrigement, child porn, theft, fraud, etc), but it is a powerful tool for privacy and legal communication. in my opinion, criminal abuses of encryption should be punished severely, because it is these abuses which hurt useful private sector encryption (protecting company trade secrets, personal diaries, personal communications, whatever). the problem with the power to strongly encrypt everything is that "bad people" gain a lot through this power, perhaps more than can be accepted for the gains that "non-bad people" receive.

Re:Encryption

[arivanov]

You are both right and both wrong.

There is no such thing as "the general purpose backup problem". There are two problems - backup as a defence against a luser error or minor failure and backup as a defence against a system failure or a catastrophic failure. Or in backup architecture terms you have to deal with operational recovery and disaster recovery.

Tapes suck rotten eggz for operational recovery. With all advances they are much slower then disk based solutions.

Disks suck for disaster recovery. Once you add removeability in the equation the cost of backup to disk for disaster (offsite or fire safe) recovery is much higher than tape. The requirement for removeability may be removed if the whole backup solution is off-site and connected via a suitable network. In that case disks get back their advantage over tape, but very few companies have the infrastructure to afford this.

By the way - that is valid for "general purpose backup". Which in fact is only a minor fraction of any backup expense. Applications usually are a much bigger pain in the neck.

Re:Encryption

[PFI_Optix]

Aside from convenience and speed, you're forgetting another component: cost. Hard drives are significantly cheaper than tape, and don't require regular replacement.

At a previous job, our typical SAN solution was a fibre RAID (or two if super-redundant) that was backed up to a larger SATA array. A hard drive backup was about half the price of tape, significantly faster, and much more convenient. With RAID 5 and a hot spare (or two) the only downside of backing up to SATA is the lack of portability...but if you're buying a SAN in the price range that we sold, you probably had a second site and at least a T1 between them. We set up quite a few secondary backup systems at remote sites for incremental backups to be run overnight.

Amazon and other storage hosting solutions can provide an excellent extra level of resilience for data. At a monthly cost per TB amounting to less than a lot of admins make in a day, a lot of smaller-than-enterprise operations would consider it well worth the expense.

Re:Encryption

[hoggoth]

> Once you add removeability in the equation the cost of backup to disk for disaster (offsite or fire safe) recovery is much higher than tape.

An external IDE case with USB 2.0 and Firewire ports is about $50 at CompuUSA.
Now explain to me how adding 'removeability' to the equation makes disk much more expensive than tape?

Re:Encryption

[Anonymous Coward]

I know of several facilities whose "fire safes" which used to intentionally left unlocked or the combination is posted. The fear is someone will break in and not realize what's inside and do damge to the contents without recovering anything of value. This isn't the case any more, but 25-30 years ago, this was not an unknown practice.

Re:Encryption

[shokk]

Obviously Amazon S3 doesn't fill that niche. However, there is inherent value to offsite storage that the local harddrive cannot fill. Disaster recovery is getting to be a big deal, less than a year out from Hurricane Katrina. The question is, if I put my data on S3, what is Amazon's redundancy plan? This is a pay service so I expect that if that datacenter gets blown away, there will be a backup of some sort. This is a pay service and you may have some enterprises relying on this.

Re:Encryption

[Milican]

I love TrueCrypt. I have been using it daily to mount an encrypted volume from a USB Flash Drive drive with portable apps like Portable Thunderbird, Portable Firefox, Portable GAIM, and Portable FileZilla on it. Very easy to use, and keeps my personal information under wraps.

JOhn

Re:Encryption

[networkBoy]

It is important to note that tape is less reliable than HDD at a consumer level. If you only back up your data (not media) then the web solution means that you are likely to pay only ~$1.80/year for offsite storage of your truely important data (scans of marrige/birth certs, passport, SS card, masters thesis, etc.)
-nB

Re:Encryption

[70Bang]

Someone on ars technica was whining about a warrant or subpoena would no longer be needed to examine your material because it would be on someone else's machine.

That's when I pointed out people wet their pants over things which can be negated with something simple[1] (encryption).

"Make things simple, not simpler." -Erasmus

Re:Encryption

[70Bang]

Once you add removeability in the equation the cost of backup to disk for disaster (offsite or fire safe) recovery is much higher than tape

It depends upon the environment & circumstances. When I was responsible for the mainframes, a couple of midrange, & related resources, I periodically made my own backups in addition to what was normally scheduled. Being the paranoid bastard I am, downtime for upgrades were scheduled long enough for me to make an extra set. Those extra sets made a trip home to my garage in nice containers "just in case".

I'm still paranoid with "personal" machines which I can reformat & start over with here at home. There's always a little piss shiver simply because of how many things have to be aligned for the machine to come up correctly and only a couple to prevent it.

Re:Encryption

[Anonymous Coward]

The benefits are, compatibility with Windows, plausible deniability and more secure mode of operation.

Re:Encryption

[Pollardito]

15c/GB is not the complete picture :

The storage space is accessed by standard SOAP and REST interfaces, and networking is handled by HTTP and BitTorrent protocols. The data streams are encrypted with customer-specific keys, and access rights are supposed to be granular enough to provide private or public storage object by object, and user by user. Apart from the storage fee, you pay $0.20 per gigabyte transferred, but there are no minimum fees and no setup costs, so you pay as you go.

BitTorrent is a strange protocol to use to deliver personal storage, is that just a fallback in the event that you choose public storage and multiple people are accessing at once?

Re:Encryption

[nizo]

Speaking of reading your backup meadia, one nice thing about backing up to disk drives is if my office building burns down, I don't have to look around for another tape drive (like the one that burned up in the building) to read my tapes; I simply find a computer and plug my harddrive in and bingo, my backups are available.

Re:Encryption

[nizo]

That and a CVS or Subversion interface and I am set.

Re:Encryption

[jandrese]

Yeah, that's been my experiance too. Tape drives are insanely overpriced, slow, and more difficult to work with than hard drives. Unless you're looking at truly enormous amounts of data, hard drives are almost always the way to go. For home users tape drives are not even close to competative.

Re:Encryption

[PFI_Optix]

At home, I have a file server with four drives on RAID 5. The total storage is somewhere around 400 GB. I've got room for one more drive, I've been thinking about adding a 500 GB drive for backup. It's expensive, but far cheaper long-term than tape.

Re:Encryption

[geoffspear]

Q: How reliable is Amazon S3?

S3 was built to the same service level standards that Amazon requires for its own web sites, with a targeted 99.99% availability, and storage of each piece of data on multiple servers in multiple Amazon datacenters.

I'm guessing there aren't very many enterprises that will be using this who are bigger than Amazon or with more to lose if their data is wiped out.

Re:Encryption

[Glsai]

True upload would be pretty bad, but I imagine anyone who is going to back up anything of large size will use broadband. Plus you could always do all your uploading at night or while at work to help mitigate this. I'm really looking forward to this as it'll allow me to backup all my digital photographs. I've already stored it on two different harddrives for redundancy, and given backups to all my family on DVD to help with offsite backup... but this will be nice as it gives me even another layer of safety, and even if I had 10gigs of photographs (think I'm only at 6 or so) I'd only pay about $1.50 a month. If google offered this for free, I'd upload a copy there for more redundancy.

Re:Encryption

[adolf]

It's not that bad, really.

I've been doing remote backups for a few months now of our mail/web/DB/whatever server at work, over what is essentially slow consumer-quality cable (384k up).

Speeds are good. The initial backup took some time (a weekend), and now consumes between 25 and 35 minutes daily (yay rsync). What's more, with it rolling on a daily basis, it seems that disk is the primary bottleneck, not the network.

I don't know why it'd be any different for a typical home user.

So, $.15/GB/month, Amazon's service sounds like it'd be a cheap and very effective (ie, automatic) insurance policy.

All the world needs now is for someone to write a usable, drop-in, encrypting, OSS backup tool for it. Preferably with some concept of copy-on-write snapshots, so that multiple backup generations may be kept at low cost.

Re:Encryption

[dubl-u]

It does, because, unlike hard-drives, the media and the reading mechanism separate components. If your read head drives on your hard-drive, it is difficult and expensive (but not impossible) to retrieve your data.

This is possibly a good argument for commercial situations, but I don't think it applies to casual users. I now have a variety of old tape and removable media items sitting around and no matching drives; they died or were never mine to begin with. But the old hard drives I have all still spin up.

Re:Encryption

[symbolic]

I didn't see too much value in this service, but I will say - after someone mentioned offsite backups, I can see where this might be very useful to some people who don't necessarily need to store huge amounts of data, but data that is critical nonetheless. Make no mistake - if I ever decided to use a third party for storage, I'd encrypt the hell out of it using a key of sufficient length as to make brute-force decryption impossible.

Re:Encryption

[jcr]

We need full as well as incremental backups (who wants to upload 10 GB every night???) that can be automated to run nightly.

So, do it. Use rsync to see what the diffs are between the full DMG you have on the remote server, and make another encrypted DMG with the daily diffs.

-jcr

Re:Encryption

[rob_squared]

Connected Dataprotector encryps locally: www.connected.com.

Disclaimer: I work there.

Re:Encryption

[metamatic]

It does, because, unlike hard-drives, the media and the reading mechanism separate components. If your read head drives on your hard-drive, it is difficult and expensive (but not impossible) to retrieve your data.

Speaking as someone who worked in data recovery for a number of years...

If your write head on your tape drive goes out of alignment, it's difficult and expensive (and sometimes almost impossible) to retrieve your data. Been there, done that.

If your heads on your tape drive get magnetized, it's difficult and expensive (and sometimes almost impossible) to retrieve your data. Been there, done that.

If your tape stretches, wears thin, breaks, or develops dropouts, it is difficult and expensive (and sometimes almost impossible) to retrieve your data. Been there, done that.

The usual solution tape proponents push to improve reliability of tapes is to never re-use a backup tape, or to use it only 10 times then throw it away. Of course, that makes tape astronomically expensive.

I'd take redundant Seagate hard drives over a tape drive any day.

Re:Encryption

[Susceptor]

you forget that DVD's allow for plenty of storage to the typical user anyway. i know a lof of users here will have 200GB or more, but the typical user will probably have only a few gigs worth of pictures and files that he/she would consider worth saving. And as far as video, most people save those on DVD's anyway. So the benefit of this enormous storage seems to be predominantly for hardcore users who have a lof of data. For anyone else, DVD's are more then enough. Then there is the new HD and bluray disks that will store anywhere from 30-100GB on a dingle disk. I would be very surprised if that is is not enough for your typical user.

Re:Encryption

[Master of Transhuman]

The other posts responding to this made most of my points.

The idea that the read heads on a disk drive are particularly vulnerable compared to tape read heads is nonsense.

Hard drives are less vulnerable to failure than tape drives - and more importantly FAR less vulnerable than TAPE MEDIA.

It's the MEDIA that is vulnerable in tape - and responsible for most of the backup failures using tape.

There's no way tape can compete with disk for reliability.

This article from Techworld about the Baptist Memorial Health Care chain demonstrates the issues involved: http://www.techworld.com/features/index.cfm?featur eID=2254&printerfriendly=1

Money quote:

"People's lives will be affected by how well this works," Weiss says. "We are also saving brain cells," he adds. "This is a technology that we do not need to think about. It simply does its job. The stress of not knowing whether backups/restores will work is gone. Our backup failure rate has gone from more than 30% to absolute zero, and we have not had a backup/restore failure since installation."

Re:Encryption

[Master of Transhuman]

Removability is not an issue.

The problem is trying to compare the cost of a removable container to the VALUE of the DATA and the risk factor of losing that data. There simply is no comparison. The cost of the hardware is minimal compared to the data value.

However, I will agree that a removable container WILL increase the risk, possibly comparable to the level of tape (a removable media). That is why backup over the network is better. I agree that many smaller companies might not be able to afford this.

However, smaller companies also don't have the volume necessary to justify large tape backup systems either. A smaller company can get probably get by with archival/disaster recovery backup to a removable disk - or even DVD backup if the backups are incremental instead of differential. Using incremental backups is not an issue if the backups are archival rather than recovery. So in the end a large company can afford network backup and can use that to avoid the issues of removable media, where a smaller company can afford removable media and use that to avoid the expense of a network backup.

Besides, as I pointed out, there is also a difference between archival backup and "disaster recovery" backup. Tape is adequate for archival because while it is possible the data may need to be restored, the company's operations are not at risk if it isn't (other than legal risk if the data is needed for legal reasons). Tape is also adequate for disaster recovery because under the conditions of a disaster, quick recovery is not required.

Operational recovery, as you correctly point out, is perfectly suited to disk backup.

I fully agree that there are a number of backup issues and they all have to be addressed with the procedures and technology suitable for each issue.

Re:Encryption

[jp10558]

This is all great, but Streamload will offer you 25GB for free, and you can dl 10MB files for free, or pay for one month of however much data you need to download. You only pay to download data files over 10MB.

Assuming important files, it's likely you'd never need to pay. That all said, for basic files, yahoo briefcase would do too.

The big problem with streamload or any web solution is our upload speeds make any real backup of more than then basic text or some pictures impractical time wise.

Re:Encryption

[jp10558]

www.streamload.com will offer you 25GB free, with limited downloads (10MB per file, 100MB per month) or pay for more downloads or storage space - the next level up goes to 250GB for $10/month.

Re:Encryption

[penguin-collective]

Except Amazon doesn't seem to support rsync, which makes uploading to it a big pain. Fortunately, many other hosting plans do.

Re:Encryption

[afidel]

How is HDD cheaper than tape? 400/800GB LTO3 tapes are $100, 400GB HDD's with no harware compression are $200+.

Re:Encryption

[afidel]

If you have a 30% failure rate you are doing something SERIOUSLY wrong. My tape system does a verify of data every night and I just had my first failure in a year recently. We clean our drive biweekly and have another system at our DR site which we send tapes to regularly to test. I've never failed to recover a file. I've had similar experience with every enterprise tape system I've ever used from reel-to-reel to DAT to LTO.

Re:Encryption

[misleb]

What you need is Linux and GmailFS. Get a few accounts and you are set.

-matthew

Re:Encryption

[Master of Transhuman]

Reread the article - look at the amount of data they were backing up. Figure out how many tapes they needed to use every day to do that - then figure the odds if even a couple percent of them fail periodically.

And you verify your data nightly and still had a failure - even after cleaning the drive every two weeks. What more do you need to know? When's the last time you "cleaned a disk?"

When's the last time you did a disk check and then had a failure to read a sector? Most disks today don't even report bad sectors - they map them out - until the disk is near failing.

Tape is history for anything but archival and last-ditch disaster recovery.

Re:Encryption

[Anonymous Coward]

You can read hard disks that survived a building fire. Cool.

Google redux

[October_30th]

Now that even Google has been ordered to hand over all the search data, you can bet that the government will want access to all S3 data, too.

Re:Google redux

[Shihar]

Google has NOT been ordered to "hand over all the search data". Google has not been ordered to do anything in fact. The judge has hinted that he intends to give the government a random sampling of the websites Google has indexed, but not individual search terms.

As far as the government wanting a pile of businesses data bases to search for porn or what not, that is simply illegal and would result in a prompt judicial smack down. If they want to know what is on someone's serve, they need to do it the old fashion way, with a search warrant... well, in theory at least. The executive branch these days seems to consider search warrants as being optional.

As tempting as it is to go off topic on the case of the executive branch ignoring warrant laws, I'll just drop a link and declare both parties worthless and pathetic.
http://www.washingtonpost.com/wp-dyn/content/artic le/2006/03/14/AR2006031401519.html

Re:Google redux

[jettoki]

This isn't flaimbait. It's true. What happened yesterday on the floor of the senate is a good reason to be afraid of warrantless search. Right now the only people standing between your private data and the executive branch are Google and Feingold.

Re:Google redux

[lbft]

So what?

I don't understand why people don't automatically distrust the security of any data not under their direct control. Data held by anyone else could be (mis)used by someone you don't want using it - be it a government agency, an employee causing trouble, a naughty contractor, a script kiddie who got access to something he shouldn't have access to, or any one of a million other people or groups.

If you have sensitive data, you should be taking steps to ensure the protection and integrity of that data, no matter what you're doing with it. Encryption is the most obvious solution, although it's not the answer for every situation.

Re:Google redux

[Anonymous Coward]

His what?

Re:Google redux

[Anonymous Coward]

mmmm your a real intellectual

"Mmmm" (capitalized), "you're".

Re:Google redux

[drooling-dog]

I know, and it's depressing. The repubs are like the bully down the street that beats up the little kids on the block and steals their lunch money, and (most of) the dems are the older kids who watch in dismay but do nothing but offer their sympathy afterwards.

Re:Google redux

[Anonymous Coward]

...or as I've stated elsewhere, if you have personal or sensitive info, DON'T STORE IT WHERE IT CAN BE SEEN! If the government, or anyone else whould like to peruse my work inb progress pictures of my robotics projects, they are welcome to waste their time doing so. But I'm not going to be storing anything online that I wouldn't want someone else to see. It's like, if you want to hide your recent online purchases from your significant other, don't leave the delivery receipts on the kitchen table. ;)

API proliferation

[gordyf]

It's good to see more sites adding APIs to their web services (Amazon already has other web services, as does Yahoo and Google of course). It's becoming even easier for "mere mortals" to link together new technologies to make innovative new systems, but I wonder if this reliance on third-party systems comes at a cost, perhaps to reliability or security?

Re:API proliferation

[Anonymous Coward]

I wonder if this reliance on third-party systems comes at a cost, perhaps to reliability or security?

If you don't do it right, oh, yes, very much yes. There are n-thousand different components along the link from your business to Amazon's datacenters. All it takes is one backhoe to cut a fibre and your company has gone dark, or one errant line of code and you're spewing critical customer data in the clear.

Do it right, though, and you can increase reliability and security. Use S3 as a backup for your small business that needs a bit of off site store. Put network monitoring in place to watch for those silly mistakes so you can fix it before it becomes a major embarrasment.

It will be interesting when Google's GDrive comes on line -- then you can be redundant across providers (RAID = Redundant Array of Independent Datacenters?).

Re:API proliferation

[maxwell demon]

With mail providers offering unlimited space, you could also do an "email backup": Just mail your encrypted data to your webmail account for backup, and read mail from that account for restore.

However, you have to be careful that you don't try to decrypt SPAM mails in the process :-)

Re:API proliferation

[amias]

i could tell you if gmail hadn't just deleted my gmailfs.

How they are implementing it ...

[DrJimbo]

For each Gig of storage you use, Amazon gets a free Gmail account and uses that for storing your data.

With virtually no cost for this storage, they can make a killing charging $0.15/Gig/month.

Re:How they are implementing it ...

[__aailob1448]

Actually, Gmail offers around 2.6 GB of storage now.

In other news...

[dcapel]

The Department of Homeland Security announced it has started a data hosting service. They encourage backing up of family pictures, journals, and irc chats, among other things. There is no monthly cost, but encrypters need not apply. When questioned on how this is a good use of taxpayers money, they simply replied that they wanted material to test their new indexing algorithms on.

Apply now at database.dhs.gov/personal/suspects/index.php

Re:In other news...

[grasshoppa]

You just *know* somewhere, behind some desk, in a dusty office, a DHS agent is reading this going, "Damn, why didn't I think of that?!"

I could see it now. All they'd need is a front company. Someone the public trusts. They could release this service...and..uh...

Wait a second.

Anyone remember GoogleZon?

[Nomihn0]

This is an interesting case of diversification. Amazon, no longer content to be the middle man e-tailer, is shifting it's weight into Google's territory with a service-based profit model. If this trend continues at Amazon, I have to wonder if Google will make a hostile bid for its newfound competitor.
 
  Here is a Link to EPIC, a speculative piece on the future of media, including the GoogleZon segment.

Re:Anyone remember GoogleZon?

[Anonymous Coward]

A merger of Google and Amazon should obviously result in GoonZoo.

Re:Anyone remember GoogleZon?

[Anonymous Coward]

shifting it's weight

"its".

Re:Anyone remember GoogleZon?

[Anonymous Coward]

Thank you-- I realized that soon after posting.

Sign me up

[Scott Swezey]

This sounds good, as of lately my maxtor one touch HDD has been getting more and more corrupt sectors, and I am sick of running chkdsk overnight on it. I figure it's going to die on me soon and I don't look forward to getting another HDD.

Anyways, back on topic, at $0.15 a gig, it would take a long time before buying a hdd would be more affordable for me. (my hdd is 250g, I use about 100g, 100g = 15$, so after 10 months thats 150$... Still cheaper than this HDD that I didn't even get a year ago, on sale, for 200$)

I wonder if they will have an issue with my "favorite Bible quotes/hope_cops_dont_see_this" folder, or all of the iso's I downloaded *shify eyes*

Re:Sign me up

[EvanED]

Anyways, back on topic, at $0.15 a gig, it would take a long time before buying a hdd would be more affordable for me. (my hdd is 250g, I use about 100g, 100g = 15$, so after 10 months thats 150$... Still cheaper than this HDD that I didn't even get a year ago, on sale, for 200$)

Um, you should look at hard drive prices of today if you're going to be comparing server prices of today. Even retail prices put a 160 GB hard drive at $120. If you are willing to count the rebate price of that drive (it was at the top of the list; I didn't choose it because of the rebate explicitly), it's $50. That's 80 cents and 31 cents per gig respectively. Even if you count just the 100 GB, with rebate that's only 50 cents/gig. In under 4 months that way you'll break even.

Besides, whatcha gonna do? Run your programs remotely? Run your OS over the internet? I don't think so. You'd need a local mirror anyway, so you'd need that new hard drive.

This service has a lot of use, but from a backup standpoint I do NOT think it's at all a good option. Too expensive and too much hassle transferring that much data to make it worth it. (Are you really going to upload 100 gig? Even at a sustained rate of 150 KB/s upload (quite good from my experience over cable) that'd take over a week.)

Re:Sign me up

[EvanED]

I know it's lame to reply to yourself, but I want to amend this by saying that I think it would work great for small backups, like if you just want to backup the documents you've written. But doing that has little impact on the hard drive situation anyway, because you'll still need to get a new one before the current one dies or face a LOT of time reinstalling stuff (at which point you'll wish you spent the money now). Putting stuff on Amazon might let you delay a bit more though and still keep the peace of mind that your truely irreplacable data is still okay.

Re:Sign me up

[Leolo]

A hard disk won't survive your house burning down.

Or put it another way: this would provide the all-important off-site backup.

Re:Sign me up

[Kludge]

This service has a lot of use, but from a backup standpoint I do NOT think it's at all a good option. Too expensive and too much hassle transferring that much data to make it worth it.

3 words: Off-site backup.

The hassle isn't too great when your place burns down, or lightning strikes your box (and don't think a "surge suppressor" is going to save you), or the cops come and grab all your computer stuff. Some stuff you don't want to lose.

Re:Sign me up

[hhghghghh]

3 words: Off-site backup. The hassle isn't too great when your place burns down, or lightning strikes your box (and don't think a "surge suppressor" is going to save you), or the cops come and grab all your computer stuff. Some stuff you don't want to lose. 17 words: Shoving A DVD-R In An Envelope And Sending It To Your Mom's Place In Your Own Name

Re:Sign me up

[tepples]

3 words: Off-site backup.

3 more: Safe deposit box.

Re:Sign me up

[menkhaura]

I don't need an envelope to send a DVD-R upstairs...

Re:Sign me up

[EvanED]

It's still easier and, in the long term, cheaper to get an extra hard drive (or really two, so that you can rotate) and back up to that. If you're talking about doing a full hard drive back up, Amazon would be way too slow and cost-prohibitive. Even ignoring bandwidth charges, it'd take under a year to recoup the extra initial investment in getting a hard drive. Bandwidth would be unmatched. Just do, say, weekly rotations of two drives. Back up to one drive, at the end of the week move it offsite and get the hard drive that is offsite and bring it back to back up to the next week.

If you're talking small quantities of data, CD-R would be still be cheaper and easier but, as I said in a reply to my first post, Amazon's service would work pretty well.

Re:Sign me up

[Anonymous Coward]

Ah, there is more to it than just $.15 a month per GB.

Pricing

        * Pay only for what you use. There is no minimum fee, and no start-up cost.
        * $0.15 per GB-Month of storage used.
        * $0.20 per GB of data transferred.

http://www.amazon.com/gp/browse.html/104-9335792-4 570331?node=16427261

Re:Sign me up

[CCW]

Mailing a DVD will cost (minimum) .39 (stamp) + .25 (blank) / 4.3GB = 0.15/GB

Hmmm. Maybe not so much cheaper after all? Especially when you consider that the DVD may not be full, which won't affect your costs in the DVD case but will be cheaper with Amazon.

Re:Sign me up

[EvanED]

The 39 cent stamp won't be charged if you can take the DVD somewhere (e.g. the office), and you didn't take into account bandwidth charges with Amazon.

Also, you need to take into account time. It's possible to do incremental writing to the DVD, so if you generate less than 4.3 GB of data/month the 25 cents/DVD (which is actually probably a bit low) should be reduced by the frequency you have to buy a new disc, or the 15 cents for Amazon should be increased to cover that period.

(This is one reason not mailing it away makes sense, so you can continue writing. Also, if you'd need to restore and you've mailed it away, you're at the whim of the person you sent it to and the USPS for when you get it back.)

Worthless

[Mindcry]

lets see... for a year of 200gigs, that's $360 USD.

couldn't I jut buy a new hard drive every year or burn hundreds of DVDs for far far less? not to mention they'd be secure from whatever prying eyes or security holes an online backup provides.

Re:Worthless

[grahamsz]

But what if that hard drive fails.

Well sure, hard drives are cheap why not just keep two of them.

But then what if your house burns down. You'd better make another set of your backup drives so that you can have one array live and one array off-site.

This seems like an amazingly good price for managed storage, if it's as reliable as amazon claim then there's certainly some data that i'll put on it.

Re:Worthless

[Shihar]

Clearly it isn't good for all uses. Does it sound damn tempting for a person a few gig of personal documents they want backedup? Hell yes. HDs and DvDs can break or be lost. For a couple of bucks a month you could back up everything, have it in a safe place that is not going to break or burn down, and know exactly where it is.

Re:Worthless

[mikeage]

you could. But not everyone wants to back up 200GB worth of data.

I have my backups categorized as follows:

High priority (documents, records, etc): 150MB
Medium priority (digital pictures, code, etc): 8GB
Low priority (movies, mp3, etc): 430GB

The first gets backup up nightly to a remote machine, as well as weekly dumps to CDs
The second is rsync'ed nightly to my website (not my machine -- shared hosting)
The third gets a RAID5 array, but that's it

For the first (and maybe the second) category, Amazon would be much more economical than doing it myself onto another, designated, disk.

Re:Worthless

[Mindcry]

and what would stop you from using encryption and a gmail file system (GFS) instead for free (unlimited and forever) storage?

Re:Worthless

[hritcu]

In that case you can buy a fire-resistant IBM ThinkPad. Or was that just an urban legend?

Re:Worthless

[MORTAR_COMBAT!]

my first use of this is to back up my itunes purchases. 507 songs, say at 4 MB per song, is about 2 GB of data. the upload will cost $0.40 and it will cost $0.30 per month to store the songs there. this is PERFECT.

(until google comes out and is free.)

Re:Worthless

[nanojath]

Well, right, it's the remote factor, not that it's cheapest way to back up. But I have to admit, I did the math on my own puny storage needs and I was like, "70 bucks a year? Shit, man, I'm not paying that for backup." I wonder what I would have thought in 1990 if someone had told me that in less than two decades I would be able to purchase the storage to maintain a library of something like 50,000 ebooks for well under a hundred dollars a year.

Re:Worthless

[dubl-u]

lets see... for a year of 200gigs, that's $360 USD. couldn't I jut buy a new hard drive every year or burn hundreds of DVDs for far far less?

Yes, and then you'd have a lot of storage in the same place as your existing storage. A theft or a fire and your data's gone. Unless you carefully schlep the DVDs elsewhere and keep them up to date. Which most people never do.

A better comparison is getting a collocated server and using it for backups. Given that people are charging $360 a year for a modest slice of a virtual server with only 12 GB of disk, Amazon's price seems pretty decent.

And it's a smart use of Amazon's brand. People are very conservative with backups. I would never use a startup to host my critical data, as there's no reason to trust them. Amazon, though, won't just close up shop one night because the bank account is empty.

People happily pay $30 bucks a month for physical storage; there's no reason they won't pay for virtual storage, too.

Re:Worthless

[Anonymous Coward]

But then what if your house burns down. You'd better make another set of your backup drives so that you can have one array live and one array off-site.

If my house burns down, I am going to have bigger things to worry about than 100 gigs of lost porn, warez and games.

Could file storage services use this?...

[lux55]

Since this is nothing more than an API to access data, I wonder if this couldn't be used as the backend storage for existing file storage services, instead of paying for servers and bandwidth yourself...

This limits costs to storage actually used (at $0.15/GB which is a very fair rate), and bandwidth actually used. The cost that could add up is the bandwidth, where you'd obviously have to direct users to the amazon URL directly to avoid using bandwidth to get the file then to pass it on too.

Plus, at $0.20/GB of bandwidth, upload/download could get expensive still, with no cap on that cost. For example, 2,000 GB of bandwidth, which is bundled with most low-end dedicated servers nowadays (ie. even the sub-$99/mo. machines), this would cost you $400 from Amazon. That's pretty steep, and may be the limiting factor making it unfeasible for this idea. Interesting nonetheless.

Re:Could file storage services use this?...

[Anonymous Coward]

It doesn't really make sense to compare this service to the cost of a hard drive or really any local storage media. (For the next several years at least,) I'm probably not going to use anything other than a hard drive to load my applications off of, local area network storage for my documents (I'm not going to wait for my large documents to load from the internet), and burnable/removable media for multimedia that I want easy access to but don't want eating up HD space.

If you're looking for a back-up service then the most obvious considerations are the potential ease, the fact that the data is off-site, and that it's probably redudant enough that you don't have to worry about it disappearing. The question is how much are you willing to pay for those?

Even if you want to store your documents on there so you can access them easily from anywhere, the question is still how much are you willing to pay for that convenience?

What's new about this idea?

[Anonymous Coward]

Internet hard drives were tried years ago during the dot-com bubble and failed miserably. What's different about Amazon's service? If I can buy a whole new hard drive for about 30 cents a gig, why would I want the hassle of uploading and downloading from a remote site?

Not to mention the time it would take to upload a few gigabytes -- cable modems are good at downloading, but they are NOT good at uploading.

Re:What's new about this idea?

[Tango42]

Upload speeds years ago were far lower than they are now. The slowest upload speed you get on British ADSL is 256kB/s which works out at just over 9 hours per gig. That's not great, but anyone wanting the upload gigabytes of data on a regular basis would probably have a faster connection (SDSL with 1 meg up isn't too expensive for the kind of business likely to have that much data - and that's just on a phone line. Special lines can get much higher). Leaving your photo collection to upload over night (maybe over a few nights) isn't that much trouble.

Re:What's new about this idea?

[reverend0]

The availability of broadband is much greater now than back during the bubble days for one thing. This model is much cheaper as well. This is not geared toward end consumers but to developers. You get redundancy and multiple storage locations. Lots of positives. There will always be some drawbacks but for the 20 meg of vital data that I have, this could be seen as super nice. Now I can get a dev account and distribute this to my family and boom we are all in much better shape.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Linux Filesystem Support ?

[Anonymous Coward]

How long before we see a network filesystem driver for Linux that supports this service ?

A gigabyte is a lot of space - at $0.15 per gig per month it's a potentially useful backup service for many home users and small businesses.

Of course you'd probably want to overlay your own encryption technology on theirs - but it's got real possibilities.

Plus $0.20 per Gb transferred!!!

[Anonymous Coward]

While $0.15/Gb/month is reasonable, the poster fails to mention Amazon will also charge $0.20/Gb on transfer. So while you will pay $15/month for your 100 Gb pr0n collection, you will also pay $20 to upload it, and a further $20 to download the whole lot to your cube-buddy's computer.

From TFA: "Apart from the storage fee, you pay $0.20 per gigabyte transferred, but there are no minimum fees and no setup costs, so you pay as you go."

Still, not bad - but the economics for the home user are a little less ideal than first reported.

Re:Plus $0.20 per Gb transferred!!!

[krunk4ever]

but the major upload/download are 1 time costs. you're not going to be uploading 100gb everyday or month. You'll probably upload 100gb the 1st time you set up your backup account, and then every night, you just update the files that changed or new files that got added, which might amount to a couple hundred megs which is a few pennies.

And you'd never have to download until you really need it, and at that time, cost is hardly ever a factor.

Re:Plus $0.20 per Gb transferred!!!

[OpticalPaul]

When Amazon lets me pay for the storage, and have other registered users pay the bandwidth charges (plus my profit) to access my content, then they'll have an interesting business. Unless Google beats them to it.

As it is, online remote storage with ongoing upload, download, and storage fees hardly seems interesting.

Re:Plus $0.20 per Gb transferred!!!

[muyuubyou]

Not that bad for 100 gigs of content, but you would probably just lend him an external HD for some days and have it back.

I prefer that than some kind of restriction on the number of people I can share the data with. Anyway I agree on the importance of that figure.

Re:Plus $0.20 per Gb transferred!!!

[spagiola]

From TFA: "Apart from the storage fee, you pay $0.20 per gigabyte transferred, but there are no minimum fees and no setup costs, so you pay as you go."

Still, not bad - but the economics for the home user are a little less ideal than first reported.

I think it seems attractive as a way to have a secure, off-site backup. ie you pay the upload fee (once), pay the storage fee, and (ideally) never pay the download fee because your primary on-site backup hasn't failed.

Re:Plus $0.20 per Gb transferred!!!

[Anonymous Coward]

You only have a 100 Gb of pr0n?

Re:Plus $0.20 per Gb transferred!!!

[Anonymous Coward]

Holy shit! That is a great business model. You can set up a subscription service to access your content, and maybe you would want to put up a page or something that displays what content is available...

They have this, dipshit, it is called "hosting" and "paypal" or a million other payment systems. Get a CMS, get a PayPal account and lease a host. Amazon is just being retarded.

Amazon's media service system has such horrible flaws in it, anyway... it'll be interesting to see how pissed off people will get.

- Ex-Yellow-Badge, tried to recruit me, but I escape the Church. Don't drink the koolaid.

Re:Plus $0.20 per Gb transferred!!!

[epanastasi]

That's for you to build, the whole point of a web service is to provide the building blocks for developers to take advantage of. You can build your own subscription download service that charges users to download from 'you' while in the back-end you fetch content from amazon's servers. What amazon is providing isn't a ftp server to host your files, it's a reliable, no-hassle back-end solution to whatever you want to build ontop of it.

Re:Plus $0.20 per Gb transferred!!!

[jp10558]

This is a worse ripoff that I thought. Again I will pimp (no I don't have stock) www.streamload.com. How /. has missed this service (that's been around since 2000) amazes me.

Look, you basically only pay for downloads. You get 25GB free. If you are a paying customer, there's no limit on data stored, though I do think that's going to go to 250GB eventually. Granted, the DL cost isn't as good as Amazons, but unless you are downloading a lot, the free uploading and storage will make it cheaper. Especially as for many people, 25GB is enough offsite storage, and important files they'd store are under the 10MB free filesize limit.

Statistics can be your friend

[the ed menace]

In general you might think that the cost per gig you pay and the cost per gig that Amazon pays is similar. And you would be right. But to obtain high reliability you will pay more.

Amazon, or anybody who tries to build a large distributed storage service, can spread out the probability of disk errors over a larger set of users than you are able to do. The marginal cost to replace a disk that has failed, on a per user basis, is therefore lower for Amazon.

Moreover, the overhead to manage many disks does not increase linearly to the number of disks. Put another way, their per user cost to manage the disks is lower than you.

The cost equation is less about purchasing the storage than maintaining it through the inevitable failures over time. This makes the gigabyte-based usage cost very fair, since it is proportional to the rate of error. The access cost manages their bandwidth expense.

What I would like from a service like this is a pricing guarantee -- if they maintain the same pricing two years from now, it will be a ripoff given the diminishing cost of storage and bandwidth. It would be nice to have it pegged to some kind of disk/bandwidth industry index.

Re:Statistics can be your friend

[Anonymous Coward]

What I would like from a service like this is a pricing guarantee -- if they maintain the same pricing two years from now, it will be a ripoff given the diminishing cost of storage and bandwidth. It would be nice to have it pegged to some kind of disk/bandwidth industry index.

Well, this is not as straight-forward as it sounds, nor necessarily reasonable. As you pointed out, cost to Amazon depends based on various factors; plus, since they are not co-ops, their goal is to be profitable. As such, what they could (and perhaps should?) guarantee would be either flat price, or such: if no one uses their service, costs are relatively higher, thus losing money; if more people use, their earn money. But having it only work based on industry norms would not have much incentive to Amazon.

At a more philosophical level, it also comes to this: does the value of service they provide change over time? Is it less valuable 1 year from now on? Perhaps, as data size tends to grow over time? I mean, otherwise, how much it costs Amazon to do it is largely irrelevant: what matters is relative costs of them providing the service, vs. someone else providing it (including you using your work time to do it).

So I would assume that this is a better case for just the good old market-based price: if the service became "too profitable", competitors would step in to offer better value (Google, MS, IBM, Sun).

Terms Of Service

[Kristoph]

8) If your Application is determined (for any reason or no reason at all, in our sole discretion) to be unsuitable for Amazon Web Services, we may suspend your access to Amazon Web Services or terminate this Agreement at any time, without notice.

I am not sure I see the point of using a storage service that has the right to unilaterally terminate my agreement and thus, presumably, destroy anything I have stored.

]{

Re:Terms Of Service

[morgus morphus]

Actually, a little further in the agreement they specify that they are only allowed to delete your data 30 days after terminating the agreement with you. So they certainly intend to give you safety against that (the details of how you get your data back after the agreement was terminated I don't know, but at least it's nice to see that they have recognized the concern).

Bandwidth

[grahamsz]

The initial bandwidth required to get a large quantity of data online is quite high, but the incremental cost is probably not.

Largely i'd want to back up my documents and my digital photos, and while i probably have about 15 gigs right now, i dont produce more than a couple of hundred megabytes a week. For $3 or $4/month i'd take that over pissing around with my own backups (or probably in addition to).

Can't wait to see...

[DieByWire]

Users with data like yours also had the following data...

Trust and the State

[Toby The Economist]

I find it disturbing that I do not trust the State enough to place my data with a third party provider for fear of my privacy potentially being violated.

Of course, my data is unimportant and the State has no interest in me; but *as such* it should be the case that my data isn't even *potentially* accessable to the State - and yet I rather suspect that it is.

As such, I am actually now being suppressed by the State; the State behaves in such a way that I, to preserve my privacy, have to protect myself.

The State is way, way too big for its own good; it's destroying now the freedoms it was created to protect.

Re:Trust and the State

[jcr]

I find it disturbing that I do not trust the State enough to place my data with a third party provider for fear of my privacy potentially being violated.

Why trust the state, or Amazon? Just encrypt anything you put on their servers. Trust the math.

The State is way, way too big for its own good;

For it's own good, or for our own good?

-jcr

Re:Trust and the State

[TallMatthew]

I find it disturbing that I do not trust the State enough to place my data with a third party provider for fear of my privacy potentially being violated.

This is a result of the current political climate.

File hosting services have been around forever, e.g. Yahoo Briefcase. It's when stories are leaked about domestic spying, governments pressing service providers for private information, law-abiding people getting bullied for being Muslim or critical of the current administration, that's when things you never thought were problematic become questionable.

Which, of course, is the point. Intimidation. Just keep your mouth shut and watch your step. How anyone could think what's going on is somehow what America is supposed to be just stuns me.

Re:Trust and the State

[Anonymous Coward]

For it's own good

"its". Jesus Christ, the OP had it right. How dense do you have to be to mispunctuate while quoting something right in front of you?

Re:Trust and the State

[ryran]

Good lord--what state do you live in?

I Sure am glad I live in North Carolina.

Link to the actual site:

[vuzman]

http://aws.amazon.com/s3

forget it

[penguin-collective]

My minimal requirements would be Webdav, sftp, and rsync-ssh; SOAP and REST I don't care about.

Oh, and also it should come from a company that isn't running a vast data mining operation.

Re:forget it

[gbjbaanb]

Apparently all the objects you create in their 'buckets' are encrypted and secure from everyone. So, fine with that.

I would like to see them implement rsync to get data to them, but as its primarily a data-storage service, and not a backup-service (ie its for your web app to hold and access data, not to dump nightly backups on), I doubt we'll see rsync ever, especially as rsync does require CPU time which I bet they have little of in comparison to the vast amounts of storage space.

Google are apparently working on a simialr storage system, so we'll have to see what they come up with. If you want backups.. bqinternet are very popular, and support rsync, and is roughly the same price as Amazon once you start storing a certain amount.

Re:forget it

[penguin-collective]

I would like to see them implement rsync to get data to them, but as its primarily a data-storage service, and not a backup-service (ie its for your web app to hold and access data,

rsync is a general protocol for transfering data between different machines; it's not just useful for backup.

Furthermore, many UNIX applications store data in directories and use file system primitives for moving them around; rsync is a good choice for that.

Re:forget it

[alanlewis0]

You are missing the point. This is a new web service targeted at developers building apps that need online storage. It is not targeted at geeks who want to upload their personal porn collection via FTP or rsynch. For web service developers, REST and SOAP are both good options. You should be looking at something like box.net (no vast data mining operation there), or waiting for G Drive (although G are the data-mining kings, so they are probably off your short list).

Re:forget it

[penguin-collective]

You are missing the point. [...] For web service developers, REST and SOAP are both good options

No, you're missing the point. Just because webmonkeys don't know any better doesn't make REST and SOAP good protocols for managing and moving data around. Storage is about efficiency and standardization of data storage operations, and WebDAV and rsync provide that; REST and SOAP don't.

No thanks...

[Nonillion]

I'll just keep MY data on my OWN machine where the government cannot get their grubby little hands on it.

Re:No thanks...

[chabotc]

Backdoors, keys to which the goverment has access, internet traffic sniffing, wiretapping phone calls and always having the backup option of sending some guys to knock on your door.. Never think the goverment can't get their hands on your data, chances are, all your data is already belonging to them :-)

Re:No thanks...

[jcr]

I'll just keep MY data on my OWN machine where the government cannot get their grubby little hands on it.

Do you have a means to destroy your drives if the government shows up with a warrant? If not, don't kid yourself: they'll get their grubby hands on your data anytime they choose to do so.

-jcr

Re:No thanks...

[Anonymous Coward]

Of course, thanks to PATRIOT, they don't have to show up with the warrant when you're there. They can come when you're gone, look through your stuff, and leave without you ever knowing. And, if you do find out they've been there, it's illegal for you to tell anyone. Including your lawyer. Welcome to 1984!

Hard drive-based backup systems

[this great guy]

No. The pb with tapes is that their data throughput is slow, they are not random access devices, not as reliable as what you would think, more expensive than hard drives, and currently losing popularity. I interviewed about 2 year ago at a petroleum company which has Petabytes of data to backup weekly and they were precisely migrating to hard drive-based backup systems for these very reasons. The higher rate of failure of hard drives can easily be compensated by making 2 or 3 copies of your backups. The funny thing is that hard drives are so cheap that such a solution would even be cheaper than a tap-based system.

Plus the advantage of hard drive-based backup systems, or online storage as the industry call it, is that your data is immediately available: no need to manually insert a tape in a reader, etc. Hard disks contructors have even created a new line of products to accomodate the nead of backup systems, google for "nearline storage".

I think I have it!

[TechnoGuyRob]

1. Use $150,000,000 to request an exabyte of information for a month.
2. Proceed to write internet worm that does a distributed upload of random garbage.
3. Amazon is unable to satisfy your requests since no one has ever produced an exabyte.
4. Demand Amazon pay you back damages with a multiple of your original investment.
5. Profit!

I rub my hands together in evil dictatorial glee. Mwahahahaha.

Re:I think I have it!

[Anonymous Coward]

1. Steal your idea and patent it
2. Sue you for patent infringement
3. Profit!

I wish you the best of luck on your endeavour!

Re:I think I have it!

[bk4u]

wow, you must have a lot of pr0n you want to store

Backup Buddies?

[cgenman]

It seems like this new service would be best for offsite backup of prescious data.

However, it isn't all that cost-effective. A local disk is very cheap comparatively, but (as a friend of mine found out) if someone steals your computer, they steal your backup too.

Are there any services out there which connects people with reasonable connections over long distances to back-up eachothers data? I'd be willing to get a new 80GB drive and make it available via a private FTP server if someone else would do the same for me.

Or are there cheaper offsite solutions than Amazon's?

Re:Backup Buddies?

[kfg]

. . . if someone steals your computer, they steal your backup too.

If you're silly enough to keep your solitary backup in your frickin' computer.

Go tell your friend that God has invented external drives. Then tell him that you'll keep his if he'll keep yours.

KFG

Re:Backup Buddies?

[dotc]

In answer to the question: Is there a "FTP-style backup" service?

Foldershare

It works on Mac and Windows. My understanding is that it sets up an automatic encrypted peer-to-peer mirror of folders between machines you designate. So for me, I have it mirror folders on my personal laptop, my work desktop, and my brother's desktop (which is the automated-internet-equivalent of what someone said in an earlier post, "mail a DVD-R to my mom"). If anything crashes, we've got two more copies of it.

Only drawback, doesn't work for Linux yet; likely never will work since they were recently acquired by Microsoft. But there's probably a way to Samba-serve a work-around for that...

Re:Storage? Oh wow!

[lbft]

Today's your lucky day.

(Probably NSFW)

Storage solution, pah!

[Kaptain_Korolev]

I was hoping for a bookshelf. I wanted a somewhere to store my books and all I got was this lousy online file store.

I think it was Linus that said...

[core plexus]

"Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" (1996)

Quickplacer, the fastest robot in the world

Re:I think it was Linus that said...

[Anonymous Coward]

next week on slashdot:

"linus torvalds home videos found on ftp site"

BitTorrent

[elrond1999]

Built to be flexible so that protocol or functional layers can easily be added. Default download protocol is HTTP. A BitTorrent (TM) protocol interface is provided to lower costs for high-scale distribution. Additional interfaces will be added in the future.

Amazon supports BitTorrent for the storage. Does that mean they run the tracker? Interesting way to save on transfer fees that :)

Re:BitTorrent

[post.scriptum]

Which means that your files will be on more than 1 server?

Very clever.... but?

[Leolo]

I read the documentation this afternoon. Very clever use of REST, SOAP and BitTorrent. They provide client libraries for many languages.

I can see why Amazon is providing this; to make money off their excess bandwidth and storage space.

I can't really see why a customer would want to use it though. Why not just use a real web host? Amazon S3 has is no minimum monthly fee, has redundancy built into it, guaranteed availability.

Compared with Dreamhost (say) which has a bundle for almost 10 USD/mo. That deal has 20GB + 1TB transfers. For the same amout on Amazon S3 you only 5 GB + 64GB transfers, and doesn't have FTP nor SSH access, nor your own domain, etc etc.

Maybe we should think of it as an inexpensive web cache, like akamai.

I suspect that even Amazon doesn't know what this will be useful for. The developed it for their own use, then polished up for resell. Now they wait for the applications to appear.

Re:Very clever.... but?

[Anonymous Coward]

You can reduce that cost to just under $30 for the first year by using the promo code 6HLA

Re:Very clever.... but?

[tgd]

I've got a number of accounts with Dreamhost, and quite a large amount of data stored there. Probably half of it is, in fact, encrypted backups of data I have at home, since the accounts have a lot more storage and transfer than I ever come close to using.

However, I have absolutely no idea what their fault tolerance story is there. They probably run them, but at best I'm sure its just RAID. I can't imagine they actually provide real backups for their hosted sites (although I may be wrong... I ran a hosting company in my distant past, and even with the drop in storage and hardware prices lately, I can't see the economics in that...)

Fact of the matter is, there is no good reliable way to ensure data is protected over the long term right now. RAID of course doesn't do it, but it helps with availability. I have a terabyte of RAID at home just because I want to reduce those risks. Offline disk-to-disk backups are an option. Half a terabyte of my RAID storage gets mirrored nightly onto a pair of external 250 gig drives that are powered down when not syncing. Thats not half bad either, at least I'm safe from a software problem that corrupts data as long as I catch it within 24 hours. Critical data is encrypted and uploaded to my dreamhost account once a week. I keep meaning to convince my parents to put a storage array at their house and I can do rsync over ssh between the two so we both can act as an off-site backup for the other.

But the fact of the matter is, I have gigabytes of data in the form of family pictures and videos I have absolutely no good way of ensuring are still available to my kids or grandkids (if I ever have them) forty years from now. DVDs won't do it, I've had lots of those fail already. Half my CDs I've burned in the last ten years have failed. Tapes aren't an option, they're not keeping pace with harddrive storage growth.

I can engineer pretty robust solutions to this for myself, but joe six pack sure can't. Amazon or Google offering some ability to really reliably store personal data long term is a big deal. Its a solution that is critically needed, and most people don't even realize they need it.

However, I think Amazon is nuts charging for it. Google will just come in and do the same for free sooner or later.

Re:Very clever.... but?

[Anonymous Coward]

I can't really see why a customer would want to use it though. Why not just use a real web host?

I think they might want to make this available under www.amazon.com/users/username so that people can upload their own javascript / AJAX stuff and be able to use more of Amazon's services.

Course I'm just talking out of my ass, but it would be an interesting way of getting around any cross scripting security violations.

It would make a nice way for Amazon to be the entire backend inventory system for any vendor.

Re:Very clever.... but?

[Iaughter]

I can't really see why a customer would want to use it though. Why not just use a real web host? Amazon S3 has is no minimum monthly fee, has redundancy built into it, guaranteed availability.

Compared with Dreamhost (say) which has a bundle for almost 10 USD/mo. That deal has 20GB + 1TB transfers. For the same amout on Amazon S3 you only 5 GB + 64GB transfers, and doesn't have FTP nor SSH access, nor your own domain, etc etc.

The point of Amazon's service is to complement a hosted web app. Say I want to start a Flickr competitor for movies (like video.google.com), I sign up for Dreamhost's $8/mo (parent is wrong at $10/mo) deal. My website, my (mysql) database, my php scripts and cron jobs are on Dreamhost, but oh crap, I've suddenly run out of disk space.

For $32 - 60 G, $64 - 90 G, then what?

Dreamhost is unusual in it's range of options, look at yahoo's hosting plan's: $9-5G, $20-10G, $40-20G. Yahoo's "Merchant" plans offer no more than 20G either, even their $300/mo plan.

There's also a real benefit in paying only for what you're using.

This begs the question as to why yahoo doesn't have a more flexible disk storage pricing plan for hosting, but I think that most web hosting companies aren't in the massive data storage business. Dreamhost and Serverbeach, etc spend their money on big pipes, not administering SANS.

I think that there is a real need for this type of service and the choice of interface (SOAP and REST) demonstrates the market. Not personal backup, not even backup at all, rather immediately accessible storage of user data to hosted web applications.

I wonder how Google Drive will compare to this

[FleaPlus]

It looks like Amazon managed to get their storage product out before the rumored Google Drive (TechCrunch article, Slashdot article). I wonder how Amazon's product will compare to Google's, whenever Google's is released. I'm particularly interested in seeing how Amazon and Google will end up competing with each other in terms of price and transfer speeds.

Giant Google vs Amazing Amazon

[ikejam]

With Googles's library initiative, the leaks on GDrive, the A9 search, Google vs Amazon is starting to look like an epic fight ( googol vs amazon - does sound like a corny clash of the titans). Can't be much longer before MS and Amazon partners unless Amazon is too (rightly) suspicious of MS's long term plans..

Good for us though, Google and Amazon seem to take different approaches to most things, and ultimately that will provide variety, and good innovative competition (unlike MS).

As an aside, the fact that theres additional charges per gb transferred as mentioned earlier in the discussion will have a major impact, on its business, impact and utility. Ofcourse it would eliminate abuse, if you can call maximally utilising a paid service abuse. Like the latest netflix saga.

This reminds me..

[atarione]

Where is the best place to get write-only media?

Re:This reminds me..

[tepples]

Where is the best place to get write-only media?

Buy the cheapest CD-R media you can find. After a year, you'll know why I suggested it as "write-only media".

Comment removed

[account_deleted]

Comment removed based on user account deletion

Not as affordable as Google.

[babbling]

Poor Amazon. What awful timing. Google are about to offer the same thing for free, yet Amazon think people will pay $0.15 per month ($1.80 a year) per gigabyte stored?

For businesses, a tape backup is better and cheaper than S3.
For home users, a DVD+/-R is better and cheaper than S3.

Google's won't be better than the tape/DVD options, but at least it will be cheaper. (free)

Of course, this is all assuming Google are going to offer their storage service for free, but I think that's a safe bet.

Re:Not as affordable as Google.

[LordLucless]

For businesses, a tape backup is better and cheaper than S3.
For home users, a DVD+/-R is better and cheaper than S3. >

Except that tape/DVD backup doesn't, in and of itself, guarantee geographic redundancy. You need to then take those tapes or DVDs and store them in a secure location offsite. That may or may not be expensive, depending on how paranoid your are, or how sensitive your files are. But even if you just take them home and put them under the mattress, it's impossible to automate.

Whereas with an offering like S3, you could simply setup a cron job to zip, crypt, and send your critical files each night. This is obviously only good when only a reasonably small amount of data needs to be transferred - even good connections are going to have trouble sending 10 gig of data each night. But for a small number of critical files (ie: Not your MP3 collection, or stash of DVD rips), this sounds like a decent solution. If you really do need to critically backup 10GB+ each week or so, then yeah, maybe media-based backup w/ offsite secure storage is the best solution. But there's definately a niche for stuff like this.

Re:Not as affordable as Google.

[1u3hr]

Whereas with an offering like S3, you could simply setup a cron job to zip, crypt, and send your critical files each night. This is obviously only good when only a reasonably small amount of data needs to be transferred - even good connections are going to have trouble sending 10 gig of data each night.

Also, TFA goes on to say: "Apart from the storage fee, you pay $0.20 per gigabyte transferred". That would add up quickly to pay for redundant physical storage of your own. It also mentions the various ways you can access the data. It's not meant to be simple backup; more a way to make your data more accessible.

If you want to backup safely on the cheap, get a buddy in a different building, (or city or country if you're really paranoid), and store each others' data (encrypted).

Tape v disc comparison

[Dibblah]

Not with any of the recent tape formats. They're all "serpentine" - That is, a very narrow track (up to 1/512 of the width of the tape) goes from the start to the end of the tape. The head then moves down a fraction, and writes the next track "backwards".

This means that the seek time is reduced by up to 512x. Of course, this isn't free - Tape wear is increased since there are many, many more passes over the tape.

Re:Tape v disc comparison

[Anonymous Coward]

Tape's fine if you don't really care about the data. It will degrade over time.

Come on, people, it's 2006. Let tape die the way it wants to!

Fine, if

[nagora]

If you don't mind paying someone to abuse the patent system. Almost everyone on /. whines about patents yet very few seem to want to do anything about it by, for example, not supporting bastards like Amazon who are working every day to break that patent system and make money out of other people's ideas because of it.

Two words: "Fuck them".

TWW

Re:Fine, if

[Raenex]

I like Amazon's service and prices, at least for books. It sucks that they abuse the patent system, but trying to live a principled life where you only purchase from "good" companies is just too damn hard. Even Google, which I really respected for a long time, has fallen in my eyes.

Do you make every purchase with such principles in mind? Clothes? Food? Ever bought an mp3 player? A DVD with copyright protection? A computer with Microsoft Windows installed? Is every single product you buy from a company without bad patents?

Re:Fine, if

[Anonymous Coward]

Has Google fallen in your eyes ? Tough luck ! Going forward, close your eyes while looking up, duh !

Re:Fine, if

[Anonymous Coward]

not supporting bastards like Amazon who are working every day to break that patent system and make money out of other people's ideas because of it.

Proof? Source? Or are you still whining about that 5 year old one click purchase patent?

Re:Ask Slashdot

[68K]

He was being funny, retard.

will they offer me products related my backup?

[cyfer2000]

Will they look into my stuff and offer me something related? Like if I upload a lot of pictures taken with a Digital Rebel camera, will they send me advertisement about camera accesories?

What they don't say...

[David Horn]

What they don't mention in the news item (and you have to trawl through 3 sites just to find the bloody thing), is that while Amazon charge $0.15 per GB used, they also charge $0.20 per GB transferred. So you actually get stung twice, or even four times if you backup weekly.

Re:What they don't say...

[Xenna]

For my purposes incremental nigthly backups (something like encrypted rsyncs) would be just perfect.

I think this is a great idea, but I think I'll just wait for the Google offering before I start on writing my incremental encrypted backup Perl script.

X.

Redundancylicious

[Anonymous Coward]

But then what if your house burns down.

Houses are cheap, why not just keep two of them!

But what if a meteor strikes Earth? Why not just send the backups in a probe to Mars!

But what if the Sun explodes? Why not just build a Quantum computer and store the backup in an another dimension!

But what if the m... *AIEEEEEEEEEEEEEEEE*

Re:Redundancylicious

[grahamsz]

The notion is that insurance can replace your house, it can replace your hard drive but it won't do anything about those pictures of your nephew from last xmas.

Re:Redundancylicious

[Ahnteis]

That's why I buy cheap webhosting and back them up there. (That and I want family to see them.)

>_>
_

Re:Redundancylicious

[dubiousmike]

you are right on about pictures. I am a pulic insurance adjuster (an adjuster that fights insurance companies to maximize claims for insureds) and insurance companies only pay 1 dollar per picture. Those of you who still do film cameras, keep your film in a safety deposit box. You will thank me later.

http://www.nongnu.org/duplicity/

[Anonymous Coward]

Duplicity is open source, gpg encrypts data before sending it, and uses rsync style algorithms to ensure it only transmits changed deltas.

Backup

[scoutts]

Running .15 per Gig plus .20 per Gig transfer is not very economical for power users, nor very practical (upload speeds etc) This is pretty well covered. However..for the typical end user that can barely patch their machine, , may or may not understand to renew their virus definitions adn only use their computer to store digital pictures of their grandkids this is a good service. I often wonder how many grandparents lose their photo collection everytime a hard disk crashes (3-5 years at most) because i doubt more than 1% of home users run a raid array. Yep, better choices, flickr etc but which allows them to be hosted and send links to friends but nice to have a generic service. Amazon "catering to the unsophisticated". got a ring to it :)

Let's compare

[EmagGeek]

Let's take a 300GB System that needs to be backed up:

900GB at $0.20/gb = $180.00 transfer fee
Monthly at $0.15/gb = $135.00/month recurring charge
Weekly incremental of 30GB = $6.00/wk = $24/mo recurring charge

So $180.00 + $159/month = $2088 just for the first year, plus whatever you have to pay your ISP for abusive bandwidth charges.

Let's look at it from another perspective:

4 WD3200SD 320GB Raid-Edition SATA Drives: About $600
1 4-Port SATA Raid Card: About $250
Expected Lifetime: 5 years

So, buying a whole other raid-5 array to mirror your 900GB of stuff costs nearly $10K to store for 5 years on Amazon versus $850 to store locally. Hell, even if you were paranoid and replaced one of the hard disks every 3 months, you'd still be at less than half the cost.

I won't even get into which is more secure. If it's not on your site or some place you have physical control over, it is not secure.

Re:Let's compare

[Shivetya]

Mom & Dad

5GB at $0.20gb = $1.00 transfer fee
Monthly at $0.15gb = $.75/month recurring charge

$0.75 for a year is $9.00 dollars.

Throw some pictures up there, taxes, and other essentials using a third party program that "helps" you gather what really should be stored in case of emergency (can you say this program might be a good idea for someone in the open-source community?)

Far better than what they have now and its safe from fire. Throw a little encryption through that 3rd party app accessing the Amazon storage and it would be secure.

The difference here is that I used numbers I expect of data that should be backed rather than just dumping stuff on the drives because its there. The amount of stuff people just dump on drives for backup is amazing and wasteful.

Re:Let's compare

[EmagGeek]

Oh come on, whose MP3 and Netflix Movie collection is only 5GB??? ;-)

Re:Let's compare

[Anonymous Coward]

Mom & Dad

Some of us were born in petri dishes, you insensitive clod!

Re: Keeping the gov't's nose out of your business

[some guy I know]

Do you have a means to destroy your drives if the government shows up with a warrant? If not, don't kid yourself: they'll get their grubby hands on your data anytime they choose to do so.

Two words: encryption, steganography.

The MPAA is behind this!

[Anonymous Coward]

Remarkable how the "Etherdrive" phenominon seems so closely linked with the "IP-VOD" Rumor-space. Here is the evidence:

Apple: .Mac capacity increases (big-uns) in Dec, iMovies in April
Google: Google Video in (Nov? Jan? Don't recall), GDrive any day now
Amazon: Movies-R-Us ambitions (like, last week), S3 Today

Conclusion, the MPAA will allow these guys to sell movie downloads as long as they keep part of the film on their own servers! How better to ensure you don't put those same vids they send you back up on Torrent than simply not to give you the whole file. When you want to fire it up, you get half from your hard drive, half from your network drive.

It's Perfect for the MPAA! The studios get the control they want, "The People" get the convenience they desire, and the cost gets pushed to the third-party vendors (who, conveniently already own complicated things like server-farms). Complete control, none of the hassle!

Of course, no one will be able to put up a fight when first the vendors, then the studios start pushing ads into the front of "your" movies to "cover the increasing costs of doing business". Remember when cable television and satellite radio had no ads. In a few years it will be just like going to the theater! Groan.

INCLUDING WITHOUT LIMITATION LOST DATA

[way2trivial]

there's the relevant part of the contract...

who do you trust? yerself, or amazon?

fees and limits

[MORTAR_COMBAT!]

so, $0.15 per GB/month storage, $0.20 per GB transferred.

questions: how do i put a cap on my storage (and more importantly transfer) so a runaway service doesn't screw me?

Burned DVDs are UNRELIABLE

[Anonymous Coward]

I have had a ton of cases personally where I have burned a DVD or a CD and a few MONTHS (not even years) later, I try to regain my backups and I get CRC errors galore. (with different computers, different burners and readers)

DO NOT trust CD's and DVD's to hold your data long term, use them only for short-term backup until you can get a more reliable backup device, like a RAID array.

Re:Burned DVDs are UNRELIABLE

[ShavenYak]

You can always use redundancy to your advantage. When burning your files to DVD, copy about 3GB into a folder, then make PAR2 files to fill it out to 4GB. Oh, and keep your individual files relatively small - any files over 100MB or so should probably be split into smaller chunks. That way if a small error in the TOC makes an entire file inaccessible, you haven't lost too much data for PAR2 to recover.

Privacy

[AviLazar]

With stuff like this going on, do you really want your data online? Sorry, I will pass and by myself two hard drives that will mirror each other. There is my storage medium - so if the gov't wants to inspect my files, they have to come into my home - not go to another company - where I may never get notified.

World's largest BitTorrent seed?

[dlaur]

If you follow this link, you'll notice that they are supporting bit torrent.

Consider.

"Win/Linux" != "Cross Platform"

[Kadin2048]

If by "cross platform," you mean "Windows XP/2000/2003 and Linux."

Call me back when it runs on OS X.

Re:"Win/Linux" != "Cross Platform"

[zhiwenchong]

Of course, Mac users like us can always upload AES-128 encrypted DMG files. That's what I do for backups -- I use disk utility to create an encrypted disk image of my ~ directory and replicate the DMG on various servers worldwide.

your time worthless?

[DuctTape]

couldn't I jut buy a new hard drive every year or burn hundreds of DVDs for far far less? not to mention they'd be secure from whatever prying eyes or security holes an online backup provides.

And then you get to go to your safe-deposit box once or twice a week to retrieve your hard drive, back up to it, and then drive back to your bank. You did want offsite storage, didn't you? How much is the gas and the inconvenience going to cost you? Encryption will take care of prying eyes.

DT

NOT a backup solution

[WPIDalamar]

From the terms of service...

2) You may make calls at any time that the Amazon Web Services are available, provided that you either: (i) do not exceed 1 call per second per IP address, or send files greater than 40K; or (ii) do not exceed the limits set forth in the Service Terms for a particular Service. If you build and release an Application, the stated limitations apply to each installed copy of the Application.

Re:NOT a backup solution

[CaptainValue]

But the relevant clause here is (ii). Reading further, it says:

2) The limitation of 1 call/per second/per IP address set forth in Section 1.A.2 above is not applicable to your use of Amazon S3. You may not, however, store "objects" (as described in the user documentation) that contain more than 5 Gigabytes of data, or own more than 100 "buckets" (as described in the user documentation) at any one time.

Re:NOT a backup solution

[fgb]

Read a little further. The TOS also states:

2) The limitation of 1 call/per second/per IP address set forth in Section 1.A.2 above is not applicable to your use of Amazon S3.

Re:NOT a backup solution

[fgb]

Oops. Left out the important part of the quote. It should have said:

2) The limitation of 1 call/per second/per IP address set forth in Section 1.A.2 above is not applicable to your use of Amazon S3. You may not, however, store "objects" (as described in the user documentation) that contain more than 5 Gigabytes of data, or own more than 100 "buckets" (as described in the user documentation) at any one time.

No Mac version

[TomatoMan]

There's no Mac version, and no plans for one either according to the current forums and FAQ. Windows + Linux != cross-platform; maybe you could call it "dual platform." Too bad, it sounds like useful software.

Re:No Mac version

[trifish]

> Windows + Linux != cross-platform

Judging from your equation, it would have to run on Atari TOS, CP/M and MS-DOS and all other available platforms to deserve the cross-platform status?

Re:No Mac version

[TomatoMan]

No, but maybe more than two. See the rest of my short comment. I don't think I'd call something I wrote "cross-platform" if it only worked on, say, Mac and Linux, or Mac and Windows.

I suppose I'd say that "cross-platform" means to me that it should run on the major current OSes, the ones that have (say) at least a few tens of millions of users. But this is pedantic and I'm not actually terribly interested in it - just bummed that this particular piece of OSS brands itself as "cross-platform" but doesn't run on Mac.

Re:No Mac version

[trifish]

> it should run on the major current OSes

Who defines what a "major" OS is? Where are any trustworthy verifiable numbers of real OS X users?

> just bummed that this particular piece of OSS brands itself as "cross-platform"

I didn't see the term "cross-platform" mentioned anywhere on their site. Anyway, their software actually is cross-platform, because it runs on more than one platform. The fact that it does not run on OS X and AmigaOS does not make it non-crossplatform.

Regular Joe Reading

[u16084]

If A Regular JOE is reading this article.... Does he Actually Understand This Stuff?

Decentralization
Asynchrony
Autonomy
Local responsibility
Controlled concurrency
Failure tolerant (Ok i know this one)
Controlled parallelism
Decompose into small well-understood building blocks
Symmetry
Simplicity

I'm sure REGULAR amazon customers will understand this. Taken From http://www.amazon.com/gp/browse.html/103-6578525-6 491034?node=16427261/

Re:Regular Joe Reading

[epanastasi]

This is a web service, it is meant for developers not regular joes... move along.

DIBS: pretty slick.

[Kadin2048]

I just looked at DIBS ... I'm very impressed.

It's a good concept -- everyone gives some space on their local drive, and in exchange gets to break up their files and store them across others' systems, in encrypted form.

It seems like another one of those projects where the most difficult part is going to be boot-strapping the userbase and community necessary to create the pool of always-available hardware resources so the thing will function. I haven't looked into it too much, maybe it's already there.

Re:DIBS: pretty slick.

[MORTAR_COMBAT!]

it would work great, until you are arrested for hosting child porn on your machine.

Re:DIBS: pretty slick.

[Kadin2048]

The data on your disk is encrypted, plus files are broken between multiple machines. This keeps your data secure when it's been stored on others' computers, and also gives you plausible deniability as to what you're holding. It's similar to the way FreeNet works.

DIBS uses GPG for encryption, and I'm not sure what cipher it uses, or whether you can choose one, but most of the commonly-used GPG settings ought to be good at least for a few years. It's not perfect forward secrecy or anything, since I suppose someone could retain your DIBS data and then decrypt it at some point when breaking 128-bit AES is child's play, but it's not trivial.

I think though that the problem you bring up as a social issue is quite significant though. How do you keep such a system from being flooded with loads of porn and MP3s that people just don't want to keep on their own machines? I don't know what sort of verification DIBS uses to make sure that you're hosting as much space as you're taking from the community, but it seems like there's a risk that risk that somebody will make a hacked client and try to "leech" space from other systems.

Google

[Supersonic1425]

Google will probably do this for free before long.

Only a Developer-level API for now...

[dpbsmith]

See this description. Amazon is just offering a raw service with an API for developers to use. Amazon does not provide a WebDAV or similar interface. So it's not ready for home users, just yet... which, at that price, is a pity.

So for now, don't dump your .Mac iDisk.

Undoubtedly we will see independent developers offering home and SOHO backup tools that use A3... and undoubtedly they'll mark up the price.

Another problem with tape...

[Sinister Stairs]

Another problem I've had with tape is needing "special" hardware to read it, and remembering what data format it was written in.

A friend of mine backed up an Apollo and AIX server, circa 1993. Six years later, I tried restoring the tape to no avail:

First, I had a difficult time finding a tape drive that could even physically accept the tape. (I believe it's 8mm.) Next, I didn't know what kind of tape it was. (It's possibly a DAT.) Finally, presuming I had the proper drive that could read the tape, I didn't know what format the data was written in. (tar? cpio? dd?)

I'm admittedly no expert when it comes to tape retrieval, but if it had been an old IDE or even SCSI HD, it could still be plugged into today's systems.

(I'd still like to restore that tape, if anyone has any suggestions how to do so... The tape purportedly contains the backups for the old Hero and Hero 2 MUDs, which is why our nostalgic interest in restoring a ~12 year old tape.)

Why this is great for backups...

[Mantrid]

People are moaning about how tape is cheaper or HDD is cheaper per GB than this. So they are planning to use that for there backups.

I definitely want an offsite backup for my family photos, this could be a good solution. Sure IN THEORY I could backup to USB Key, HDD, CD-R etc, and in fact I do have a one touch drive. But what happens if my house burns down (fairly unlikely), or worse get's broken into and my shiney looking HDD and PC gets stolen (possible)...bang photos gone.

So what about moving CD's or HDD's offsite or to a safe deposit box? Sure that's great, but here's the thing realistically it'll never happen. Or I might make a backup every six months, and of course a disaster event will take place at 5 months and 25 days or something. Those who are very organized and good at data security can definitely get away with cheaper alternatives, but a one click backup to internet? Something you don't have to worry about after? That's just awesome. Sure Amazon could burn down or go out of business, but in that case I will just have to revert to my own backups, and make new copies right away.

Not really backup..

[drspliff]

A lot of the posters here seem to think it'll be used as their main or secondary backup source (at 15c/GiB per month it's not bad), and whichever market Amazon try and target it'll always end up being used by the lower end..

Who will use it? Perhaps not your grandma.. but it'd be nice to have a conveniant place to put all these pictures I took with the camera my bought me, or backups of my web/art/academic/porn portfolio or my e-mail inbox because my the company I work for enforce small quotas!

I can't see it being used by SMBs or anything larger - they can afford HDD/Tape backups and the rental of a local self-storage locker.

Basicly the people who want this sort of storage (and more importantly know why they actually need it) will use it because it's cheaper, more reliable etc.

All Amazon have to do now is convince people who don't know why they need this sort of service to signup and store crap on it.

Why rent when you can own?

[Mister Mudge]

I can buy massive amounts of fast storage for under $1/GB. I can run it in a RAID and/or pull a mirror off for offsite storage - all for under $1/GB.

So why pay somebody about the same amount every 6 months for rent, when I can own it, especially when the breakeven timeframe is so short?

Re:Why rent when you can own?

[sugarmotor]

because it is "connected" to the Internet ?

"Cross Platform" =~ "Win/Linux"

[sasdrtx]

Cross-platform means something that can be served on one platform and accessed from a client on a different platform. Nobody said anything about universal-platform, or all-platform, or cross-platform-including-mac.

Re:"Cross Platform" =~ "Win/Linux"

[metamatic]

Actually, the original poster said "any platform". Go back and look.

If he'd meant "either platform" he should have said so.

Re:"Cross Platform" =~ "Win/Linux"

[sasdrtx]

You are right. As you may have surmised, I didn't read it the first time. Sorry.

I use rsync.net for this already - encrypted, etc.

[Anonymous Coward]

I have been using a service just like this at rsync.net. It is a little bit different, but the concept is the same - online backup, redundant, etc.

The difference (I think) is that I can't imagine amazon.com will have theirs open for plain old ssh and ftp, which is exactly how I use my account at rsync.net.

I scp to and from it, rsync to it from cron, and have it mounted as a local drive over sshfs (and I can't say enough about FUSE/sshfs - I should submit a slashdot story just for them).

I think the price is comparable - the $2/gig I pay seems like a lot more, but I note you have to pay 20 cents per gig per transfer with Amazon, whereas with rsync.net I get unlimited transfer.

EULA is interesting

[shawn_f]

Here is section 5 of their EULA:

"5. PROVIDING CONTENT TO AMAZON You may provide text, images, reviews and other informational content ("Content" about a product to Amazon. If you do so, you hereby grant to Amazon a perpetual, paid-up royalty-free, nonexclusive, worldwide, irrevocable right and license to: (i) use, reproduce, perform, display and distribute the Content in any manner; (ii) adapt, modify, re-format and create derivative works of the Content for any purpose;(iii) use and publish your name in the form of a credit in conjunction with the Content; and (iv) sublicense the foregoing rights to its affiliates or any third parties. Additionally, you represent and warrant that: (a) the Content is your original work or you obtained the Content in a lawful manner, and (b) Amazon's and its sublicensees' use of the Content as specified above will not violate any third party's rights, including but not limited to copyright rights. Amazon reserves the right, but is not obligated, to use and publish your name in the form of a credit in conjunction with the Content you supply to Amazon, and by supplying any Content, you grant Amazon the irrevocable right to do so."

Ouch... Now look at this part of the EULA that pertains to S3 service...

"B. Amazon S3 - Simple Storage Service 1) You may use Amazon S3 to store, retrieve, run, and/or operate software applications and/or data owned or lawfully obtained by you ("Your S3 Content"). You acknowledge that neither Amazon nor its affiliates are responsible in any manner, and you are solely responsible, for Your S3 Content. While we may track information regarding your usage of Amazon S3, we will not disclose, sell or license Your S3 Content in any way that is inconsistent with this Agreement, except as may be necessary to comply with subpoenas, court orders, or other legal requirements. 2) The limitation of 1 call/per second/per IP address set forth in Section 1.A.2 above is not applicable to your use of Amazon S3. You may not, however, store "objects" (as described in the user documentation) that contain more than 5 Gigabytes of data, or own more than 100 "buckets" (as described in the user documentation) at any one time."

So, can they use your information that is stored? I do not think it is very clear...

At the very least something to think about before jumping on board...

Re:EULA is interesting

[TheSkepticalOptimist]

Um no,

It clearly states that Content in this statement concerns information about Amazon's products and services, that is, if you store Content that originated from Amazon, they have the right to use that content.

This does not mean that your pictures, music, etc, etc, i.e. Personal Information is valid for Amazon to use.

The part about tracking your content or accessing your content through supeonas, etc is valid. Look, don't store pirated movies and porn on Amazon's web service. Use it as a backup for personal photos and music you legitimately own, and you won't have a problem. Amazon wants to ensure they are not being used to move pirated content over the net or used for illegal purposes, and they have the right to ensure you are not violating their trust by using them to easily distribute illegal content.

I think your trying to read something into this that doesn't exist. The language might be a little difficult to understand, but it is clear what they can and can't do with your personal content and what they are allowed to do about their own content that you may store with the service.

How does it add up?

[grahamsz]

Given that i'd really only backup content that i create, then it's bounded by how fast i can create it.

Surely when we kick off it'd be a little expensive, but on a month to month basis i doubt i'd upload much more than a gigabyte.

quit talking about backups, thx

[epanastasi]

Everyone is like, OMG backup/OMG too costly for backup/LOLOLOl!!!1one!!.

WTF, where's the backup my harddrive button??

Amazon didn't start runing an FTP server for you to store your crappy pictures and emo mp3s. They are providing a web service, the whole point of a web service is to provide the building blocks for developers to take advantage of.

OMG is costs so much.

You can build your own subscription download service that charges users to download from 'you' while in the back-end you fetch content from amazon's servers. What amazon is providing is a reliable, no-hassle back-end solution to whatever you want to build ontop of it. Or use BitTorrent, they support it too...

ZOMG backup.

Will everyone quit talking about backups. Start thinking about what else you could do with with this an the rest of Amazon's web services. plz? kk, thx

Re:quit talking about backups, thx

[Anonymous Coward]

There are not enough levels of irony you could possibly be using to make me not want to punch you in the throat for your atrociously worded post. I'm sure you and your buddies think !!!1one! is hilarious or maybe even so stupid it's funny or something more inane. Just stop it. Christ. Say what you have to say without all that goofy-style nonsense.

That said, kudos for being one of the only people to get it. I also want people to shut up about backups.

Re:forget it - www.rsync.net (also look at sshfs)

[Anonymous Coward]

See my post below - someone already does offer rsync access to your data on a service like this. (rsync.net)

I use rsync for incremental backups with my rsync.net account, I use scp to grab individual files from wherever I might be, and I mount it as a drive on my FreeBSD and Linux systems using sshfs.

sshfs rules. You really need to take a look at it.

Rant.

[JDALaRose]

Stories that link only to an article on another site that itself doesn't link to the actual thing that is being discussed suck donkey balls.

According to their Terms Of Service

[Muad'Dave]

From Amazon's Terms of Service:

2) You may make calls at any time that the Amazon Web Services are available, provided that you either:
(i) do not exceed 1 call per second per IP address, or send files greater than 40K;

or

(ii) do not exceed the limits set forth in the Service Terms for a particular Service. If you build and release an Application, the stated limitations apply to each installed copy of the Application.

I hope that 'or' means that if the Terms of Service of a particular webservice are less restrictive then those terms apply instead of part (i). If it's the lower limits that apply, this isn't very useful.

Backing up the encryption key?

[aspenbordr]

I think the single biggest problem with encrypting online backups is that it ties your backup right back to your local computer. If all the information on Amazon is encrypted with a key that is stored on your personal computer, you may as well have all the data on the personal computer in the first place, since your data goes *kaput* if you hard drive fails anyways.

This introduces the need for multiple storage sites for your key, which will presumably have the key in cleartext. All that Big Brother would need to do to get the data would be to figure out where else the key was stored, and use that to decrypt the data.

Any thoughts on this? Or is the best way to just use a long passphrase, store the key on the server, and hope that when you need to recover when your hard drive fails 2 years from now, you remember your long, hard-to-guess password...

Seems awkward to me...

Re:Backing up the encryption key?

[aminorex]

The standard approach is to store your crypto keys on a USB drive, usually in one big password-protected keychain.

Re:Backing up the encryption key?

[Xepherys2]

No, this makes perfect sense, actually. First of all, many keys like this are static keys, like a license file. It's easy enough to back up to a USB key, a CD, another system, a floppy disk. Secondly, if the key then needs a passphrase, well, just choose carefully, and hell, write it down. If you're using this at home, who are you trying to "hide" the key from? If you're a business, the IT department needs to be filled with trusted folk anyway. If you are a business, however, I recommend using a keychain app and encrypting your passphrase locally.

Re:SFTP, Rsync-ssh

[cdailing]

You should check out strongspace.com. They have a monthly plan thats very affordable and if you go through one of their sister companies textdrive.com theres a special right now where you can buy a lifetime account for one payment. Good deals.

Allmydata

[P!Alexander]

I've tried Allmydata in the past and had some success. They use a peer-to-peer system to back your files up. The free plan requires a 10:1 ratio (you share 10 megs, you get 1 meg). Not bad if you have a lot of free drive space sitting there doing nothing. I think it uses bittorrent as its transfer method but I can't remember exactly how it works and the web page is short on details.

I just got the most recent version (1.3) and haven't played with it much but the last one I had a lot of trouble with, sometimes files wouldn't upload, it was hard to tell if the program was actually working or if it had died, and it seemed like it couldn't remember which files I had chosen to persistently back up.

Still in beta though, but interesting if they can get the kinks worked out.

Comment removed

[account_deleted]

Comment removed based on user account deletion

New Here?

[Anonymous Coward]

Are you new to non-FOSS computing? Almost every EULA out there allows unilateral changes by the licensor.

Also encrypted sparse image

[Kadin2048]

This is true.

Actually on a Mac what you can do is make a free-floating encrypted Sparse image. It's the same way that the OS handles FileVault encrypted home folders. It's superior to just making an encrypted DMG, because it's readable and writable like a regular filesystem, plus it can expand and contract depending on what's stored in it.

It doesn't have the steganographic or deniability benefits of Truecrypt, but it's good encrypted storage. (Plus if you're ultra-paranoid you can put it inside your FileVault encrypted home folder, so that the data on disk is encrypted twice.) Plus I don't think you need to be an Administrator to do it, so it could be useful if you only have a user account on a system and don't trust the person with the master password.

The only "trick" is that Disk Copy will not make one, you need to do it from the Terminal with hdiutil.

% hdiutil create SecureSparse -size 5g -encryption -type SPARSE -fs HFS+ -volname SecureImage

Where "SecureImage" is the name of the file you want to create and 5GB is the maximum size (which is not necessarily the space it will take on disk).

There are a few caveats though. You can't share it with someone who doesn't have a Mac, hdiutil is not open source and there is not to my knowledge a Linux version, and I'm not sure what happens if you try to copy it to a FAT filesystem and back. I've copied one to a Linux fileserver (EXT2) and back and it seemed to survive okay, but I have always been told to use caution when moving sparse files around.

(I originally learned about this procedure from this page, so all credit to them.)

outsourced data persistence tied to a perl hash

[davidnicol]

rsync.net
DirDB::FTP

What else can you do

[0x0666]

Instead of keeping different copies of the same files in my desktop, laptop, etc. and going thought the pain of syncing them up. I would write a web application to store and access my data from anywhere. you can even develop a web-hosting service based on this... without the pain of maintaining your own servers/admins, etc. I can see start-ups using this as an alternative to investing a lot of resources when they do not know how the business will turn out.... So, it is not aimed at individuals really, but small-businesses

Other alternatives

[failedlogic]

I'd like to know if anyone has any other suggestions (never hurts to shop around).

I'd like a service which offers SFTP access (so I can run with a shell script or something on my Mac). I have a small amount of data (Max is about 200 MB for daily rotating backups). Just for peace of mind for backing up my university work.

G-Mail probably isn't a bad idea, just not sure on the gurantees for backup.

i don't care about plausible deniability

[MORTAR_COMBAT!]

my main concern was not simply being arrested, or found guilty.

it would be: "big surprise" -- actually helping to host child porn. actually helping to encourage the molestation of kids. it doesn't matter if you are never prosecuted.

My assumption, apologies.

[Kadin2048]

my main concern was not simply being arrested, or found guilty.

it would be: "big surprise" -- actually helping to host child porn. actually helping to encourage the molestation of kids. it doesn't matter if you are never prosecuted.

Ah, understood. I suppose in that case I don't have a solution for you, nor do I think there really is a solution. And I think the problem you are describing could be extended not just to DIBS, but to almost any file-storage technology, and perhaps to any communications technology. If I work for an ISP, am I indirectly facilitating the exploitation of children? I suppose I am, in a way, since there are undoubtedly people using their connections for that purpose.

Although I respect your feelings and see where you're coming from, I believe in this case that the good of such a technology outweighs the bad. I would say the same thing about FreeNet, or the Internet in general. I suppose that comes down to a matter of faith, since I'm not sure it's possible to prove that the amount of good gained from any form of anonymous/psuedonymous communication outweighs the evil that people will undoubtedly do with it (not to mention relative standards of evil, although I certainly agree with you on child porn, there are other areas that are more grey; e.g. "hate speech").

I do have to admit you caught me off-guard; the typical objections to FreeNet, et al, are less altruistic and more self-interested.

More importantly...

[neo]

You can't put your local disk on a network that S3 has. If you have an application you've just distributed it across the country. That data is going to load very quickly and is probably redundant on Amazon's network. You can't get that same performance from your local drive/network.

5Gb limit - curious

[Anonymous Coward]

It's interesting that it is a 5GB limit per object. Just above the capacity of a DVD.
Expecially in relation to the rumours regarding pending a potential movie download launch from Amazon.