Slashdot Mirror
Secure File Storage Over Non-Trusted FTP?
hmckee writes "Does any software exist that enables me to store/backup/sync files from my local computer to a non-trusted FTP site? To accomplish this, I'm using a script to check timestamps, encrypt and sign the files individually, then copy each file to an offsite FTP directory. I've looked over many different tools (Duplicity, Amanda, Bacula, WinSCP, FileZilla) but none of them seem to do exactly what I want: (1) multi-platform (Windows and Linux), stand-alone client (can be run from a portable drive). (2) Secure backup (encrypted and signed) to non-trusted FTP site. (3) Sync individual files without saving to a giant tar file. (4) Securely store timestamps and file names on the FTP server. Any help or info on alternative solutions appreciated."
This guy was always complaining about headaches. He would constantly be pounding his head into his fist and whimper to me that he felt like his head would split open. He took pain killers all the time, and for a long duration was addicted to a certain prescription pain medication. But none of that helped because as soon as the medication started to wear off, the pain would come right back again.
Finally, I had had enough of his complaining. I told him to stop pounding his head with his fist. Whaddayano! His headaches went away in a day.
Moral of the story: Don't try to find workarounds for your problem. Fix the problem.
I have explicitly asked my web host provider for either SFTP or FTPS. They basically said that it wasn't possible to provide that on a shared host. This seems untrue to me, I just can't state it as a fact since I haven't attempted it myself. But to get what the OP wants, one would essentially need a secure file system implementation on top of FTP. Ie. only the client can see the unencrypted file, not the in between transport over FTP, or the server side disk drive.
"Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
Am I the only one thinking this is like someone saying they want privacy then running around butt naked then wondering how they can keep their privacy at the same time.
What are you trying to achieve? It sounds like there is a problem there that you are trying to solve, but I'm sure there could be a better approach than sending encrypted files to an insecure FTP site.
First off, everything you send over the web using the FTP protocol is non-encrypted - even your password. Secondly, to achieve your goal, you would need the modern-day technological equivilent of a '60's-era 'scrambler' telephone device - a coder on your end, and a decoder on the other (in this case, one on the server). I'm not so sure many hosts allow their clients to install programs on their servers (chuckle).
All you desire exists in a protocol that uses one additional letter - sftp. Its existence is partly due to the weaknesses of FTP, so I wouldn't worry yourself over trying to make an older outmoded technology 'work'. If your host won't take the SFTP (SSH) protocol, I suggest you find another.
No, no sig. Really.
ThePromenader
"secure" and "untrusted" don't go hand in hand. If you want security, don't put things in untrusted spaces. Period.
Are you sure about that? I consider my SSH connections secure even tho' they traverse untrusted links. Same goes for my encrypted mails, https connections to my bank, etc.
Anyway, to the submitter - is areca close to what you want?
There are shills on slashdot. Apparently, I'm one of them.
rsnapshot?
http://www.rsnapshot.org/howto/
I might be wrong but i think he is asking for a way to backup files to an ftp server storing them in an encrypted format. so although its on an ftp server and not really secure the files would have to be decrypted to be of much value.
If you're looking for a backup solution - using rsync might work for you? I think this can send files acros FTP. Also - it backs up incrementally - so you should file that your bandwidth isn't screwed in the process. As for being portable, it might take a bit of work, but we've got it running on a windows box through cygwin. http://en.wikipedia.org/wiki/Rsync
I'm working on a backup solution that allows people to back up their data to a remote server securely and efficiently. For "efficiently", think rsync: only the differences are sent (and some information necessary to identify what the differences are). For "securely", think assymetric cryptography: your backup is stored in encrypted form, so that only someone who possesses your private key can use it.
All this is currently in very early stages of design. I'd welcome any suggestions for protocols or software I could use. Currently, I am thinking to implement a transactional network block device protocol, and implement the backup protocol on top of that. I still need to decide on a programming language I can use for parts I need to write myself, too (something safe (no buffer overflows, please), yet with byte level access...and no Java or .NET, please).
By the way, this is going to be a commercial product, but the code and the protocols will be open. I'll charge for the storage and bandwidth. :-D
Please correct me if I got my facts wrong.
The cross-platform for starters. Maybe? I don't think there's such an application.
Unison might fit the bill, but I'm not sure about the FTP part (it does work over ssh, I think).
The thumb drive req might be another problem, because I was about to suggest writing a Python or Perl script to do this (relatively easy). Most Linux distros have Python and Perl, but OS X and Windows I think you'd have to pre-install them. And Perl doesn't ship with an FTP client lib, I don't think.
If a user is transfering the files over Windows XP then he better start worrying about security holes of an OS long before FTP issue.
Even veals have more autonomy!
See http://www.truecrypt.org/ for cross platform encryption...you can throw your files in there.
Obligatory blog plug: http://www.caseybanner.ca/
I use Amazons S3 service and a great multi-platform UI called JungleDisk. S3 costs a little bit, but you get security (encryption), backup, reliability for a cheap price. Check out: http://www.amazon.com/s3 and http://www.jungledisk.com/
i thought filezilla supported sftp out of the box. at least i've been able to use it so. forportable cross platform, u might want to put linux/windows static binaries on your pen-drive.
"secure" and "untrusted" don't go hand in hand. If you want security, don't put things in untrusted spaces. Period.
I disagree. Everywhere you can store your files should be considered "untrusted". And "securing" the files is what we do to mitigate that reality.
duplicity combined with ftplicity:
"Anyone storing data on an unfamiliar FTP server needs to encrypt and sign it to ensure reliable protection against prying eyes and external manipulation. duplicity is just the tool for this, and the ftplicity script from c't magazine makes working with it child's play."
http://www.heise-online.co.uk/security/Backups-on-non-trusted-FTP-servers--/features/79882
http://duplicity.nongnu.org/
If you want security, don't put things in untrusted spaces. Period.
Completely, utterly incorrect. It's a sad comment on the ambient understanding of data security that this got modded insightful.
Trust is seldom a good approach to security. Good security is when you can trust nobody and still sleep at night. That means strong encryption. That is exactly the approach implied by the article and it is exactly the right thing to do.
I think it is very unwise to ever assume any level of trust in the storage of backups, certainly offsite backups. The whole idea of backups is that you keep them around for a long time, in several copies and several locations. The more valuable your data, paradoxically, the more copies you need and the more widely dispersed they should be. This is antithetical to maintaining trust. The right way, indeed the only way out of this paradox is strong encryption.
When hammering a nail into a wall, should I use a shoe, or a glass bottle?
Don't use FTP. Use something secure.
No, but I did throw granola at a deaf person once
some people prefer there nudity that way.
Areca might work, I'll have to give it a spin. Thanks.
Ok ftp supports reading chunks of data from files, i.e. byte range n-m.
However it doesn't support (I strongly suspect) _writing_ chunks. Sure you can say, 'REST n' and start writing but I think the file would be truncated.
This means, encrypted images like Truecrypt containers are out,s ince you'd be writing the entire file over and over again.
So you'll have to stay with single files.
Mod up. Its one of those issues that people who don't think very hard about a problem before making a judgment will get wrong 90% of the time. These are not the type of people to take security advice from. Listen to the parent, his wisdom shines through.
Areca might work, I'll have to give it a spin. Thanks.
No problem. Note that my post is not an endorsement of Areca, I just searched freshmeat for ftp encryption and perused a few of the matches. Have a read through the other results, you might find something else worth looking at.
Not sure what sort of budget/skillset you're working with, but it'd also be pretty trivial to script up a solution that does what you're after too.
There are shills on slashdot. Apparently, I'm one of them.
https://launchpad.net/ensure
Try the FUSE filesystem, encfs
Use in in reverse, provides an encrypted view of your data.
Then send that data anywhere you like.
If the backup is going to be stored in encrypted form then how is efficient "rsync-like" difference identification going to be possible?
A small change in a source file will likely change everything following it in the encrypted version.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
Untrusted storage site means others can access the files.
Access means they can decrypt them. Given enough cycles, encryption can be broken.
If you want your encrypted files to be secure, keep your keys protected and do not allow access to the files.
IMO, preventing access to the files is priority, encryption is only there in case preventing access fails.
It boils down to acceptable risk.
TANSTAAFL GIGO Acronyms to live by!
At the moment I'm having an intern write such a program in Bash as a proof of concept. We use GnuPG to encrypt.
How it works is:
- create a hash of each file in the dir to backup
- the hash is placed in a 'map file' with the original name and path
- if on the server side no file with that hash as a filename exists, the script encrypts the file, uses the hash as a filename en ftps the file to the remote server.
- the map file is encrypted as well and gets the date as a filename and is send to the remote server as well.
Pretty simple. The clue is that you have to trust the remote site only to provide the service. You do not have to trust them not read your files. As a hoster you can safely say that you really cannot read their files as the encryption is done by the client.
By using the hash check, we make sure only changed files are sent. This method also gives us history. You can simply use the map file of a certain date to revert to a version of the file at that date.
It works, but is absolutely not ready to be released.
---
Try Super Flexible File Synchronizer http://www.superflexible.com/ I've been using it to backup and sync my files over SFTP and FTP to two different FTP sites. It can use zip file encryption on each individual file, and uses file name mangling to retain the date information in the file. All this is transparent to the user. It can run portably if you use Universal Extractor (a portable app) to extract the contents of the setup file, then after first run, in the Options, tell the program to use a single .ini file. This one tool does all that you need.
FTP has no chance of being secured. It's also painful to support in firewall terms. If you need secure file management and your ISP provides only FTP, you need a new ISP.
WebDAV over HTTPS, however, is built right into Windows' "Network Neighborhood", Konqueror, lftp, and lots of multi-platform Java Apps. It's a core component of Subversion over HTTP or HTTPS, and most web servers support it quite easily. It's also often easier to get running than SSH with chroot cages or SFTP.
If you want untrusted, email me and you can FTP your files to my site.
I'm not sure if SSH or HTTPS is a proper connection comparison. SSH attempts to secure the untrusted lines or roads the information is taking when the two endpoint are secure. It doesn't secure the endpoint. Same with HTTPS. From what I gather from the question asked, it seems like he wants to use HTTPS to access a regular HTTP site for some reason.
I'm not really sure why someone would be concerned about securing a link between two point when I can defeat that security at least as one of the end point. It would be an illusion to think you would be using any security at that point unless you want to avoid someone sniffing your connection to find what your sending. But the problem with that is that they will still know where your going and they can simply view it once you put it there. If they are quick enough, they can see what was there and what is there after your transfer an figure out what your sent. Password protection on the FTP site isn't enough to stop someone from viewing it. All they have to do is compromise a user with higher access levels which could see all your directories and with regular FTP, your user name and password isn't encrypted. You would need at least SFTP or some HTTPS equivalent at least on the server side. Of course encrypting the file before hand is an option but then you lose the ability to browse the files without a special viewer that can read the encrypted files or decrypting them first. Something like SFTP or STUNNEL or even SCP might work well with a script or even a program that can use it but I guess there are some server side requirements that might not be able to be addressed.
I'd translate "wasn't possible" to "couldn't be bothered".
I'd translate it as uneconomic.
Deleted
This may well mean that despite whatever you do, encypt etc, someone can sniff the password and then simply come in and delete all your files. i.e, whatever other steps you take, this is inherently worthless.
If you can ssh to the site, you should be able to do sftp, which is basically ftp over ssh. That is about as secure as it gets without personalized encryption keys.
If you cannot ssh to the site, then you should find another host.
This is why they designed programming languages.
Grab Python, it's going to cost ~60 lines.
You feel sleepy. Close your eyes. The opinions stated above are yours. You cannot imagine why you ever felt otherwise.
Well, it's feature list is exactly what you want and some more :). Here's the project description:
Manent is an algorithmically strong backup and archival program. It features efficient backup to anything that looks like storage. Currently it supports plain filesystems ("directories"), FTP, and SFTP. Planned are Amazon S3, optical disks, and email (SMTP and IMAP). It can work (making progress towards finishing a backup) over a slow and unreliable network. It can offer online access to the contents of the backup. Backed up storage is completely encrypted. Backup is incremental, including changed parts of large files. Moved, renamed, and duplicate files will not require additional storage. Several computers can use the same storage for backup, automatically sharing data. Both very large and very small files are supported efficiently. Manent does not rely on timestamps of the remote system to detect changes.
Check it out: http://freshmeat.net/projects/manent. It's under active development (the UI and the setup are currently in fetal stage) but the basic functionality is there and is well tested.
Disclaimer: I am the author.
get duplicity
funny nobody mentioned it yet...
t
http://duplicity.nongnu.org/
Unless your planning on accessing your data from one location only, I'd suggest you forget it. I've been to different locations around the country and found that some bigger hotel chains as well as some so called 'hot spots' will actually firewall FTP. It's not unreasonable to believe ISP's are doing the same thing. I cannot access my ftp services from where I work. So, I'd be thinking of some other setup.
If possible, keep it simple. This is what I do - it is from UNIX, I don't know if Windows can handle it, but probably through a proper UNIX subsystem:
(cd /source/directory;tar cf - *)|ssh user@target '(cd /target/directory;tar xvf -)'
The left side will copy the whole directory tree under /source/directory and put it out on stdout in tar format; the right side will route the stdout to the target machine, where it will be unpacked under the target directory. If you don't want to copy everything, there are ways of handling that too - read the man pages of tar and find.
I've looked into this in the past. There is nothing better than Duplicity.
I eventually gave up and started backing up my data to servers that I do trust. You should too. You can rent a VPS for only $20 per month. It's just easier and *know* that you're the only one who has root access (assuming that you keep updating your system, of course).
I can see a relative easy solution for Linux and that is just scripting the whole thing. Almost any backup script should be able to do what you want and can get the files from Windows machines as well. That will be in CLI, which should not be an issue as backups should not run in GUI anyway, but automagicaly with cron.
It becomes different if you also want the restore to be in the same tool.
Don't fight for your country, if your country does not fight for you.
It is untrue. But just because something is possible technically doesn't mean that every host will do it. If secure file transfers are important to you, find a new host.
Then you should have put this as a requirement in your query. But I would ask WHY you want a gui? Backups should be set-and-forget! My USB sticks have multi-platform autorun scripts to execute my backup. I only need an interface if I choose to expand or shrink the backup set--I can edit a text file that has the list of what to exclude.
Python is pretty easy to put on a portable hard drive and there are multiple portable versions.
Blame this on my not writing up a really thorough spec for the small summary. You can see some of my other posts for more info, but this was sort of a query to see if anyone had done something similar because it seems like a simple project that might be useful.
As to the GUI, I was thinking it would be nice if it could double as a backup tool and a remote file system tool, ie access the files from another computer.
There is already at least one commercial product for live sync between machines: DoubleTake. It's not designed for public use over the Internet, though, more for replication between data centers. I'm currently taking part in the Dropbox beta test, and this looks like it's designed for normal users. It basically does what you describe - sends only the changed parts of files, replicates them between multiple machines, and keeps multiple file versions on their servers (so you can revert to an older version of a file). Have a look at the video on the website - describes it better than words. Files and transfers are encrypted, though they don't seem to do anything fancy with public key encryption.
(this is not a
This is my biggest problem with ftp. Is there any way to transfer files without mangling the date info and without having to wrap them in some kind of archive?? Is there a logical reason that to this day that ftp cannot keep it intact? Is there a reason that the date stamp is not contained in the file itself, as opposed to being "metadata"? What is the problem here? File management seems to be as primitive as it has been been since the 70s.
What?
I used a modified an amanda modified disk-backend with encrypt on the fly to store my amanda chunk on
SRB (an ftp like system). I you are familiar with amanda, look at amanda/chg-disk shell script, add some ftp command line in place of file commands and you will get a solution.
seems like /. is full of sysadmins who wouldn't think twice about reallocating some of their employers ample hardware to do secure backup, particularly of their divx and mp3 collections. their (hinted at) responses: stop trying to solve the problem and configure your own secure backup space on dedicated secondary hardware.
the rest of the world has to work inside of artificial constraints driven by cost, time and hassle factor e.g. they have only a single box and their hoster will give them ftp backup space cheaply but you don't want to take the files off of your own box unencrypted.
when i ran into this problem i used the lftp backup receipt outlined on the rimuhosting.com website to mirror a folder. then the trick is to only mirror a folder containing encrypted versions of your sensitive files. for that you would have to write your own script that scans your unencrypted folder structure and gpgs a copy into the mirrored folder. that is a none trivial script with checking and setting of timestamps. personally i didnt do that but had a cron job that tarred and gpged my database backups on a nightly basis.
surely it is fair to ask 'has someone not already solved this problem succinctly'? for example if lftp had an encryption option then a single tool would provide a zero hassle solution.
So I wrote it myself. http://freshmeat.net/projects/manent
It depends on what you put behind the word "security".
"Backup" is also "security". And a cheap of-site backup is better than no off-site backup at all.
I have the same need as the submiter as my ISP provides 10 GiB of public web space available only through FTP (r/w), HTTP (r) or HTTP+PHP (r/w). I have the storage, I need the software to use it while hiding backup content from my ISP and from other web eyes.
After thinking about this, I realize that I never should have written "non-trusted" in the description. I was thinking about FTP in the sense that it's inherently non-secure, which would make "non-trusted" redundant. I actually trust the FTP server as much one can be trusted.
This made me think about side question to which others in this thread have alluded: When is FTP as secure as SFTP? When you're on a non-trusted computer. It's a minor issue, but if you're trying to access an encrypted file from a non-trusted computer, a compromised SFTP client will do you no good if you use the same password on the SFTP server and your encrypted data.
It's been educational reading all these posts and thinking about various security issues with any of the solutions. I'm glad I submitted this question.
http://duplicity.nongnu.org/
Looks like it needs some work though..
I understand this is slightly different from what you described in the request, but I'd suggest (and maybe others did already) to have a look at:
http://www.jungledisk.com/
It allows you to do all you mentioned, except it places your files in Amazon Storage. So you pay a (incredibly small) fee for the storage and everytime you move files around.
Have a look.
Brackup is a Perl script for doing incremental backups. It's a bit like JungleDisk (see other posts here), except the client is freeware.
While JungleDisk uses S3 for storage, Brackup has the option to store on S3 or on a filesystem. You basically end up with (optionally) encrypted blocks, referenced by keys.
I've taken a peek at the code. You wouldn't need to be much of a Perl guru to hack in FTP support if that's what you want.
This isn't an endorsement for Brackup - I don't use it myself, although I've given it a trial run, been impressed, and may well take it further when I get around to it.
Rsyncrypto
http://rsyncrypto.lingnu.com/
Encrypts while making the above sentence untrue - a small change in the file results in a (relatively) small change in the cypher text.
It's stable, been around a while, and is written by me :-)
Shachar
P.s.
It is, in fact, written by me for the purpose of a commercial backup service. The software itself, however, is free.
you're trying to sync over FTP which inherently does not let you change the timestamps of the files you put up.
Access means they can decrypt them. Given enough cycles, encryption can be broken.
What are you talking about? Encryption that can be broken with any feasible level of computing power is worthless. If you're assuming that once the bad guys get your ciphertext they'll be able to decrypt it sooner or later, why encrypt your data at all?
Certainly I'd prefer to have my valuable data stored with both physical security and encryption. But if I had to choose one or the other, I'd definitely choose encryption. If you compare the cost of the security measure with the cost to circumvent it, strong encryption is many orders of magnitude better than physical security.
Are you sure about that? I consider my SSH connections secure even tho' they traverse untrusted links.
Nothing can be truly trusted - encrypted or not. Sure, with a sufficiently large key and matching cipher then the content might be secret for your lifetime.
Nothing is secure because it's only a matter of time/computing power to brute force it.
I drink to make other people interesting!
All encryption can be broken. The solution then, is to ensure that the encryption cannot be broken within a useful timeframe. I really don't care if you manage to decrypt my credit card number if the card has already expired. If I'm having a secret meeting this time tomorrow then the encryption only needs to last just over 24 hours, since by the time you work it out it will be too late.
I actually think you've got it backwards. Encrypt them strongly and you can put the data on a billboard in the centre of picadilly circus and no one will be able to work it out in a useful timeframe. Ever seen Kryptos? http://en.wikipedia.org/wiki/Kryptos The data is public, there are many thousands of people attempting to break it, and yet the hardest section remains unresolved. The acceptable risk is related to the minimum amount of time that you can allow for the code to be broken, which determines how strong you need your encryption.
This may well mean that despite whatever you do, encypt etc, someone can sniff the password and then simply come in and delete all your files.
i.e, whatever other steps you take, this is inherently worthless.
Hardly. As long as the data is encrypted well enough to stop people from stealing or modifying the data in ways that could have serious privacy and financial implications this is a net gain in data availability.
Even if the chance of someone doing this was as high as 5% over the period in question, it would still mean that there was 95% chance of you having a good off site backup. That is better than nothing as long as you realise that there is still a 5% risk and don't act like it is totally secure.
As a simplified example; if your PC at home is 95% sure of retaining all of its data in the period and your portable USB hard drive is 95% sure of retaining all of the data, the chance of you losing any data at all is 0.0125%. Even with exaggarated risk factors, this is not bad.
I use Cobian backup. It has an option to backup to an FTP location. It also supports strong encryption and password protected ZIP files. It has options to do full/incremental/differential backups so on a daily basis you're only backing up changed files. It's windows only, but version 8 (Blackmoon) is open source, so you could probably tweak it for linux. Or virtualize it. Whatever. Heck if you wanted to do a Linux port, he might even release the code for the latest version to you. J
I had thought of something like this a while ago.. what I came up with was basically a proxy FTP server, which encrypted the data and passed it onto the real FTPd..
So when you PUT a file, it passes it to encFTPd, which encrypts the data on the fly (using something like gnuPGP or RSA keypairs?) and uploads that file to the real FTP server.
Try Druvaa inSync (www.druvaa.com), the best feature is that it saves single copy of duplicate files saving time, bandwidth and storage.
The files are encrypted and compressed during transition and on storage.
and the best part if that its free for home users.
Im doing this and this an this and this and this and this to make sure my data is secure. I then I transfer it to an internet accessible site using an insecure protocol with clear text, unencrypted passwords over an unencrypted data stream where ANY host with a promiscuous mode interface on the same subnet (either source OR destination) could have his password, and complete access to all of his "secure data" within seconds of his first (and last) login. Answer: there is NO SECURE STORAGE using a NON_SECURE PROTOCOL. Next.
Armaments, 2-9-21 And Saint Attila raised the hand grenade up on high, saying, 'O Lord, bless this Thy hand grenade' N
Encfs stores all the file permission and date info and each file individually. Just FTP the encrypted version.
I have a web host for my website where I have a lot of storage. I wanted to backup some of my stuff to the site for off-site storage from my house. So I asked them to enable SSH for me.. After that I'm using some of the suggestions seen in this article http://ttsiodras.googlepages.com/backup.html which covers using an encrypted file system on the other side.
Automize 8.
-cp (Anonymous Coward)
Thank you.
Kid-proof tablet..
Have you tried super file sync..... It does everything you are asking at a reasonable cost... and last I saw it was multi platform.
Hardly. As long as the data is encrypted well enough to stop people from stealing or modifying the data in ways that could have serious privacy and financial implications this is a net gain in data availability.
Until some bored 16-year-old script kiddie virgin decides to be a "1337 h4x0r" and delete all of the files because My Chemical Romance broke-up. One could argue that it's no loss in data availability (as you still have the data on the hard disk and your external disk), but that's not very comforting if your house burns down and you don't have your data.
It's a sad comment on the ambient understanding of data security that this got modded insightful.
The fact that he said "Period." has me convinced.
At work we send secure student data over FTP to be scored and retrieve the scored data in the same manner. Instead of a secure channel we encrypt the data using each others' public keys. It's fairly easy to set up and use. http://www.gnupg.org/
I think you miss the point of the parent post. Even if all the files are encrypted, timestamped, hashed, verified, etc, all of that is relatively moot if it's all being stored on a plain vanilla FTP server. With a plain vanilla FTP server, anyone can easily sniff the password and have access to the encrypted files. Sure, they won't be able to decrypt and make use of the files themselves, but they can still delete them, which renders the "backup" fairly useless.
So, you really told the guy to quit using Windows.
If you are willing to drop the Linux requirements then I think Cobian Backup has everything you need.
Securing the protocol isn't an answer to the ops question. I certainly wouldn't trust my unencrypted files to a 3rd party. And if you're going to encrypt your files, the tranport method matters somewhat less (though I'd still prefer sftp especially for the authentication).
rsync and for security rsync + ssh. There are both Server and Client versions of the software for Win, Mac, Windows. Or Unison. All of it is Free.
How about a php script that sits on the server accessed thorough SSL. Server Script is an upload script that accepts local files that you have encrypted, and a key. Local script can encrypt your files based on a key. One problem (of probably many more)... the encryption might secure. Another problem... I am assuming you can use PHP and SSL.
I think you miss the point of my post entirely.
If you can make the risk of other people misusing the data minimal with encryption, the other concern is whether you can reduce the risk of losing access to the data yourself. For this the risk that someone will log in and delete the data is a risk like any other risk concerning data availability.
Compare this to the risk of
a) your hard drive failing
b) your house burning down
c) someone stealing your USB hard drive
d) etc.
Each of these risks are fairly small, and they can be combined to reduce the risk of losing the data.
Similarly, the risk of someone deleting your data off an FTP store is a real, although fairly small risk and combining it with any of the above, decreases the chance of you losing your data.
Thus it is not useless, it gives you additional data safety that you would not have if you didn't use this service. Remember: just because someone can sniff the passwords and delete your data, doesn't mean it will happen.
Ever heard of PERL?
While this is nice you just keep a fob for retrieving stuff remotely. Why not just get a 500G usb disk drive that works with all systems. Or even smaller laptop drive setup with usb?
For several years the project languished without a real maintainer and without an up to date, stable version, but because of this bounty:
http://www.rsync.net/resources/notices/2007cb.html
and because of the great work of Ken Loafman, there is a live and vibrant community of duplicity users and an up to date, stable version.
I personally use duplicity on my own server as well as on my remote rsync.net storage space.
BTW, this is not the first time that my provider, rsync.net, has gone out of their way to perform very useful, very pro-OSS work for the community ... as I say every chance I get, this says it all:
http://www.rsync.net/resources/notices/canary.txt
http://www.rsync.net/resources/notices/2007cb.html
Prior to that bounty, duplicity had not been updated or worked on for a few years, and thanks to the claimant of that bounty, Ken Loafman, there is not only a new, stable version, but a responsive community working on the project.
I use it every day, and now consider it indispensible.
Don't help him. Don't tell him anything. He's a lazy asshole who wouldn't take the time to do the research himself.
Which you should have known if you'd read the GPL3 and Affero GPL3.
But you'd rather spout FUD because you don't like GPL hippies.
Linux user eh?
Seriously though. Get Cygwin for your Windows box and then write a PERL script to generate an MD5 (or SHA) hash of your file, then encrypt it using AES, and then upload via FTP which is PERL scriptable. That's all!
I understand now. Thank you.
http://www.cis.upenn.edu/~bcpierce/unison/
Uses SSH (secure)
Cross Platform (we back up our Macs to a Linux server with it)
GPL
Darn easy to set up and maintain (one app on the client and server, create a config file on the client, setup server backup folder for client data and that's about it)
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
You may not store individual encrypted files, as SOMETIMES the file length could be used to guess it's content!!! You should always encrypt & pack as many files as you can in a single archive to avoid this. However, this way you'll be unable to do incremental backups :(
BTW, check out the "file length match trick" applied to the new ThePirateBay SSL system:
http://slashdot.org/firehose.pl?op=view&id=803927
It's relatively trivial to write a windows batch file and corresponding shell script to back up your files with the surprisingly-powerful ncftp, as encrypted and signed by gpg. It's not an all-in-one-executable solution, but it's pretty close. I reckon with a bit of effort you could even make your shell script and batch file the same file ;).
"But everyone should know everything." -markab
For Windows, I've found EaseBackup from KIE Soft to be a good, cost-effective solution.
The same software can be used for backup to media or over FTP. It makes zip files, can pgp encrypt them, and support either incremental or 'patch' mode (where it only uploads essentially diff files)
He has a Linux client, but I don't know if it reads the EaseBackup files.
http://www.kiesoft.com/
Of course encrypting the file before hand is an option but then you lose the ability to browse the files without a special viewer that can read the encrypted files or decrypting them first.
One of his four design requirements was "secure backup (encrypted and signed) to non-trusted FTP site"
So yeah, it sounds like he means to leave the file encrypted on the remote host. Almost as if he doesn't trust the remote host, but wants to use its storage for backups.
http://amanda.zmanda.com/
I stumbled upon brackup not too long ago, trying to solve a similar problem.
I believe brackup solves (1) I believe they want to support windows, and test on it, You can put the script + cygwin on a usb drive (2) (Dunno if it has an ftp plugin, but you can snag a perl dev to add that; it supports amazon s3, and sftp at least), (3) supports incremental updates, (4) does that too.
http://search.cpan.org/~bradfitz/Brackup/
Svn trunk and his release here:
http://brad.livejournal.com/2205967.html
Why aren't you encrypting your e-mail?
I think you've missed the point. If you're not allowing access to the files then encryption isn't particularly important now is it?
The whole point of encryption is that you could email it to your arch-nemesis and they would still be unable to decrypt it in a useful time-frame. Take AES with a 256 bit key. That would (on average) take all of the computers in the world millions of years to brute force. It's possible that someone could get lucky, but they'd have to dedicate years of processing time on the off-chance that you had encrypted something particularly juicy.
OK, there are ways that you can treat an FTP site as if it were a disk volume. Start by looking that up.
Now, encrypt the volume. If anyone mounts it without the volume crypto key they get gibberish.
Use Mike Rubel-style rsync-based backup methods (with or without batch mode files as you wish) to make your backups to the encrypted volume. The encryption will happen on your side so the keys and unencrypted data will never be passed over the insecure FTP connection to the insecure site.
You'll want some kind of checksum/hash/whatever so you can detect tampering or bit rot, but I'd expect that a good crypto filesystem will already have something of the sort.
Note, I haven't actually done this - I use SFTP instead of FTP and I don't use untrusted sites to store backups - but it'll work in theory, and I've seen all the pieces of it in various places.
That's fine and all but it isn't terribly convenient. I'm not sure how many times I have needed to restore a specific version of a file and looking at the file for the specific content is the best/most reliable way to do so.
Another note, storing a file like that is like putting your savings deposit in a bank bag with a lock on it and then just putting it on the sidewalk for strangers to see. One of them, maybe by looking where they shouldn't could think they could open it and take it then you don't have what you need when you need it. Regular FTP transmits usernames and passwords in the open which sort of makes anyone capable of sniffing either connection capable of accessing the files to do whatever with.
Answer: there is NO SECURE STORAGE using a NON_SECURE PROTOCOL. Next.
Sure there is. I will encrypt my 10 KB file of sensitive data and bundle it up with 10,000,000 similarly sized files containing pure random gibberish, and upload the whole mess to two dozen different FTP sites. Only I will know the name of the one file that contains encrypted data, and you can have the time of your life wasting your botnet time on attempting to "decrypt" all that trash. Elaborations on this theme abound, I'm sure there are plenty of ways to securely store data over insecure protocols.
I did "dd if=/dev/zero of=/tmp/test" for a 200MB file. Encrypting it with rsyncrypto resulted in zero repeats that I could find. In 200MB of zeros, not one ciphertext repeat was generated. This was due to non-alignment between the decision function triggers for the compression and the encryption, which meant that the same IV block for the encryption part started on different places in the compression block, resulting in no repeats over the entire 6.3MB of encrypted file.
The source is out there. Feel free to run your own test.
Shachar
Every part of that is right, but what he's doing might still make sense from a risk analysis point of view.
Any security person is going to hate seeing a vulnerability as gross as a plaintext password when there are zero-cost alternatives, but look at the threats and impacts.
If someone overwrites his backups, he'll detect it via the signatures. So overwrite and deletion have the same impact. The impact is limited by his other backups. The likelihood -- I'd say "certain in the long run", which is also true of defective backup media, which would have the same impact (losing one or more files).
Overlooked risks: FSM help him if he uses the same password anywhere else as he uses for the FTP session. Plus, do a threat analysis. What is going to be the motivation for the attacker here? We're talking someone who is sniffing an ISP's traffic. OP's username/password pair is going to be among the least interesting to an attacker with human limitations. Random vandalism is frowned on by old-school intruders and doesn't bring profit to contemporary financially motivated ones, so though it's going to happen eventually other things are more likely. By the time someone deletes his backup, there will have been multiple incidents of his storage being used as a warez drop or worse. Would the illegal files get "restored" to his working machines? Working from a catalog file and checking the signatures would prevent that.
This kind of analysis is overkill here, but you need it in general for real-life security issues. In summary, he's actually adding protection by making one more backup, unreliable though it will be, at the cost of increasing legal risks and spending time to monitor for unwanted files.
Will people PLEASE stop using "FTP" and "Secure" in the same sentence!?!?! We've been over this before. Unless you are referring to the "SFTP" functionality provided by SSH/SCP, just don't do it.
Duplicity rocks. I'm a long time user.
Your comment, likewise, is inherently worthless.
Yes, the password could be sniffed. No, this does NOT make the whole system worthless. It very much depends on what attack-vectors you are trying to mitigate. It is perfectly sensible to use unencrypted FTP if the server is on the same, non-internet-facing switch, for instance. Plenty of hosting providers will offer you FTP space to back up your server's data, too -- and it'll never leave their network. The better ones put a secondary interface in a vlan of its own, together with the backup server.
FTP is still quite usable if you do not need encryption -- and yes, there are cases where encryption is just not needed and only unduly burdens the CPU.
It may be useful in the author's case, though -- but even there it's a risk assessment -- what's the risk of his hard disk frying and a cracker deleting his backup server's contents at the same time ? How does it compare to the cost of that backup provider ? Is the trade-off acceptable ?
But yeah, blabber on about inherently worthless stuff.
Whats the point of storing it encrypted if your going to TRANSMIT it in clear text.. Freaking retarded.
Have you looked into PowerArchiver? I think it has at least some of the capabilities you're looking for, maybe all.
www.powerarchiver.com
sftp?
http://www.ahsay.com/en/home/index.html
This backup system does everything you want.
The simplest way would be to use an incremental rsync backup script, encrypt each individual file (or directory) in bulk as well as it's increments and upload the changed files to FTP with a decent syncing script. You'll need a local image of your backup though. Otherwise, find another storage area.
Custom electronics and digital signage for your business: www.evcircuits.com
After reading your submission, since there's no RTFA applicable, I took a look at the files on the FTP server in question. Your book proposal for a "Dummies Guide to Dialing Telephone Numbers " isn't going to be a bestseller let alone worth all the hassles of asking Slashdot for opinions. Sorry! ;)
I tried doing something like this a few months ago. I have a colo in a datacenter with an empty 250GB disk and wanted to use it to back up the contents of my file server at home. I needed secure authentication, transmission, and storage. And although I'm not doing anything illegal nor do I distrust the datacenter staff, I didn't want to keep any encryption keys on the colo box itself. My homebrew solution was to use SSHFS (which uses FUSE) combined with EncFS (also uses FUSE) plus rsnapshot to do the incremental backups.
This worked, but with one major flaw: the datacenter routers have a tendency to reset TCP connections after some random amount of time (between 1 minute and several hours, usually) whether there is any traffic on the connection or not. (I've tried talking to their networking staff about it, since it affects regular SSH sessions as well but they insist that it's a feature rather than a bug.) SSHFS isn't smart enough to deal with a broken connection and instead of reconnecting or issuing an error, the client just blocks forever when accessing the then non-existant device.
Because of this setback, I've nearly resigned myself to buying an external USB hard disk and doing my off-site backups sneakernet-style. I'm happy to entertain any suggestions on an encrypted remote filesystem that works (or at least breaks gracefully) with unreliable connections.
This software encrypts files and compresses them and allows you to customize the encryption so that other copies of the software can't decode your files.
http://www.download.com/The-Vault/3000-2092_4-10580448.html?tag=lst-1&cdlPid=10714591
And everybody who looks like you.
You don't understand core concepts, have a laundry list of imaginary needs and can't tell fuck from reality, then troll your get your happy ass on Slashdot looking for clues. Lemme put it to you straight pud... Your the kind of guy who can't handle security so give it up and go through life as though you have none, which in the final analysis you don't and probably never will. Schmuck.
FTP? Unsecure host? You figured out part of an answer being encryption but then have the audacity to dream rsynch or similiar. Via FTP over an unsecured link to an unsecured host? Even better your so called host won't lift a finger to help you. Who can blame them not wanting to waste time on some cheap clueless pinhead who can't comprehend hosting economics.
Ok nutter, listen up. You need an encrypted link and I don't care how you do it. Second, give up on partial updates within archives. You might have some success using WebDAV over HTTPS but don't hold your breath. If all you have to work with on the other end is FTP then your SOL. You could FuseFS over FTP but your username/passwords are in the clear which is the problem with plain unwrapped FTP in relation to security.
But then you don't know any of that having nothing more than a laundry list of dreamy wants and wishes, no fundamental understanding and even less resources to work with. Fucking luzor. Now hop on here and tell us where to send the bill for wasting our fucking time you piece of shit.
(1) multi-platform (Windows and Linux), stand-alone client (can be run from a portable drive). (2) Secure backup (encrypted and signed) to non-trusted FTP site. (3) Sync individual files without saving to a giant tar file. (4) Securely store timestamps and file names on the FTP server.
TrueCrypt (answers items 1 and 2, other posts talk about why signing is or is not needed) plus rsync (answers items 3 and 4; read the man pages on how to preserve time/date stamps and use an encrypted transfer protocol) and you're done. You don't seem to care about transferring whole files, as long as they aren't tomes of files and you don't seem to care if the transmission of said files is secure (nor its storage location for that matter).
I think the issue here for you and others is mostly psychological. I don't think you (the OP) is comfortable with your level of security and I think a lot of folks in this thread that are posting are uncomfortable with your level of security because their either a). exceptionally paranoid (acceptable, but sometimes silly) or b). don't know enough about what you're really doing to be able to answer you effectively. It sounds like you're relatively new to this whole "encrypted-backup-offsite-storage" thing and you need to do some more homework. Google is your friend. Asking /. is not usually a good idea unless you have a clearly defined question in mind and a lot of details to share, or you want a lot of useful but somewhat overkill answers along with the occasional troll and mocking answers.
fuse-ftp+encfs?
Too bad it's complete nonsense. Many of the same factors that create that 5% risk of loss overlap to both systems: viruses, fire, software errors, etc. Even if those factors are only 1/5 of that 5% risk of loss, their overlap for the two media will keep the overall risk at roughly 1%. This is basic statistics, which you should have learned by doing the actual science and engineering and gathering actual data, instead of just juggling numbers with a calculator.
Why look for a 'secure' solution and then use an insecure protocol like FTP?
Use SFTP if you want to do that. Accept that you will probably have to script some things.
If you don't like that then buy another hard drive, do a backup to that and keep it offsite in a secure location.
Thanks to many informative posts, I've figured how I will move forward with this, before I get to the answer a few comments.
I should have removed the word "non-trusted" from the description since the FTP server is one my accounts to which only I have the password. With this in mind, I can trust the FTP server as much as they can be trusted. Also, I would not use the same password for the FTP account and file encryption. For the one stop tool to answer the question, it appears that manent might fit the bill. A tool that I did not find when searching Google and SourceForge, but was on Freshmeat.
It was obvious to me that I was trying to build a secure pseudo-filesystem on top of FTP and I should have expanded my search to include that. That probably would have pointed me to the FUSE solution (which is not yet platform portable on the client end).
I also should have noted the point of this was to backup selected files to the FTP space on my web hosting service. My thought was, "Why pay for more storage elsewhere when I have this spare storage available?" As someone noted, this is probably a violation of my terms of service (and it is) so I will not actually use this to backup/sync my personal files.
So what is my solution? Since I will not be storing that much, I will probably go back to using Amazon S3 and JungleDisk, which I've already used and satisfies all the requirements. But if my hosting service ever allows backups to FTP, I'll probably go back to doing just that.
Even with all the trolls, ridicule and flames (many of which I found quite funny), I learned of a few interesting projects that never appeared in any of my searches. I hope that someone else finds this article educational, too.
SSH attempts to secure the untrusted lines or roads the information is taking when the two endpoint are secure. It doesn't secure the endpoint.
Except in this case, the endpoint is the user, not the FTP site.
Of course encrypting the file before hand is an option but then you lose the ability to browse the files without a special viewer
Sounds like he's asking for just such a viewer.
Don't thank God, thank a doctor!
Another note, storing a file like that is like putting your savings deposit in a bank bag with a lock on it and then just putting it on the sidewalk for strangers to see. One of them, maybe by looking where they shouldn't could think they could open it and take it then you don't have what you need when you need it.
Doesn't really work -- if they wanted to try to see what's inside it, they would copy the file (you can't copy a bank bag). For them to actually delete the file would require more than that -- it would have to be a deliberate act of malice.
Given that the files would also be signed, that's the worst they can do -- delete the file.
Now, I agree that it'd be better to do this some other way. That said, suppose I gave you free FTP access to a server with, say, 10 terabytes of space on it. For some reason, it's trivially easy for me to give you this FTP access, and be reasonably sure you can't do anything other than store files on it, but I'm hopeless to give you any more secure access.
And of course, you don't really trust me, anyway.
So the question arises -- would you rather take your chances that your backups could be destroyed, but not read? Or would you rather pay for backup (implying that you trust the endpoint more than me)? Or would you rather not backup at all?
All of this is very hypothetical, as that situation does sound contrived, even to me. But the idea of having an encrypted backup isn't a bad one -- I can see myself paying for storage on something like Amazon S3 (or, hypothetically, a cheaper competitor), but I can't really see myself trusting Amazon with everything I'd want to backup (SSH keys, porn browsing habits, etc). So if I was to do something like this, I'd probably encrypt the files.
Don't thank God, thank a doctor!
Nothing is secure because it's only a matter of time/computing power to brute force it.
Ah, but you see, when the time required is longer the heat-death of the universe, and the computer needed would have to be assembled from all of the matter in the universe, I'd call that pretty damned secure.
And the math behind that is simple: CPU time required to encrypt scales linearly. CPU time required to crack scales exponentially. A 4096-bit key is that secure.
No, it's not brute force. It's something you haven't thought of, like trusting your girlfriend with your password, or leaving your private key on a laptop that got stolen. Or you thought DRM could work, which means you gave the key to the attacker and hoped they never used it.
Or it's quantum computers (which also fall under "something you haven't thought of"), but that's because they operate in a fundamentally different way -- with a quantum computer, the time to crack scales linearly, meaning it really is only a matter of time -- to crack RSA. But people are already working on crypto schemes which would work well with quantum computers.
Again: Quantum computers don't kill RSA because they're faster. They kill RSA because they're different.
Don't thank God, thank a doctor!
Access means they can decrypt them. Given enough cycles, encryption can be broken.
When "enough cycles" is going to take them longer than the heat-death of the universe, I don't care.
Look, I know Uplink makes it look easy. The real world doesn't work that way.
Don't thank God, thank a doctor!
Although I prefer SFTP because the exchange is secure as well as the contents, I've run into the scenario more than once where the other end either does not understand how to use SFTP or prefers to encrypt the files themselves instead.
PGP (or GPG for a free option) will allow you to encrypt any file before sending it. It also has the added benefit of only encrypting the file with the public key of your intended recipient. That way, even if a man-in-the middle grabs your file stream or hacks the FTP server, your file is still relatively safe.
www.tumbleweed.com
Been using it fo over a year and it is great. http://areca.sourceforge.net/ Encypts, does incremental and ftp to my personal site, can't beat it. Even did a restore once from it very smooth. Has everything you could ever want.
Half of writing history is hiding the truth.
You can try: http://www.iqstorage.com/applet/ That's a free FTP applet, but it doesn't address all of your problems. It is cross platform, supports all type of encrypted FTP, easy interface, etc.
It's written in Java, therefore works on all platforms. And the command-line client may do what you want. http://secureftp.glub.com
TIMTOWTDI
I sort of prefer spelling it "ruby". You may have heard of it -- it is like Perl. It supports descriptive identifiers, comments, and (mostly) sane code organization. Unlike Perl, its programmers are encouraged to use these language facilities, rather than laughing them off as crutches for the weak.
If I sound bitter, blame 60k lines of Perl accompanied by 100 lines of comments written in barely literate Japanese and spread over 200 files. Yes, that does mean that many files had no comments whatsoever. After all, the reason e_st_item.pl exists should be obvious, right? (Screen for student course registration screen #3, in English.)
Help poke pirates in the eyepatch, arr.
No, rsync isn't a very good solution for a couple of reasons. First, unless there's some capabilities that I'm not aware of, rsync has no encryption capabilities....
Woah, stop right there. First, it would appear that you don't encrypt your files automatically locally before any type of remote sync. All things being equal given your local Security policy being the weakest point, perhaps the easiest solution is to find and/or build a trusted remote storage point? PuTTY SCP/SFTP could probably handle such a task if perhaps some of the concerns of untrusted destinations and crypto for cryptos sake were modified. Again, as another has pointed out, perhaps the easiest solution is to find the real problem
Well, yea, that is the point unless I wasn't clear. SSH doesn't make an insecure endpoint secure. It simply secures the transmissions to the endpoint.
Good luck. That will be bandwidth intensive as hell with no server side support.
I've had several backup/maintenance schemes set up at home over the years, often spanning Windows and Mac machines. Being a Java guy, I found Apache Ant to be a really good tool in place of shell or batch scripts. It's cross-platform, and it's quite extensible. And of course Java has pretty comprehensive cryptography API. And it wouldn't be much to wrap it all in a decent GUI, if that strikes your fancy or you want to roll it out to someone else to use, who's not comfortable with the lower level stuff.
Alexey
That will be bandwidth intensive as hell with no server side support.
Moreso than FTP itself? If so, how so?
Don't thank God, thank a doctor!
SSH will ask you to accept the host key the first time you connect to this host.
This pain is acceptable if the number of hosts in the pool is low. But an ISP may have tens or hundreds of servers in the pool. So host authentification can not be efficient if you connect each time to a different host.
How many hosts is there in the Cornell ECE department's pool?
They basically said that it wasn't possible to provide that on a shared host.
It wasn't easy or secure to do until recently. I think it was earlier this year that OpenSSH released trivial jail support. Figure out which version of OpenSSH they're running, and if it's current it might be time to ask again, with a link to the directions.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Really? +5 Informative? I suppose it contains information, which would be relevant to many questions about cross-platform encryption, but this doesn't seem really in the neighborhood of what the OP is asking for. They want a tool for uploading small encrypted files to a untrusted server, and the solution is to create an encrypted volume on the server (with, by the way, no mention of how to easily mount it over FTP).
And if they had access to copy the files what make you think they wouldn't go the malice route? They are already in an area they aren't supposed to be which show some malice in disregard of the rules. And your only right about the bank bag because we lack the specific technology to duplicate the bank bag with all it's contents. But there would be nothing stopping the person from cutting and pasting the file (from windows) which would be the same as taking the bank bag.
Ok, lets say that is all they can do. Why is the file being backed up? Whats going to happen when it isn't there if you need it? Or that specific revision of it?
Well, fist of all, I wouldn't be under any illusion that any security was there at all. That means nothing sensitive, nothing critical, and nothing that would/could potentially cause trouble would be stored there. Second, I would probably use the storage for non-important stuff or as a layer of convenience for stuff that isn't important like I already stated. This might include keeping copies of programs I use a lot so I can download on if needed, it might include copies of websites that are already public, maybe code for OSS software I'm working with (I'm not programing but already patched something to work in a specific environment) and stuff like that. It would be a convenience for non secure stuff but it would be in addition to other measures that ensure it isn't lost or corrupted and is availible is needed.
With all that being said, no, I wouldn't trust you because I know how the security of FTP works. This doesn't mean I wouldn't use your offer, but I wouldn't use it for anything important regardless of how many times I encrypted something. And yes, I would pay someone for a more secure backup that I could trust. Even if that backup person is me with a collocation somewhere.
Encrypting the files isn't the issue here. Or at least it isn't the ones I'm raising. The problem is that of an illusion that encryption would do anything to make you more safe and reliable. Hell, I'm not even sure that the files would actually be safe from being read even if encrypted. A brute force attack on a AES-256 bit key spread out over a 1.5 million node bot net could get lucky a couple of times.
With no server side support you would have to transfer the entire contents of the file every time you you wanted to view the encrypted file. Then if it isn't what you want, it would transfer all back. With plain FTP, it just gives you the header name so you can see the file is there in much the same way as ls or dir displays a directory listing. If you want to view the contents, you have to download it or the server can render it like a HTML if it is plain text depending on how it is configured.
With server side support, you can create a secure connection, have the server decrypt the files as needed, render the contents in a viewer that transmits only the characters your looking at, at a time. In this case, only what your looking at gets transfered and is done so in smaller chunks over a secure connection. You can even use server side support to open large archives or zipped files and present individual files within them. I remember doing something like that (without the encryption) at an accounting firm on a novel net ware server back in 1999 or so. We kept their master documents in a password protected rar file and a viewer/explorer app that would open whatever text editor/word processor they wanted inside it, allowed it to be edited and stored somewhere else. I now it cuts the transfer size down enormously and it sounds just like what he wants.
I think brackup supports your particular use-case. It was originally designed for backing up to S3.
Taral
WARN_(accel)("msg null; should hang here to be win compatible\n");
-- WINE source code
With no server side support you would have to transfer the entire contents of the file every time you you wanted to view the encrypted file.
Erm... why? I don't have to read the entire contents of my encrypted partition every time I want to seek somewhere in it.
Then if it isn't what you want, it would transfer all back.
WTF?
This kind of makes me question your understanding of computers beyond "This is a mouse."
It's called copying. I download the file to view it, which is copying it -- if I didn't want it, I do not then have to re-upload the entire file -- it will still be on the server, because I copied it, not moved or deleted.
That's probably not what you meant, but the way it's worded, it sure as hell looks like it.
With plain FTP, it just gives you the header name so you can see the file is there in much the same way as ls or dir displays a directory listing.
Right.
If you want to view the contents, you have to download it
FTP supports resuming a download. Even if it doesn't support specifying that you only need some tiny chunk of it, you can always close the connection (canceling the download) once that amount has been transferred.
So no, you don't have to download the entire file.
Same with HTTP -- only moreso; I believe you can request a specific byte range of a file.
you can create a secure connection, have the server decrypt the files as needed
Both of which assume you trust the server -- because the server now has unfettered access to your files.
I don't know about you, but I'm a little more paranoid. I'd rather the server not be able to get at my files.
render the contents in a viewer that transmits only the characters your looking at, at a time.
Ok, first off, that is a retarded idea, if you're actually talking about characters (as in, chunks of a text file). Text is small enough, and compresses well enough, that it makes more sense just to send the whole file. It'll use more bandwidth, but it will perform much better, because you won't need a roundtrip to page down.
I'll grant that it does make more sense for low-latency, low-bandwidth scenarios. But I can't think of such a scenario -- the only place where that bandwidth might matter is dialup, which tends to have a high latency, mostly because it has so little bandwidth that any other, simultaneous activity will saturate the connection and slow everything down.
Second, as stated above, there's absolutely no reason you can't do this with encrypted files stored via FTP.
You can even use server side support to open large archives or zipped files and present individual files within them.
And you know what? You can use client-side support to do that, too!
For a really impressive example, take HTTPFS -- you can mount an ISO over HTTP, and it's actually fast enough to browse files and such.
The only place where this would matter is for things like tarballs -- in which case, you're forcing the server to read the entire archive from disk (probably decrypting it in the process) before sending pieces of it back.
For an archive of any size where it'd make sense to do this, it probably makes much more sense to unpack the archive, or repack it as something like Zip, which is seekable -- and either situation puts us back to Square 1, where encrypted files over FTP can do the same thing.
Now, given the question that was asked, I'm assuming it's massively easier for him to add client-side support for this than server-side support. Given what I've just explained above, you can get almost all the same features with client-side supports, and the only ones that would really benefit from server-side support (that you've mentioned) are a bad idea anyway.
Don't thank God, thank a doctor!
Whats going to happen when it isn't there if you need it? Or that specific revision of it?
Then I'll go to a different server, where I've also got them backed up. Or I'll notice that the file is gone (by auto-checking for it) before I have a catastrophic failure.
Either way, I'm better off than with no backup, unless people are constantly deleting every single file.
Well, fist of all, I wouldn't be under any illusion that any security was there at all.
Erm, crypto is pretty good these days.
With all that being said, no, I wouldn't trust you because I know how the security of FTP works.
Assume for the moment that I'm using SFTP. Would you trust me then, any more than if I was using normal FTP?
And yes, I would pay someone for a more secure backup that I could trust. Even if that backup person is me with a collocation somewhere.
Unless that colo is, say, in the apartment of someone I trust implicitly, I don't really see how it improves things.
A brute force attack on a AES-256 bit key spread out over a 1.5 million node bot net [informationweek.com] could get lucky a couple of times.
Could it really? I don't actually know the mathematics on this.
I do know that it would likely have no chance against, say, 4096-bit RSA.
Now, generally, RSA isn't used to actually store the encrypted document -- that'd be AES. My point is, barring huge advances in quantum cryptography, I can probably trust that my data is safe. Even if a 1.5 million node bot net could compromise it, how long would it take? It'd be a hell of an expensive attack (considering that botnet could be used to generate revenue), which means I'd have to really have something of value, and they'd have to know.
Don't thank God, thank a doctor!
Lol.. If your going to pay for insecure redundancy, why not consolidate the expense and use a secure solution from the start? Outside of doing something to look busy, I don't understand why this is even a question.
Your missing the point, FTP isn't secure. You transmit the user name and passwords in plain text. Part of a security model is control over access. And even though crypto is pretty good, it isn't fool proof nor it is impossible to break. Unlikely is a good term but impossible, no.
After checking your credentials, making sure you as well as your server is in the US and under the US's jurisdiction, yes, I probably would. I'm not against a remote backup, I'm against thinking an insecure implementation of something requiring security is anything but insecure.
Well, there are some things that you have to trust that ethic laws and plain old laws will stop someone from doing something. I would make sure the colo is in the same legal jurisdiction as I am. Currently this is the US, If i was in the EU pr any other country, it would be to. In any colo, there will be a limited number of people with physical access and I can configure the machines to accept secure communications like SFTP. In a trusted apartment might be a good solution too as long as you can be garenteed reasonable access in cases of emergencies. Even if you get drunk and screw your friends wife and he hates you know.
Going through all the possible keys, it would take more time then "we" have on this world. But dividing the keys into segments equally across 1.5 million computers means that we are guessing in the middle of the middle of a middle of the possible answers multiple times. To illustrate this, imagine we had a 5 digit key with 10 possible numerical characters. That would be something like 10^5 or 1,000,000 possible answers. Now, if we divide that space between 10 computers, we would have every computer looking for 1,111 answers each. Lets just say our key is 45698. The first computer would be looking at 00000 through 11111, the seconds 11112-22223, third 22224-33335, fourth 33336-44447, fifth 44448-55559, sixth 55560 and I'll stop there because we already reached our key number. Now, if the computers can do one check per second, it would take 1,250 seconds instead of the 1,1111 seconds. That is something like 21 minutes to find a key compared to the 185 minutes to run through the possibilities. Of course this doesn't mean a key will be found with with larger keys, but it takes the infinite number of possibilities and breaks it down by a factor and then relies on luck that each key wouldn't be at the very end of the range of possible answers. AES 256 is something like 10^(76) possible answers which is huge. But by breaking down the possibilities, you have a better chance of getting lucky.
No chance requires a level of luck that to not be present. You see, as I pointed out before, the key isn't always going to be the absolute largest possible answer and when you divide the pool down to the number of nodes, you could end up with it bei
Lol.. If your going to pay for insecure redundancy, why not consolidate the expense and use a secure solution from the start?
That's assuming a "secure solution" exists. I think I adequately proved that such a "secure solution" is at least as impractical -- it pretty much involves a machine that you've configured running in the apartment of someone you trust not to let it be physically tampered with.
Your missing the point, FTP isn't secure. You transmit the user name and passwords in plain text.
Yeah, I got that. So what?
And even though crypto is pretty good, it isn't fool proof nor it is impossible to break.
Fine then -- so suppose I use your "secure solution" from the start, and I'm using rsync over ssh to a machine in my friend's apartment.
What's to stop them from cracking the SSH?
After checking your credentials, making sure you as well as your server is in the US and under the US's jurisdiction, yes, I probably would. I'm not against a remote backup,
Neither am I.
Suppose for a moment that you wouldn't. Does encrypting the files start to make sense?
If you're already encrypting the files, SFTP doesn't buy you a lot over FTP.
Well, there are some things that you have to trust that ethic laws and plain old laws will stop someone from doing something.
Unless my files are encrypted. Then I don't actually have to trust "ethic laws" or "plain old laws" -- short of being myself kidnapped and waterboarded, I'm safe.
AES 256 is something like 10^(76) possible answers which is huge. But by breaking down the possibilities, you have a better chance of getting lucky.
Right. But let's do the math. Assume a million computers:
10^76 / 10^6 = 10^70.
Is it really that much more likely that you will get lucky with 10^70 possibilities than with 10^76?
when you divide the pool down to the number of nodes, you could end up with it being within the first day of calculation.
Right. Still isn't going to happen.
See, Quantum Physics also predicts that it's entirely possible that every molecule in your body could simultaneously jump a foot to the right. It's just so ridiculously improbable that, in fact, no one has ever seen it happen -- and no one ever will.
Let's assume, for the moment, that each compromised machine has a dual-core 2 ghz chip -- likely far less on most, but let's just assume. Assume 1.5 million of them. Assume that it takes only one cycle to attempt one key -- it doesn't, but pretend it does. And assume that threading is perfect -- you can keep both cores warm -- you can't, but pretend you can.
Note that all of these allowances are in your favor.
So, that's 2 ghz = 2 billion operations, times 2 cores per machine, times 1.5 million machines. By my count, that's 6 quadrillion tests per second. That's a lot, right?
Let me give you some more numbers: 60 seconds per minute, 60 minutes per hour, 24 hours per day, 365 days per year. So, a little over 31.5 million seconds in a year.
It will take 52849653306274315810851830343929408752938685099409408 years to search that keyspace.
That also means that the probability of a hit in the first 100 years, assuming that the key could be anywhere in the keyspace, is approximately 0.00000000000000000000000000000000000000000000000000189216 -- percent.
Adjusting for Moore's Law, and assuming that we'll have 2^(2*50) = 2^100 times more power by then -- and then, to make the math simpler, assuming we have that kind of power from now until then -- it's still about 0.0000000000000000000023986 -- again, that's a percentage.
I keep trying to make your odds better, and I still can't get anything remotely realistic to suggest that even if a botnet were devoted full-time to cracking my personal backup, that there's even a reasona
Don't thank God, thank a doctor!
Well, I suppose that should be:
Winning the lottery, and then getting killed by lightning...
Don't thank God, thank a doctor!
As I said before, there probably isn't any 100% secure solution. What you do is make something unlikely or improbable. So using FTP verses SFTP would be like walking around all day mumbling the combination to your locker and expecting no one to attempted to get it. It's something you just don't want to do. Attempting to hide behind some illusion of security is like putting yur hands over your eyes and thinking if you can't see them, they can't see you. It doesn't work that way BTW.
Lol.. remind me to never use you for anything that is sensitive. The so what is that if it is that important that someone needs to use security, then regular FTP is completely inadequate. I'm not sure why you want to argue that braodcasting a user name and password to a server storing the information is in the least but secure.
You do understand the difference between giving someone something and making them work for it right? Offer $100 to the first person who shows up at your door and then offer another $100 for 10 hours of light to medium work. See how many show up and which offer gets more given the same advertising. First, there is no guarantee that someone can crack the SSH. Second, it requires a skill set a little more in depth then running a sniffer program. By running SSH, you have effectivly excluded a good portion of potential attackers. It's like leaving your car unlocked and the windows down. Locks only keep honest people out but without a lock, a lot of honest people can become dishonest really quick.
If I store something important that needs to be secure, it isn't going to be on an insecure server. Period, end of story. I'm not going to even mess with FTP or anything because I want it to be there when I need it and I don't want something discovered by a Romanian, Chinese, or Russian hacker that gets around any encryption out there to make my illusion of security inadequate. Encrypting the file from the start makes sense as part of a solution but it can't be all of it. We have encrypted codings that get broke every year for fun and profit. How well have you don't your job at securing some backup if it isn't there when you need it or through either some hole discovered or getting lucky on brute forcing a key, anyone can open and view it just because you used FTP which broadcasted your username and password effectivly allowing someone easy access if they are sniffing either side of the connection? The answer to that is not very well.
Actually it is. If you have to rely on stumbling across the key, you are 1.5 million times more likely to get the right key on the first try. It increased with ever subsequent attempt. Sure, there remains a large set of possibilities but not all keys will be the very last one you try. The example I gave showed this and it showed how using multiple computers can take an extremely long time and cut it down quite a bit. The problem is that you simply do not know how long it will take to discover the ke
So using FTP verses SFTP would be like walking around all day mumbling the combination to your locker and expecting no one to attempted to get it.
If, when they do, they find a locked safe, I'm really not all that worried. In fact, I'm probably more secure than the locker next to me, who relies entirely on the locker itself, and the lock.
I'm not sure why you want to argue that braodcasting a user name and password to a server storing the information is in the least but secure.
Because, as I've demonstrated above, my username and password to that site don't have to be secure.
First, there is no guarantee that someone can crack the SSH.
There is no guarantee that someone can crack the files.
Second, it requires a skill set a little more in depth then running a sniffer program.
So does cracking AES256-encrypted files.
By running SSH, you have effectivly excluded a good portion of potential attackers.
By encrypting my files, I think I'd exclude the same portion.
So, in summary, encrypting my files is no worse than using SSH with unencrypted files -- and probably a bit better.
If I store something important that needs to be secure, it isn't going to be on an insecure server. Period, end of story.
In other words, "I read your argument, and couldn't find anything wrong with it, except the conclusion."
I don't want something discovered by a Romanian, Chinese, or Russian hacker that gets around any encryption out there...
Hey, here's a tinfoil hat. It protects you from aliens and their brain scanners.
You know what else doesn't exist? Romanian, Chinese, or Russian hackers who can get around other encryption.
We have encrypted codings that get broke every year for fun and profit.
With what algorithm? Single DES?
anyone can open and view it just because you used FTP which broadcasted your username and password effectivly allowing someone easy access if they are sniffing either side of the connection?
Whoops, looks like you still haven't figured out about SSH -- which broadcasts the contents of your files to everyone. It just happens to encrypt them first.
If you have to rely on stumbling across the key, you are 1.5 million times more likely to get the right key on the first try.
Correct. 1.5 million times the probability that every atom in your body will jump a foot to the right is still a stupidly small number.
The example I gave...
You mean the one I tore to pieces?
I don't need to try all possible keys.
You're right. You've also failed to read why this doesn't matter:
the probability of a hit in the first 100 years, assuming that the key could be anywhere in the keyspace, is approximately 0.00000000000000000000000000000000000000000000000000189216 -- percent.
No, you're not trying the entire keyspace. The point is that the probability that the key is within the area that you've tried in 100 years is still ridiculously, absurdly, impossibly small.
The math is nothing but an illusion.
Actually, math is hard fact -- the only hard fact.
I'm not sure more likely is the question that should be asked.
Actually, yes, it is, and that's the question you seem to be asking here:
We all know that the web browser can be exploited to compromise a local system.... It is an entirely different story when a man in the middle can take information gained and then gain this access.
Erm, how so?
What, exactly, is stopping the man in the middle from be
Don't thank God, thank a doctor!
CoreFTP www.coreftp.com has many of the features you're seeking but it's a Windows tool. If can compare timestamps to see if the client file is newer. It encrypts each individual file before sending it across the FTP link. The files are stored individually on the host. The only piece that I'm not sure it can do is be on a USB drive.
And while not open source, the 'lite' version is free.