Arkeia Network Backup Agent Remote Access

hdm writes "The Metasploit Project has published a security analysis of the Arkeia Network Backup Client. Anyone able to connect to TCP port 617 can gain read/write access to the filesystem of any host running the Arkeia agent software. This appears to be an intentional design decision on the part of the Arkeia developers. A long-winded description of this issue, complete with screen shots, demonstration code, and packet captures can be found in the research article. Arkeia has been credited with being the first commercial backup product for the Linux platform."

Comments

Somebody has to say it

[Renegade Lisp]

Well, to state the obvious: Would this problem have survived for so long if Arkeia Network Backup had been open source software?

Large enterprises migrating to Linux now should be careful not to throw away the biggest advantage of their new platform by committing to all sorts of closed source software that happens to run on it.

For the time being, I guess I'll stick to my proven, open source (free software even) backup solution involving tar, gpg, and ssh.

Re:Somebody has to say it

[badfish99]

No it wouldn't, because people would have spotted the decision at an early stage and told the developers that it was stupid.

With a commercial product, it took someone with a network sniffer to discover this. So it's just a lucky fluke that someone other than the bad guys knows about it.

Re:Somebody has to say it

[Eric Giguere]

if Arkeia Network Backup had been open source software

Well, it kind of is open source software... install it and it opens up your source (and pretty much anything stored on your computer) to anyone who wants it!

Eric
See what headers your browser is sending

Re:Somebody has to say it

[Donny Smith]

> For the time being, I guess I'll stick to my proven, open source (free software even) backup solution involving tar, gpg, and ssh.

You aparently either run a limited number of basic backup jobs and/or have plenty of time to write backup scripts, which is not bad as you need something to make yourself look very good at your job.

And FYI Arkeia Light is free (as in "one can use it at no cost"), see arkeia.org.

Re:Somebody has to say it

[Rich0]

Hmm - doesn't look like it fits my bill (a shame - I'be been looking for a better backup solution and have yet to find it).

Here are my requirements:

1. Backups are encrypted.
2. Backup data can be split across media.
3. Backups can use include/exclude criteria.
4. Corrupted backup files are recoverable.
5. Backups are compressed.

I've yet to find anything free which does all of this. Instead I'm using a short shell script combo of tar/bzip/gpg/split which gets the job done, but not elegantly. I'm not 100% sure how successful #4 would be with this setup. I think gpg has some support for corrupted files.

Honestly, I don't care that much about ECC and all that. My main concern with #4 is that if one byte in the backup file is messed up, I don't lose the ability to read everything else in the file. I can tolerate having one file on my system which gets lost in a disaster...

Re:Somebody has to say it

[nurd68]

Have a look at dar (should handle 1-3, and 5) with par2 to get 4.

Actually, even without par2, dar will be able to get all the files except the ones in regions with bad data, IIRC.

Re:Somebody has to say it

[Epsillon]

Well, it kind of is open source software... install it and it opens up your source (and pretty much anything stored on your computer) to anyone who wants it!

Looks like someone took Linus' quote and ran with it:

"Only wimps use tape backup. Real men let everyone else mirror their data!"

got root?

[cgranade]

Seems to me that the only way to get r/w access to the entire filesystem is if either a) the backup daemon is running as root, or b) if the backup daemon's user or group has r/w access equal to root's. In either case, the sysadmin would have to be on crack to do that. Not that read-only access is OK by any stretch, but just making the point. Oh, and before idiots start saying "see, open source isn't secure," let me remind them that this is a commercial product that was comprimised. If anything, I'd take this as further evidence of the virtues of open source.

Re:got root?

[Zocalo]

It's a piece of backup software, at the very least it needs to have read access to everything it is going to be used to backup. If you are planning on doing a full system backup, that means it needs read access to the whole filesystem or it can't do it's job. That doesn't mean it needs to be running as "root" of course; ideally such a tool would be running with a dedicated user and group. On a Windows box however it's not uncommon to see backup utilities running with higher priviledges than the "administrator" account because that's the only way to sidestep things like system file protection and other tricks Microsoft uses to protect the system from abuse.

Re:got root?

[danielrose]

Perhaps allowing read only access to the physical device, ie /dev/sda, and backing up at a bit level, of course this does not seem to be as economical or easy as backing up individual files... but i may be wrong.

Re:got root?

[cgranade]

Why not give the daemon read-only access to the source for backups? Seems like it's be a straight-forward enough thing to do.

Re:got root?

[TheRaven64]

Backing up from the raw device rarely a good idea. Backups of this nature need to be atomic, so you need to unmount the filesystem, copy it, and then remount it. You will need to keep the FS unmounted for long enough to do a complete copy. A better solution is to provide rôle-based access control (SELinux does this, for example), and create a backup rôle which has read access to all files on mounted filesystems.

Re:got root?

[FLAGGR]

To the second point..

but thats the whole point of the /etc/shadow... passwords... everything else is in the public readable /etc/passwd. In a network with more than 10 users you want to keep a backup of the /etc/shadow somehow.

Re:got root?

[Scorillo47]

>>> On a Windows box however it's not uncommon to see backup utilities running with higher priviledges than the "administrator" account because that's the only way to sidestep things like system file protection and other tricks Microsoft uses to protect the system from abuse.

That is not true.

All you need to read a file system in Windows is the backup privilege. You don't even need to be an adminstrator. So if you have this privilege enabled, you can use the BackupRead API to backup stuff.

Re:got root?

[Jester99]

On a Windows box however it's not uncommon to see backup utilities running with higher priviledges than the "administrator" account because that's the only way to sidestep things like system file protection and other tricks Microsoft uses to protect the system from abuse

Actually, you'll almost inevitably see backup utilities running as LOCALSYSTEM, which is indeed higher than Administrator, because that's how Windows works.

If a user registers a program to run as a service (note: You must have Administrator-level access to perform this step), then that program will run for all users regardless of who logs on (ideal to ensure that a backup program runs every night), and programs that run in this mode are always LOCALSYSTEM.

It's not a hack or a "trick," that's just how the security model works. But since you've got to be the equivalent of root to install such a program, it's not exactly a problem.

Not a bug; it's a feature?

[physicsphairy]

"This appears to be an intentional design decision on the part of the Arkeia developers."

Does this mean that, possibly, they were anticipating people *not* being able to access TCP port 617? I.e. "we trust you know how to properly configure your firewall."

So far, I can narrow down to either that, them being drunk when they coded this, or this being a case of the improper usage of the word "intentional."

Re:Not a bug; it's a feature?

[Zocalo]

Even if they were making the somewhat idiotic assumption that all of their users were behind a properly configured firewall, so what? That makes absolutely zero provision for a potential cracker having already circumvented the firewall by other means or even the possibility that they might be an employee. Or haven't they seen any of the reports that a significant amount of computer crime is committed by aggrieved employees?

I don't think it's so much improper usage of the word "intentional" as an incorrect synonym for the term "brain dead".

from the arkeia site

[Dr.Opveter]

Arkeia.com

I was looking for a Client-Server backup system that could offer me the possibility of backing up Unix/Linux and NT Servers on a single tape system.
After long research my choice went to the Arkeia solution, because it has all the benefits I needed. Since then, it runs like a black box, without any need of additional Service.

Tom Weber, IT Manager
RTL TV (Europe)

The backup system running like a black box might not be a good thing here eh?

Re:from the arkeia site

[DingerX]

I'd say the worse thing here would be being a published user of a system with an "interesting" security hole like that; all of a sudden, a friendly testimonial becomes an advertisement of a vulnerability.

Unless, of course, they've got everything firewalled to tuesday.

Zzzzapp

Nope, metal.

Specifications

[Fox_1]

It's very frustrating when you find previously unknown and undocumented features in software that you have purchased. I remember having to provide clients with full copies of the specifications and code for software so that they would be able update/repair/modify if I was hit by a bus or something. Security through obscurity is not safety, that should be validated by now simply by the sheer number of stories similar to this Arkeia one. Open Source Software at least has the beauty of the source code being readily accessible so that the user/admin/owner can see what they are installing on their system. This poor guy in the article ended up having to reverse engineer his software to find out the security dangers. Which may be against a law somewhere, ha - putting a backdoor into software you give me not illegal, finding that backdoor - may be me in trouble. I love it.

Re:Specifications

[TheRaven64]

I think your post is probably the best one I've read on Slashdot explaining the benefits of open source, or free, software. It's not about giving the code away to everyone free of charge, it's about ensuring that those people who rely on the code have the ability to modify it.

Re:Specifications

[hunterx11]

Obscurity would be hiding the fact that your safety is reliant on a number between 2^0 and 2^69 (as opposed to say, a word). If you do use such a number and don't feel the need to hide this fact, then your security is not through obscurity, it's through hardness.

Re:Specifications

[Fox_1]

Exactly - choosing a number that is hard to guess is security through hardness
not telling me that number even exists would add security through obscurity
The point is though that this software relied on obscurity to protect the built in backdoor, once that obscurity is gone the software doesn't even have something as brillant as a hard to guess number protecting the backdoor.
I call it the jerk arguement
- I can call you a jerk behind your back - security - obscurity
if you hear about it though - i'm hosed
- I can call you a jerk to your face, while holding my louisville slugger
security - Hardness (maple in this case)
(no offense I don't even know you and of course don't mean to suggest anything negative about you, just creating an example)

Re:Specifications

[Spoing]

1. It's very frustrating when you find previously unknown and undocumented features in software that you have purchased.

Well, for this situation finding a potential problem is easy: Port scan, security scanner. Two things that you should be doing on every network enabled device.

The time consuming part comes with the follow up where you check the results of the scans on the local machines and determine if you trust that the exposed services are being handled by secure apps. If in doubt, use an encrypted tunnel or yank the service -- whatever is appropriate. (If neither is an option, determine the danger and try and deal with it as best you can.)

Along with that, setting up a filter to check for supposedly unused ports can catch some clever developers.

Not perfect (it doesn't handle piggybacked dynamic connections on port 80 for example), though it is a good initial test.

Re:One more strike

[bferrell]

Arcserve is nice. But what about bacula?

http://www.bacula.org/

The oldest excuse in the book

[HeghmoH]

"It's not a bug, it's a feature!"

What a bunch of morons. It's one thing to accidentally write a security hole in your software. It's another thing entirely to claim that you deliberately make it so your software leaves your users' systems wide open to anybody who feels like taking advantage.

A good saying

[Capt'n Hector]

Never attribute to malice what is explainable by stupidity. (though the Bush admin. has stretched my imagination...) Though it appears intentional, there is probably a very good explanation for all of this. Needless to say, we'd better be hearing soon from Arkeia as to exactly WHAT that explanation is.

It may have been said before...

[caluml]

Well, let me be the first to say that I for one welcome our new nmap -sS -PS617 -iR 0 -p 617 -ing overlords.

Security available, just not enabled by default

[Anonymous Coward]

Arkeia provides both authentication and encryption of the connections - if you enable it. There is a part of the manual that covers how to enable security.

It is indeed bad that it is not enabled by default. On the other hand, enabling authentication of the backup server on the backup clients means that it is slightly harder to set up a backup client.

The problem is not much worse than, say, nfs. (Where impersonating a host can get you everywhere unless authenticated rpc is used.

Hum off topic'ish.

[zijus]

Hi there.

Well I just dealt recently "simple" backups via rsync + ssh. If you can rsync something from remote onto target with no special protection regarding rsync... If target is compromised, a malicious user can run arbitrary commands through rsync. And rsync server provides full read access to FS. (Well, within user permissions though.) Isn't it a bit the same problem that this software has? I would not be surprised to hear that you can customize the backup server to limit access/actions for better sefety. Which is exactly what you have to do with ssh on remote server: filter commands passed through ssh before running them. I mean: each remote you want to back up will have to be worked on a little.

It's off topic but FYI: Rsync server can take as a file list an arbitrary unix command.

rsync user@remote:'`\rm -rf /`' .

Pretty efficient isn't it ? (unix file perm will limit the damage though).

Bye bye.

Z.

Re:Hum off topic'ish.

[Anonymous Coward]

Well I just dealt recently "simple" backups via rsync + ssh.

I'm assuming you are doing really simple backups...how do you handle complicated tape library management (ie: tape robots, backup aging, onsite/offsite backups) automatically without having to use software more complicated than the basic Unix command line utilities? I'm not targeting you in particular, but there seems to be a lack of realization in general in this thread that backup systems are usually more complicated than just sticking an 'rsync' or 'dd' command into your cron files.

Re:Hum off topic'ish.

[Chris Croome]

If target is compromised, a malicious user can run arbitrary commands through rsync.

I agree this is an issue, the best solution I have found is Push Mirroring with this the command that can be run is put in the ssh public key and then the compromised client can only run this specific command.

Re:Hum off topic'ish.

[mmurphy000]

how do you handle complicated tape library management (ie: tape robots, backup aging, onsite/offsite backups) automatically without having to use software more complicated than the basic Unix command line utilities?

By not using tape. rsnapshot going to a sufficiently-large RAID array or drive covers your regular backups, including aging. A separate rsnapshot or rsync can do nicely for offsites, pushing the backups to another server. For enterprises, this approach probably is insufficient, but for smaller firms (e.g., ~70 employees, 5 offices), this works well.

Only wimps use tape backup...

[ttys00]

...real men just install Arkeia for their important stuff, and let the rest of the world mirror it :)

Uh...

[warrax_666]

... if the software doesn't need the port to be open on the internal network then why is it open?

Firewalling the port on each indivudual system behind the main firewall would then imply that the software couldn't actually function (for any reasonable definition of the word "function").

Re:Uh...

[prefect42]

Have you never used a firewall? Think filtered not blocked. Configure it such that it'll only allow packets from the backup server to that port. Bingo, job's a goodun.

Call to slashdot from a now ex-Arkeia customer.

[Ritontor]

Well well, isn't this interesting. I've had Arkeia running for a while now, backing up a number of different machines with a variety of linuxes, and I chose it because it was the only one that had any sort of support for Debian Sarge. It's been fine, apart from some unstable MySQL support, but other than that, a great piece of software. Until now.

I can't ever trust these guys again. When I first installed it, this issue occured to me, and I just assumed "no way could those guys be that stupid, they must have some internal IP restrictions" - and indeed, seeing as when you install the client it asks for the host server, I figured everything would be fine. If only I had've been wearing my tinfoil hat...

So. Who's got any better recommedations? I want some network capable, high quality backup software. Amanda doesn't cut it, and that was the best of the freeware stuff I saw. What else is out there that has support for a variety of linuxes? Veritas Netbackup wouldn't even touch a Sarge install, it was a dependency hell that I didn't have the time nor patience to get in to. I've got Redhat boxes, from 7.2 to 9, that all need backing up too... So what are the pros out there using? Is there anything that isn't rsync and a few mt commands in a bash script?

Addendum :

[da5idnetlimit.com]

"Said Linus Torvald, 10 Minutes before a HDD crash made him lose most personal notes, emails, docs and latest kernel modification his cron job didn't get a chance to duplicate..."

Check it up...

Easy: Use QuickPar or some form of PAR2

[jvbunte]

Here is an instance of the warez-monkies indirectly contributing something useful. PAR2 is essentially a RAID5 type data parity for files. Warezpups use it to add a layer of parity to their hundreds of RAR files (or whatever). If one (or more) RAR's go bad, the parity files can be used to reconstuct the bad file. Much like RAID5 however, there is a space sacrifice for this extra parity layer.

http://sourceforge.net/projects/parchive/

Its frequently used on USENET binaries groups now as well to solve the missing part problems.

I'm sure this exact strategy could be integrated into your backup solution with minimal effort.

Re:Easy: Use QuickPar or some form of PAR2

[Fweeky]

"Much like RAID5 however, there is a space sacrifice for this extra parity layer."

But it's settable; so if you want to be able to recover fully from losing/corrupting 20% of your backup you just set it to 20% of your backup size, and if you only care about a few minor bit errors or so, you can drop it to a couple of percent or less.

Be nice if vendors provided PAR2's for their ISO/DVD images/anything else big; it sucks when you find the MD5 of your download doesn't match the one they provide (or that 400MB setup.exe throws a checksum mismatch and refuses to run), and you know it's probably just a single bit flipped somewhere but can't do much beyond redownloading the entire thing. rsync helps, of course, but that's a *lot* heavier on the server both in resource use and administration cost.

Arkeia Backs-up Great - Restore Is a Problem

[zentec]

I ran Arkeia with a large web hosting firm for about 2 years mixed with Linux and Windows machines. We tested the backups extensively before deployment and spent $18,000 with Knox for licenses.

All seemed well until we needed to restore data. The logging indicated a perfect backup, but time and time again our restores were either failing or incomplete. On Windows, it simply wouldn't restore anything.

The solution, according to Arkeia was to purchase an upgrade ($12,000) which would solve all our problems. And since we refused to spend another 15% for a support agreement, that was our only alternative. I don't think so.

Needless to say, we went with someone else. Veritas had a great enterprise solution that worked with Linux and Windows (the server app runs only on Windows) and supports a huge array of tape drives. And it was one-third the price.

I can't definetly recall, but the Veritas agent also has some security peculiarities that raised some eyebrows. If you run any enterprise backup, I guess the answer is to make sure you're firewalled.

In this day and age of cheap disk drives, I wonder if anyone is using USB or Firewire drives and just using those for back-ups. A Lacie 250 gig Firewire drive is <$200.