Hackers Wipe US Servers of Email Provider VFEmail (zdnet.com)

← Back to Stories (view on slashdot.org)

Hackers Wipe US Servers of Email Provider VFEmail (zdnet.com)

Posted by msmash on Tuesday February 12, 2019 @08:50AM from the attack-and-destroy dept.

Hackers have breached the severs of email provider VFEmail.net and wiped the data from all its US servers, destroying all US customers' data in the process. From a report: The attack took place yesterday, February 11, and was detected after the company's site and webmail client went down without notice. "At this time, the attacker has formatted all the disks on every server," the company said yesterday. "Every VM is lost. Every file server is lost, every backup server is lost. This was more than a multi-password via SSH exploit, and there was no ransom. Just attack and destroy," VFEmail said. The company's staff is now working to recover user emails, but as things stand right now, all data for US customers appears to have been deleted for good and gone into /dev/null.

91 of 157 comments (clear)

Min score:

Reason:

Sort:

There were NO offsite backups????? by sconeu · 2019-02-12 08:54 · Score: 4, Insightful

No offsite backups? No tapes????
Who designed the disaster plan for these guys?

--
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
1. Re:There were NO offsite backups????? by TigerPlish · 2019-02-12 08:55 · Score: 2
  
  No offsite backups? No tapes????
  Who designed the disaster plan for these guys?
  Same geniuses as Wells Fargo?
  
  --
  The "Civilized World" jumped the shark ca. 1973.
2. Re:There were NO offsite backups????? by spudnic · 2019-02-12 08:56 · Score: 2
  
  It's all in their private cloud, of course!
  
  --
  load "linux",8,1
3. Re:There were NO offsite backups????? by leehwtsohg · 2019-02-12 08:59 · Score: 2
  
  You mean offline.
  Nothing happened to any particular location.
4. Re:There were NO offsite backups????? by lgw · 2019-02-12 09:01 · Score: 2
  
  No offsite backups? No tapes????
  Who designed the disaster plan for these guys?
  The plan was a disaster - mission complete!
  An online copy is not a backup, guys. It can be a great cache of a backup, but it's not a backup. Who still doesn't know this?
  
  --
  Socialism: a lie told by totalitarians and believed by fools.
5. Re:There were NO offsite backups????? by Anonymous Coward · 2019-02-12 09:12 · Score: 3, Interesting
  
  The business plan probably.
  If you do make backups, you are too expensive, certainly cannot compete, and will go out of business. No income for you.
  If you do not make backups, you may make a nice buck for a while before the thing explodes in your face. Hell, maybe you are lucky and it never explodes at all.
  Regardless, at least you will make money for a while. So this scenario is clearly the winner. Screw the damage to your future ex customers, that is not your problem.
6. Re:There were NO offsite backups????? by knarfling · 2019-02-12 09:18 · Score: 1
  
  A more important question is "Why were the backup servers accessible from the email servers?"
  A good network design has the backup servers isolated from the production servers. Only the ports need for backup should be allowed. Even if using a copy over SSH, it is possible to set it so the backup servers can access production, but block all access from production to the backup servers. I should never be able to gain console/terminal access on the backup servers from production.
  
  --
  Great civilizations have lived and died on false theories. Don't mess up mine with a few facts.
7. Re:There were NO offsite backups????? by Anonymous Coward · 2019-02-12 09:27 · Score: 4, Funny
  
  From a cannon. Into the sun.
8. Re:There were NO offsite backups????? by Anonymous Coward · 2019-02-12 09:39 · Score: 2, Insightful
  
  What has a higher chance of getting owned? A network accessible box wide open to the web, or a backup server that can only be accessed by SSH via a specific management VLAN?
9. Re:There were NO offsite backups????? by Headw1nd · 2019-02-12 09:42 · Score: 1
  
  It appears that the backup servers did not share any authentication and were accessed and destroyed separately.
10. Re:There were NO offsite backups????? by Anonymous Coward · 2019-02-12 09:45 · Score: 2, Funny
  
  This is why DevOps is a bad idea.
11. Re:There were NO offsite backups????? by bobbied · 2019-02-12 10:20 · Score: 2
  
  No offsite backups? No tapes????
  Who designed the disaster plan for these guys?
  No, no.. The Admin E-mailed the backups to himself every night.... They are all in his inbox... Don't worry, he encrypted them.
  
  --
  "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
12. Re:There were NO offsite backups????? by rickb928 · 2019-02-12 10:26 · Score: 3, Insightful
  
  It *is* a PITA to put a tape in your bag, open up the fireproof safe at home, throw it in, get the *correct* one out, put it in your bag, and remember the next day to put that where it needs to be. And repeat. /s
  I did that for years. And I slept a little better.
  
  --
  deleting the extra space after periods so i can stay relevant, yeah.
13. Re:There were NO offsite backups????? by nanospook · 2019-02-12 10:33 · Score: 1
  
  What? Me worry?
  
  --
  Have you fscked your local propeller head today?
14. Re:There were NO offsite backups????? by rickb928 · 2019-02-12 10:41 · Score: 3, Insightful
  
  Once you're in the front door, you're going through the system. Only offline backups can be trusted to 'be there'.
  And no offline copies of the VM environment? I think of those as especially precious. DO I want to rebuild those from scratch? Nope.
  
  --
  deleting the extra space after periods so i can stay relevant, yeah.
15. Re:There were NO offsite backups????? by rickb928 · 2019-02-12 10:42 · Score: 1
  
  If you've compromised the server, you're well on your way to all the connections. Firewalling failed, for sure.
  
  --
  deleting the extra space after periods so i can stay relevant, yeah.
16. Re:There were NO offsite backups????? by drinkypoo · 2019-02-12 12:05 · Score: 1
  
  And no offline copies of the VM environment? I think of those as especially precious. DO I want to rebuild those from scratch? Nope.
  They probably didn't build them to begin with, odds are they did it all with someone else's containers and they had no clue what was actually running on those systems.
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
17. Re:There were NO offsite backups????? by brausch · 2019-02-12 12:41 · Score: 1
  
  It also implies that a rogue employee could have done this at any time.
  I ran a large credit union IS department for years and made sure that no one person, even me, could have pulled this off. Various on-line (but in-house at local and remote site) backups done minute to minute in most cases and off-line backups done daily. Various permissions required to access electronic data stores, and different people with physical access. Tapes taken offsite every day AND MOUNTED AND READ AND VERIFIED at the remote site then stored in the custody of yet other employees. No single point of human or electronic failure.
  
  --
  "Almost every wise saying has an opposite one, no less wise, to balance it." - George Santayana
18. Re:There were NO offsite backups????? by MachineShedFred · 2019-02-12 14:07 · Score: 1
  
  You want to explain that one?
  What the fuck does this have to do with DevOps? And besides, if their DevOps guy can find his ass without a flashlight and a map, he'd have some kind of disaster recovery plan, even if it was just daily scripted snapshots of the server VMs.
  Unless of course they were running on bare metal, in which case 2002 called and wants to introduce you to a product called VMware ESX.
  
  --
  Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
19. Re:There were NO offsite backups????? by JustAnotherOldGuy · 2019-02-12 14:18 · Score: 1
  
  They probably didn't build them to begin with, odds are they did it all with someone else's containers and they had no clue what was actually running on those systems.
  ^^^^^THIS.
  Yep, they most likely took some base container, maybe modded it a bit, and threw it into service. They likely have no idea at all how it was configured or what was in it.
  People are always concerned with the data and forget about the infrastructure that it lives in.
  
  --
  Just cruising through this digital world at 33 1/3 rpm...
20. Re:There were NO offsite backups????? by rossz · 2019-02-12 19:32 · Score: 1
  
  My guess, bean-counter type management after ignoring advice from the technical people.
  
  --
  -- Will program for bandwidth
21. Re: There were NO offsite backups????? by rickb928 · 2019-02-13 00:51 · Score: 1
  
  Actually I had three clients I did off-site tape rotations for, for about 6 years. One made regular random requests for tapes to do a directory test on. The other was a bank, they did a full scan and compare. My own, the company tapes, since I was the sysadmin, I did compares quarterly. DAT and higher capacity tapes were in use, LTO and such. Never had to restore my own, don't know if the bank did, but the other client was fastidious.
  I was much lazier with my own server backups, having just software tapes updated rarely, and data less often, they were DNS, email, and web. When we were pwned by the creeps in Atlanta we rebuilt from scratch, with a boot diskette, wgets, compiling the kernel, this you could still do in the late 90s. Today it would be very different.
  And over 14 years with the company I got very lazy, testing randomly, then delegating to the #3 & 4 techs. Our clients not so much. Then though our bank client didn't have us doing any fiduciary stuff we had to participate in DR (now BCP) drills, semi annually, and that was annoying and tedious. And instructive.
  The rest of my clients we maybe verified tapes 30 days after they came on board, never again unless they asked. We would log stats and replace tapes for wear, manufacturer specs. Never had a client left without data, but no, 90% of my work I did not do regular verification. The backup software was flaky enough to force us to fix and test when it went bad.
  
  --
  deleting the extra space after periods so i can stay relevant, yeah.
22. Re: There were NO offsite backups????? by lgw · 2019-02-13 01:50 · Score: 1
  
  Any backup is a backup
  
  Semantics.
  
  A copy is a copy in the copy is a backup and it doesn't really matter where you store it as long as your whole strategy makes sense.
  Anything online, especially at the same site or accessible from the same site, is a convenience, not a backup strategy. Sure, it's handy when someone says "oops", but it doesn't protect your business.
  
  I certainly don't blame people for trying to get off tape backups, and they are kind of a huge and slow pain in the ass.
  LTO-8 is 360 MB/s assuming no compression, for a single drive. An internet pipe that give you better than 3.6 Gb/s upload is impressive for a small business.
  Cloud backups make sense when all your data is already in the cloud. You don't really have much choice at that point. Or if your data changes slowly, as you can do that first upload via physical means (e.g. snowball, or even snowmobile for PB). But at least do a daily key rotation (or a service that promises WORM) to limit the damage a malicious actor can do.
  
  --
  Socialism: a lie told by totalitarians and believed by fools.
23. Re:There were NO offsite backups????? by pnutjam · 2019-02-13 02:25 · Score: 2
  
  At the very least, you should be using a 2nd cloud service for backup, like rsync.net or those guys that are always releasing hard drive stats, backblaze.
  
  --
  Cheap storage VM.
24. Re:There were NO offsite backups????? by bartle · 2019-02-13 05:35 · Score: 1
  
  I don't know if, in this case, that's a fair criticism. VFEmail is providing a realtime service and going offline for any length of time has very serious repercussions.
  Tape backups aren't going to have the users' most recent emails and it could take days to fully restore prior emails. From the users' perspective, this is extremely inconvenient and they're probably going to take their recovered emails and go elsewhere.
  To run an online service in today's world, particularly an email host, means continual uptime. Downtime, even to recover lost data, is not a realistic option. Service providers thus need to focus their attention on redundancy at all levels. The question they failed to ask isn't, "How do we maintain quality offline backups". Instead, it should have been, "How do we prevent a single rogue employee or hacker from destroying our company?"
Backups? by byteherder · 2019-02-12 08:55 · Score: 2

Time to pull yesterday's backup tapes. You do have the tapes from yesterday, don't you?
1. Re:Backups? by Anonymous Coward · 2019-02-12 09:02 · Score: 1
  
  Plot twist: the last remaining copy of the encryption key is backed up on the encrypted backup tapes.
2. Re:Backups? by zlives · 2019-02-12 10:21 · Score: 1
  
  OH the bitcoins
3. Re:Backups? by bobbied · 2019-02-12 10:22 · Score: 2
  
  Plot twist: the last remaining copy of the encryption key is backed up on the encrypted backup tapes.
  Yea, but it's "12345".... What idiot uses THAT as a combination?
  Remind me to change the combination on my luggage..
  
  --
  "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
4. Re:Backups? by pnutjam · 2019-02-13 03:12 · Score: 1
  
  I backed up my work laptop with borg and misunderstood the password protected a key-file, not the archive....
  
  My other backup missed all my dot files....
  
  pour some out for the lost data...
  
  --
  Cheap storage VM.
5. Re:Backups? by bill_mcgonigle · 2019-02-13 04:01 · Score: 1
  
  :drinks:
  On the other hand, don't hire somebody who hasn't realized this mistake exists.
  
  --
  My God, it's Full of Source!
  OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
6. Re:Backups? by pnutjam · 2019-02-13 06:35 · Score: 1
  
  Luckily it was just my config files, I had another backup of my ssh keys.
  
  --
  Cheap storage VM.
nice!! by zlives · 2019-02-12 08:55 · Score: 2

offsite tape backup is sounding good right about now
1. Re:nice!! by bobbied · 2019-02-12 10:23 · Score: 1
  
  offsite tape backup is sounding good right about now
  Don't worry, the admin was E-mailing the backups to himself every night for safe keeping...
  
  --
  "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
You mean just the online backup servers... by SuperKendall · 2019-02-12 08:56 · Score: 4, Interesting

Every file server is lost, every backup server is lost.
So, that's the online backup servers, but what about the offline backups... there were offline backups, right? RIGHT???
I am starting to wonder if I don't need to ask every single electronic service I interact with to put in writing what tighter backup policies are. I imagine my stuff on gmail servers is safe... but that is truly only my imagination, who can say for sure even they have offline backups (that can be restored from)??

--
"There is more worth loving than we have strength to love." - Brian Jay Stanley
1. Re:You mean just the online backup servers... by jythie · 2019-02-12 08:59 · Score: 1
  
  Sounds like they are trying to restore data from something, so hopefully they had offline backups of some type.
2. Re:You mean just the online backup servers... by jythie · 2019-02-12 09:00 · Score: 2
  
  Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.
3. Re:You mean just the online backup servers... by war4peace · 2019-02-12 09:01 · Score: 1
  
  They probably don't. Not in the sense we think of, e.g. tapes.
  
  --
  ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
4. Re:You mean just the online backup servers... by b0s0z0ku · 2019-02-12 09:05 · Score: 2
  
  Also, they may only keep backups for a few days for security reasons -- i.e. they want their users to be able to "truly delete" data.
5. Re:You mean just the online backup servers... by bobbied · 2019-02-12 09:31 · Score: 4, Insightful
  
  Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.
  AND, when they check, some 70% turn out to be insufficient or not restorable. Most turn out to be nearly useless for anything but giving you a warm fuzzy feeling as you trot them off to offsite storage.
  Having a backup plan is one thing, TESTING your backup plan is the next level.... However, revising your backup plan and TESTING your backups are restorable on a regular basis is the only way to know it will work when the chips are down. IF you don't do all this work, it's NOT really backed up, regardless of how many tapes you put into storage.
  
  --
  "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
6. Re:You mean just the online backup servers... by SuperKendall · 2019-02-12 10:42 · Score: 1
  
  The only thing I would realistically worry about with Google is, what happens with a really big natural disaster that destroys one (or more) entire data centers? I know they replicate a lot, but would the realistically have everything fully replicated spatially...
  At least as someone else mentioned about POP email, I do have a local copy of all my email - I would possibly just lose some attachments, but probably nothing I cared much about anyway.
  This incident is a good time to consider that issue though, especially as I was thinking about moving to Protonmail...
  
  --
  "There is more worth loving than we have strength to love." - Brian Jay Stanley
7. Re:You mean just the online backup servers... by rickb928 · 2019-02-12 10:44 · Score: 1
  
  That's the grunt work you farm out to the intern.
  You DO have interns, right?
  
  --
  deleting the extra space after periods so i can stay relevant, yeah.
8. Re:You mean just the online backup servers... by lgw · 2019-02-12 11:00 · Score: 1
  
  Several years ago Google lost all their online storage for some chunk of users. They talked about restoring from tape, so a least once upon a time they had tape backups.
  
  --
  Socialism: a lie told by totalitarians and believed by fools.
9. Re:You mean just the online backup servers... by PrimaryConsult · 2019-02-12 11:39 · Score: 1
  
  This was how I discovered backups weren't quite working right for a specific piece of closed source software (something like Sharepoint before there was a Sharepoint). The company had gone out of business so there was no support. I was tasked with learning this piece of unsupported software so my first step was to restore the product's proprietary backup file using the instructions provided by the vendor in order to create a dev environment. After working with this environment for a few weeks I realized the restore process produced something that on the surface *looked* like it worked but was actually missing random chunks of content. After pointing this out to higher ups, I was tasked with finding a restore procedure that would work. Due to the nature of the content being an unholy mess of data stored in flat files and a database, restoring ones taken while the product was active resulted in a non-functioning product. The ultimate solution was to shut down the software daily just before the nightly file system and DB backups.
  Lessons learned:
  1) anything less than actually performing a restore and using the resulting service is insufficient backup testing
  2) if you ever encounter a web application that "closes" every night as if it were a brick and mortar business, some nonsense like this is probably why.
10. Re:You mean just the online backup servers... by MrKaos · 2019-02-12 12:23 · Score: 1
  
  >This incident is a good time to consider that issue though, especially as I was thinking about moving to Protonmail...
  I use both. I haven't checked my VFEmail account yet however I'd have no hesitation going back to them. I found their service to well put together, I paid for an account mainly because it wasn't scanning every email to serve me ads.
  I've found Protonmail to be excellent, straightforward UI which is clean and intuitive. Nothing to complain about either of these services.
  I think though if you are not backing up stuff from your email accounts you are missing the point of using these services. If you are relying on some entity that can shutdown access at any time then you shouldn't be too annoyed if you can't access your data. Your data is your responsibility after all.
  
  --
  My ism, it's full of beliefs.
11. Re:You mean just the online backup servers... by Solandri · 2019-02-12 12:43 · Score: 1
  
  The backup for some of the VoIP servers I've seen at companies was actually an identical server with the exact same hardware. It would be kept up to date by restoring the backup of the operational server onto it. This served as both a backup in case of hardware failure, and a sanity check to confirm the software backups were working and restorable,
12. Re:You mean just the online backup servers... by MachineShedFred · 2019-02-12 14:16 · Score: 1
  
  Unless they are a publicly traded corporation under Sarbanes-Oxley review.
  That's one of the things any competent audit will ask for evidence of - working backup restore.
  
  --
  Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
13. Re:You mean just the online backup servers... by JustAnotherOldGuy · 2019-02-12 14:26 · Score: 2
  
  The only thing I would realistically worry about with Google is, what happens with a really big natural disaster that destroys one (or more) entire data centers?
  
  AWS.
  AWS makes lots of copies of every damn file and scatters them all over the world in geographically different Availability Zones.
  So even if the entire us-east-2 (Ohio) AZ is blown off the map by a nuke, AND eu-north-1 (Stockholm) is also blown off the map, along with Tokyo, Sydney, and Frankfurt, your file is still floating around in us-east-1 or eu-west-3 or ap-northeast-3, etc etc, about a dozen other AZs.
  You'd need a genuine global disaster to lose files from AWS, and at that point I probably wouldn't be giving a shit about files, I'd be concentrating on food and ammo.
  
  --
  Just cruising through this digital world at 33 1/3 rpm...
14. Re:You mean just the online backup servers... by b0s0z0ku · 2019-02-12 14:34 · Score: 1
  
  Small providers catering to privacy-conscious techies may very well care. M$, Scroogle, and the rest of the big corporate scum won't care, of course.
15. Re:You mean just the online backup servers... by CSMoran · 2019-02-13 03:02 · Score: 1
  
  Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.
  Perhaps that's when the ransom request shall materialise.
  
  --
  Every end has half a stick.
16. Re:You mean just the online backup servers... by pnutjam · 2019-02-13 03:22 · Score: 1
  
  or a compromised account...
  
  --
  Cheap storage VM.
Re:Talking Heads... by jfdavis668 · 2019-02-12 09:00 · Score: 1

Love that song.
IMAP/POP3 provider... by b0s0z0ku · 2019-02-12 09:00 · Score: 4, Interesting

Thankfully, VFEmail was primarily an IMAP/POP3 provider. I suspect that the majority of its users had a local backup in the form of an email client with a local store...
1. Re:IMAP/POP3 provider... by chiefcrash · 2019-02-12 09:36 · Score: 3, Informative
  
  Which, hopefully they've been paying attention: the current state of recovery means if you reconnect your client to your new mailbox, all your local mail will be lost (according to an update on their website)
  
  --
  Show me on the 1st Amendment bobblehead where the moderator touched you...
2. Re:IMAP/POP3 provider... by b0s0z0ku · 2019-02-12 14:35 · Score: 1
  
  That's generally not true; most mail clients flag mail as deleted in the local archive. You have to run a "compact" on the archive in order to totally delete "deleted" messages.
No backup can be a feature by b0s0z0ku · 2019-02-12 09:03 · Score: 4, Interesting

That can be both a bug and a feature. No backups mean that there's no cache of deleted emails. Some users may want the ability to truly delete data, not have it able to "appear" due to legal proceedings 5 years from now.
I'd say it's on the users to back up their email using a client that locally caches IMAP folders or downloads via POP3.
1. Re:No backup can be a feature by Aighearach · 2019-02-12 10:09 · Score: 3, Insightful
  
  It would seem more practical to just limit the stored backups to the last n copies, like you do with rotated log files.
  If it can only come back for two weeks or something, that is sufficient for most use cases.
2. Re:No backup can be a feature by rickb928 · 2019-02-12 10:29 · Score: 1
  
  I don't really back up my mail. I copy one mailbox to two others via IMAP and POP3. Another mailbox I copy via IMAP. And then I also have a copy in my beloved (/s) Microsoft Mail. It's not really a backup, I think, just copies. And I have a glorious spam library dating back to the 90s in some of it, just too lazy to clean it up.
  If you remember spam from the 90s, you know why some of it I've had to delete.
  
  --
  deleting the extra space after periods so i can stay relevant, yeah.
3. Re: No backup can be a feature by bev_tech_rob · 2019-02-13 09:11 · Score: 1
  
  I believe that if you publicly state your email retention policy (at least in the States), then anything outside of that time period makes you S.O.L. in regards to legal discovery. At least that is how it was explained to me at the last company I worked for that was in the Healthcare industry and was always getting legal discovery requests...
  
  --
  You're messin' with my Zen Thing, man.....
Sounds like a cleanup operation by misnohmer · 2019-02-12 09:09 · Score: 4, Interesting

Maybe someone needed an email to disappear to avoid public embarrassment or legal trouble.
1. Re:Sounds like a cleanup operation by Anonymous Coward · 2019-02-12 11:14 · Score: 2, Insightful
  
  If you're in a tight enough spot that you need to contact some hackers to annihilate an email company then you also probably don't have the time to wait around while they figure out if they can even get into that email company to do the job.
  So:
  1. They were already in and held the sword of Damocles over this company's head for a long time without them even knowing it just waiting for someone to fork over enough money to make it worth their while to let the sword fall
  or
  2. They had help from an insider employee-- which would have had to be tunneled in way beforehand (which costs much more than simply leaving the backdoor of Damocles in place)
  or
  3. There were no hackers, it was done by people at the company itself (i.e. CEO trying to evade indictment for insider trading or something)
  or
  4. Some kind of psychopathic rehearsal for a real cyber war? (no skin off, say, the Chinese Communist Party's nose if some Western email company gets splattered)
  The real news here isn't that the company got whacked, but the whacking itself.
  What purpose?
  No ransom demand?
  Why wipe all the servers instead of simply continuing to harvest data from them?
  It's like a drug cartel nuking a city without warning. Cartels are in the business of making money. A glassed crater doesn't yield an income.
  So as the parent post points out: there's something going on here besides some mean ole hacker-dashery.
2. Re:Sounds like a cleanup operation by MachineShedFred · 2019-02-12 14:20 · Score: 1
  
  Hillary? Is that you?
  
  --
  Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
3. Re:Sounds like a cleanup operation by bill_mcgonigle · 2019-02-13 07:35 · Score: 1
  
  Coincident with the Muller cover-up news? This is an easy job for the spooks who hoard zero-days.
  
  --
  My God, it's Full of Source!
  OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Re:Talking Heads... by slickwillie · 2019-02-12 09:09 · Score: 1

It's Life During Wartime, piker.
Pull not push for backups. by unkmar · 2019-02-12 09:20 · Score: 2, Insightful

First onsite backup
Second offsite backup that pulls, not pushes.
- A push backup leaves a trace that there is a backup and to where it is being pushed.
- - Just track the push and wipeout the backup as well.
- A pull backup is only visible from the pulling location and, anyone inside that knows it exists.
- - No trail to trace and wipeout. If it is wiped out, Then it is clearly an inside job.
- - A pulling backup does mean the pulling system has access to the onsite backups.
- - - But the onsite backup can be isolated from the onsite system and data.
Conclusion:
- Onsite hack can wipeout onsite system and data and onsite backup. but not offsite backup.
- Offsite hack can wipeout onsite backup and offsite backup, but not onsite system and data.
- Internal knowledge required to hit both targets.
1. Re:Pull not push for backups. by pnutjam · 2019-02-13 03:25 · Score: 1
  
  Unless your cleaning your logs, you'll see a pull backup authenticating to the system. This also means you have to trust your backup endpoint, I avoid pull backups for this reason, but a proper backup account shouldn't have access to trash your backup destination.
  
  --
  Cheap storage VM.
2. Re:Pull not push for backups. by Joey+Vegetables · 2019-02-13 05:32 · Score: 1
  
  Not a terrible strategy. Mine (for a personal system with around 500GB of important data) is to rotate four backup devices - external HDDs, not tapes - daily (kept on my person), weekly (to office), monthly and yearly (both to safe deposit box near me). I also have copies of the most important files (kids' pics and videos mainly) with my in-laws in Europe. The backups are encrypted before being put on the external drives. I'm aware of a few flaws that I am working to address. (a) I'd have limited defenses against an attack that would change or delete small numbers of files . . will eventually store with the backups a manifest with a MD5 or similar hash, and compare them on every backup such that changed or deleted files in folders that shouldn't happen would trigger a warning. (b) Rubber-hose decryption would work just fine, although I don't think there's much of anything that people would go to that length to get (I'm not nearly interesting enough for that to be the case). (c) Some of my data would be lost in the event of an attack that simultaneously wiped out (a) home, (b) office, and (c) safe deposit box, all of which are within 6 miles of one another. A nuke or EMP detonated directly above Cleveland would probably do that. But as noted above, if all those things happened, I think my family and I would be facing much bigger problems, presuming we were even still around.
  
  --
  
  Nonaggression works!
Backup Architecture by Anonymous Coward · 2019-02-12 09:23 · Score: 1

Trivial, the right Backup Architecture is to have online backup that is done via something like remote btrfs snapshots (for zfs snapshots), and have those servers be secure. But, this does raise the interesting question, how do you know your appliance is secure? No patches in 20 years, and proven to be correct, with 30% market penetration or more... that might do it.
Frankly, I surprised we don't hear more of this type of total wipe more often. Makes for a great test case for the backup strategies that companies use, to see if they can withstand a bad actor.
1. Re:Backup Architecture by Aighearach · 2019-02-12 10:19 · Score: 1
  
  If you backup data instead of backing up the disks then it shouldn't be that hard to have append-only backups with very limited access permissions.
  Then it also is pretty easy to do incremental offline backups of the changed data.
I wonder by renegade600 · 2019-02-12 09:27 · Score: 1

I wonder which government officials used them.
1. Re:I wonder by mejustme · 2019-02-12 10:10 · Score: 1
  
  I wonder which government officials used them.
  Hillary, of course.
Doesn't sound like "hackers" by eneville · 2019-02-12 09:34 · Score: 1

This sounds a lot like an internal job, more than external attack. Why risk getting logged on the way in, unless you are a disgruntled employee or competitor. Most likely an employee with unfavorable bonus.

--
Why UNIX?
1. Re:Doesn't sound like "hackers" by Aighearach · 2019-02-12 10:21 · Score: 1
  
  It could easily happen if their sysadmins suck, everything is put together by hand, and somebody cracked the backup server. The backup server might have access to everything.
2. Re:Doesn't sound like "hackers" by pnutjam · 2019-02-13 03:27 · Score: 1
  
  pen-tester screw-up...? oops
  
  --
  Cheap storage VM.
Re:Physical access? by chiefcrash · 2019-02-12 09:34 · Score: 1

Why did they "obviously" gain physical access?

Off-site backups can be accessed without physical access if it was designed poorly, and there's no reason to assume they had off-line backups...

--
Show me on the 1st Amendment bobblehead where the moderator touched you...
Re:Physical access? by bobbied · 2019-02-12 09:40 · Score: 1

Backups are quite often useless and offline backups are usually weeks if not months old and take many hours to restore. Some 70% of "backups" turn out to be broken in some way or another, including not actually backing up the right data, not backing up data that's in a restorable format, and when compressed (as is often done) has unrecoverable bit errors or dropouts that render the whole backup set as good as empty.
Why? Few folks take the time to do backups right, verify they can read the data off the media clean, verify they got the right data and verify they know how to restore it. Even fewer regularly review their backups to keep up with the ever changing system configurations, including doing all the testing outlined before so MOST backups are junk.

--
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Demonstration attack by Rick+Schumann · 2019-02-12 09:41 · Score: 1

Sounds like some hacker(s) needed to demonstrate their operational efficacy to potential clients. Either that or just some too-edgy vandal wanted to burn something to the ground. Small probability: someone needed something specific wiped and needed there to be no fingerprints left behind.
Replication != Backup by bodog · 2019-02-12 09:47 · Score: 3, Insightful

Looks like ZFS replication may have been their backup plan? https://www.vfemail.net/design...
Recall an email by casualgeek · 2019-02-12 09:48 · Score: 1

That's a terrible way to recall an email.
No backup - no pity by gweihir · 2019-02-12 09:50 · Score: 1

Seriously, what are these people doing?

--
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Even MST3K knows... by jddj · 2019-02-12 09:51 · Score: 1

"Keep Circulating the Tapes!"
Just recover it? by DontBeAMoran · 2019-02-12 10:23 · Score: 2

...as things stand right now, all data for US customers appears to have been deleted for good...
Damn, talk about annoying.

...and gone into /dev/null.
Oh! So they do know where the data ended up. Just restore it! You know, like in the movies?

--
#DeleteFacebook
1. Re:Just recover it? by sacrilicious · 2019-02-12 12:43 · Score: 1
  
  So they do know where the data ended up. Just restore it! You know, like in the movies?
  "Computer: enhance!"
  
  --
  - First they ignore you, then they laugh at you, then ???, then profit.
Re:Criminal Incompetence by DontBeAMoran · 2019-02-12 10:24 · Score: 1

It depends on the definition of "criminal".

--
#DeleteFacebook
Not up to Starfleet standards by DontBeAMoran · 2019-02-12 10:25 · Score: 1

No secondary backups? Talk about amateurs.

--
#DeleteFacebook
Just do a restore from Wikileaks. by jfdavis668 · 2019-02-12 12:02 · Score: 2

I'm sure they have a recent copy.
No backups? Really? by JustAnotherOldGuy · 2019-02-12 14:14 · Score: 1

So they have no current backups at all? Seriously?
It's so easy to do these days that there's no good excuse not to. Hell, use a secured AWS bucket and stash your backups there.

--
Just cruising through this digital world at 33 1/3 rpm...
So, nobody does backups anymore? by roc97007 · 2019-02-12 15:12 · Score: 1

Or, they do backups, but keep all the copies online? For an app connected to the raw internet? And someone thought this was a good idea?

--
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
Not a "hacker," A disgruntled insider. by bill.pev · 2019-02-12 16:29 · Score: 1

This reeks of internal job. Complete and total devastation with no apparent purpose? Its too comprehensive to be an advanced script kiddie or random attack and therefor also too good to be anything without purpose. But there is no apparent purpose, so it must be an inside job. The offline tapes were probably deleted too, and that requires very skillful cracking indeed!
FAQ is a lie! by Anonymous Coward · 2019-02-12 21:51 · Score: 1

From the FAQ
> What is your backup strategy / data retention policy?
> VFEmail feels it's important to provide a long-term, stable, environment for our users. In that effort, we perform nightly backups to an offsite host from all on-site and off-site mail storage locations. This backup runs at 12am CST (-0600) and contains all user data.
> 3rd party storage of user data is generally not wanted by privacy-conscious users. If you fall into that category, you will want to use POP3 and download your mail daily. Our backup is on a daily/weekly rotation, initiated by a snapshot. If you do recieve mail between your last POP and the snapshot at 12am, it will exist on backup for a week - unless it's on Saturday night, then it's a year. You should set your POP program to download every 5-10 minutes in order to avoid having your mail caught on backup.
Hahaha
Slogan : Making email safe for the masses! by grumpy-cowboy · 2019-02-13 00:24 · Score: 1

It's so safe that now even NSA, FBI, ... cannot have access to it! Nice job!

--
Will $CURRENT_YEAR be the year of the Linux Desktop?