Slashdot Mirror
Hackers Wipe US Servers of Email Provider VFEmail (zdnet.com)
Hackers have breached the severs of email provider VFEmail.net and wiped the data from all its US servers, destroying all US customers' data in the process. From a report: The attack took place yesterday, February 11, and was detected after the company's site and webmail client went down without notice. "At this time, the attacker has formatted all the disks on every server," the company said yesterday. "Every VM is lost. Every file server is lost, every backup server is lost. This was more than a multi-password via SSH exploit, and there was no ransom. Just attack and destroy," VFEmail said. The company's staff is now working to recover user emails, but as things stand right now, all data for US customers appears to have been deleted for good and gone into /dev/null.
No offsite backups? No tapes????
Who designed the disaster plan for these guys?
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
i deciphered the code
Time to pull yesterday's backup tapes. You do have the tapes from yesterday, don't you?
offsite tape backup is sounding good right about now
Every file server is lost, every backup server is lost.
So, that's the online backup servers, but what about the offline backups... there were offline backups, right? RIGHT???
I am starting to wonder if I don't need to ask every single electronic service I interact with to put in writing what tighter backup policies are. I imagine my stuff on gmail servers is safe... but that is truly only my imagination, who can say for sure even they have offline backups (that can be restored from)??
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Life in wartime.
How could it not be an inside (or former insider) job.
OK, so obviously these "hackers" gained physical access to everything, including off-site and off-line archives / backups? That doesn't sound like a hack, it sounds like an inside job. Either that or they didn't have off-sites, but what kind of idiots would run a company that way?
Thankfully, VFEmail was primarily an IMAP/POP3 provider. I suspect that the majority of its users had a local backup in the form of an email client with a local store...
n/t
That can be both a bug and a feature. No backups mean that there's no cache of deleted emails. Some users may want the ability to truly delete data, not have it able to "appear" due to legal proceedings 5 years from now.
I'd say it's on the users to back up their email using a client that locally caches IMAP folders or downloads via POP3.
Was it an attack by TrumPutin? Is this proof of Russian Collusion!?!?!?!!!!!!!!!
Maybe someone needed an email to disappear to avoid public embarrassment or legal trouble.
Where somebody else's fuckups can destroy my mail! Yay!
Oh, wait, I actually don't use webmail, and their fuckups can only destroy those handful of messages that might be in flight at the moment.
Never mind.
First onsite backup
Second offsite backup that pulls, not pushes.
- A push backup leaves a trace that there is a backup and to where it is being pushed.
- - Just track the push and wipeout the backup as well.
- A pull backup is only visible from the pulling location and, anyone inside that knows it exists.
- - No trail to trace and wipeout. If it is wiped out, Then it is clearly an inside job.
- - A pulling backup does mean the pulling system has access to the onsite backups.
- - - But the onsite backup can be isolated from the onsite system and data.
Conclusion:
- Onsite hack can wipeout onsite system and data and onsite backup. but not offsite backup.
- Offsite hack can wipeout onsite backup and offsite backup, but not onsite system and data.
- Internal knowledge required to hit both targets.
Trivial, the right Backup Architecture is to have online backup that is done via something like remote btrfs snapshots (for zfs snapshots), and have those servers be secure. But, this does raise the interesting question, how do you know your appliance is secure? No patches in 20 years, and proven to be correct, with 30% market penetration or more... that might do it.
Frankly, I surprised we don't hear more of this type of total wipe more often. Makes for a great test case for the backup strategies that companies use, to see if they can withstand a bad actor.
No way that happened the way they're saying.
I wonder which government officials used them.
This sounds a lot like an internal job, more than external attack. Why risk getting logged on the way in, unless you are a disgruntled employee or competitor. Most likely an employee with unfavorable bonus.
Why UNIX?
Sounds like some hacker(s) needed to demonstrate their operational efficacy to potential clients. Either that or just some too-edgy vandal wanted to burn something to the ground. Small probability: someone needed something specific wiped and needed there to be no fingerprints left behind.
Looks like ZFS replication may have been their backup plan? https://www.vfemail.net/design...
That's a terrible way to recall an email.
I wonder whose emails were on one of the customer accounts?
Seriously, what are these people doing?
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
"Keep Circulating the Tapes!"
I see this service has a paid service which I would hate to have paid for and then loose all my email. But its interesting they got into backup servers as well as primary. Sounds like a poorly setup if the hackers to gain access to both.
The article says the disks were reformatted. If that's true, something like testdisk will be able to recover everything in no time. So I'll assume their IT doesn't know what they're doing or the hacker wiped everything instead of reformatted everything. I guess another possibility is they're on a shared hosting system and other hosts instantly gobbled up the free HDD space, except reformatting should have reused the same space so that option isn't valid.
Seriously, at what point does incompetence become criminal negligence?
Damn, talk about annoying.
Oh! So they do know where the data ended up. Just restore it! You know, like in the movies?
#DeleteFacebook
No secondary backups? Talk about amateurs.
#DeleteFacebook
someone had compromising emails and needed to make sure they were deleted... So might as well bring down the entire service.
So clever...
I'm sure they have a recent copy.
Webmail places everything in one spot to be lost, or become inaccessible. Glad I still download copies via POP3.
So they have no current backups at all? Seriously?
It's so easy to do these days that there's no good excuse not to. Hell, use a secured AWS bucket and stash your backups there.
Just cruising through this digital world at 33 1/3 rpm...
Or, they do backups, but keep all the copies online? For an app connected to the raw internet? And someone thought this was a good idea?
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
I just canâ(TM)t help but have that horrible feeling that attacks like this are going to become more common with devastation of similar or worse magnitude.
This reeks of internal job. Complete and total devastation with no apparent purpose? Its too comprehensive to be an advanced script kiddie or random attack and therefor also too good to be anything without purpose. But there is no apparent purpose, so it must be an inside job. The offline tapes were probably deleted too, and that requires very skillful cracking indeed!
From the FAQ
> What is your backup strategy / data retention policy?
> VFEmail feels it's important to provide a long-term, stable, environment for our users. In that effort, we perform nightly backups to an offsite host from all on-site and off-site mail storage locations. This backup runs at 12am CST (-0600) and contains all user data.
> 3rd party storage of user data is generally not wanted by privacy-conscious users. If you fall into that category, you will want to use POP3 and download your mail daily. Our backup is on a daily/weekly rotation, initiated by a snapshot. If you do recieve mail between your last POP and the snapshot at 12am, it will exist on backup for a week - unless it's on Saturday night, then it's a year. You should set your POP program to download every 5-10 minutes in order to avoid having your mail caught on backup.
Hahaha
I find that hard to believe! Normally a significant part of even a slack it department is managing backup tapes. It's not unusual to keep a year's worth of backup tapes, but in their case they would probably just need a day or two.
part of managing a backup tape system is periodically checking the actual date on the tapes!
It's so safe that now even NSA, FBI, ... cannot have access to it! Nice job!
Will $CURRENT_YEAR be the year of the Linux Desktop?
Write only storage is for losers!
ehyup, jist wut weh nehd, mar uv ar data in someone ehlses cuntrole. oh, and JERBS.