Slashdot Mirror
Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider (bleepingcomputer.com)
An anonymous reader quotes BleepingComputer: Verelox, a provider of dedicated KVM and VPS servers based in The Hague, Netherlands, suffered a catastrophic outage after a former administrator deleted all customer data and wiped most of the company's servers. Details of what exactly happened aren't available, but according to posts on various web hosting forums [1, 2, 3], the incident appears to have taken place Thursday, when users couldn't access their servers or the company's website.
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.
Did they not remove the ex-admin's credentials, or what?
They should... but if you're sitting with the keys to the kingdom you might have the domain administrator account password, root passwords, various service accounts set up for particular purposes including but not limited to integration with external access... Yes, all could be done with the proper procedures in place. But very often the responsible for such IT procedures is the admin and the admin is the one keeping tabs on what everyone else has access to. Plus you often have the rights to create undocumented loopholes that you might reasonably excuse as being a test account and an oversight if discovered. Not to mention the setting you'd bring this up, either you're basically questioning the loyalty of one of the most trusted men in the system or it looks like you're setting him up to be fired.
Live today, because you never know what tomorrow brings
At least these two stories are from different perspectives: https://m.slashdot.org/story/3...
The story stays the same - don't fuck over your admins and have proper procedure and backup.
Custom electronics and digital signage for your business: www.evcircuits.com
Nobody with a brain stores important data on someone elses server.
...without a backup.
Just cruising through this digital world at 33 1/3 rpm...
Why no secure backups?...
The article(s) seem to indicate that most, but not all, customer data can be recovered. So it seems there were working backups. But in a hosting environment, not everything is backed up continuously, and that may be where some of the data will be lost.
and this is obviously one of them. Criminals come from all walks of life, sysadmin isn't a position immune to containing the occasional bad apple.
So many questions of course, a lot of which boil down to "They must have had some serious lapse in procedure to have allowed this to happen." That's not really the case though. Back doors and logic bombs are serious threats when a person has been a trusted system administrator. Done "right", they can be extremely difficult to detect. It's a bit like the widely accepted advice of "Server was hacked? Don't try to clean it, you might miss something. You must wipe and reinstall it." (same really applies even to a home desktop) A departing admin (on bad OR good terms) is basically the exact same issue, a compromised system, but we only very rarely see such an extreme response. It's much less practical to nuke-n-pave when it's your entire network that is basically now classified as "compromised." Is this how we should respond? When you really stop and think about it, it starts to show itself as a really difficult question to answer. Rebuilding everything when an admin leaves when your system is large is just really hard to justify. But if your system is big, it's also more difficult to review it all and proclaim it "clean". It's just a bad position to be in, and that's why admin departures are such a headache. If you're big enough you have several admins and better compartmentalization of access, more robust isolation of systems, better logging, security software that's under the control of the CIO and not the admins, etc. They have a better chance, but it doesn't look like this one was big enough to have those benefits.
The lack of backups is the most troubling though. That's what stung the other recent post on the cleanout-from-inside. There's just no excuse for that.
I work for the Department of Redundancy Department.
I wouldn't hire a guy who copies all my data to his house.