Slashdot Mirror
Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider (bleepingcomputer.com)
An anonymous reader quotes BleepingComputer: Verelox, a provider of dedicated KVM and VPS servers based in The Hague, Netherlands, suffered a catastrophic outage after a former administrator deleted all customer data and wiped most of the company's servers. Details of what exactly happened aren't available, but according to posts on various web hosting forums [1, 2, 3], the incident appears to have taken place Thursday, when users couldn't access their servers or the company's website.
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.
Did they not remove the ex-admin's credentials, or what? I mean, regardless of how the ex-admin gained access to the data to wipe it, it's a crime. But I'd like to know if Verelox was stupid enough to not remove his credentials, or this happened some other way. (Like, did he do this his last day of work as a "fuck you" to management for firing him?)
Mr. Hu is not a ninja.
Hey dumb motherfucker, ever heard of a logic bomb? Or backdoors? If this guy went and deleted everything, what exactly makes you think that he didn't also backdoor everything or planted a logic bomb to delete it all.
Sounds like you are just as stupid as the guys who work at Verelox who think that just removing a account/passwords solves all security issues related to firing a sysadmin.
Firing a sysadmin is perhaps one of the most dangerous things a company can do.
At least these two stories are from different perspectives: https://m.slashdot.org/story/3...
The story stays the same - don't fuck over your admins and have proper procedure and backup.
Custom electronics and digital signage for your business: www.evcircuits.com
Nobody with a brain stores important data on someone elses server.
...without a backup.
Just cruising through this digital world at 33 1/3 rpm...
Why no secure backups?...
The article(s) seem to indicate that most, but not all, customer data can be recovered. So it seems there were working backups. But in a hosting environment, not everything is backed up continuously, and that may be where some of the data will be lost.
Nobody with a brain stores important data on someone elses server.
...without a backup.
You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt. The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house.
I know I can do it in an hour because when Digital Ocean was having trouble at one of their data centers a few years back, I spun up new VMs and migrated everything to another data center.
There's no place like
You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt. The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house.
You're the kind of guy I'd want running my IT department.
Just cruising through this digital world at 33 1/3 rpm...
Executives also read the press release, though. The mighty Cloud was supposed to mean much easier administration so we didn't need to handle most IT matters in-house.
In actual $$$ terms, at both the low end and the high end the Cloud often works out more expensive than self-hosting, often by quite a wide margin. There's a zone in between where that doesn't always seem to be the case, but I'm not sure how wide it really is, and it's usually based on TCO rather than the hardware and connectivity expenses alone. The thing is, it turns out that you can't just delegate all responsibility and get good results.
How many times has a significant chunk of the Internet gone off-line because a major AWS data centre was knocked out for a while? Sure, all those beautifully-architected, Cloud-hosted web apps could have just switched over to a standby -- the AWS infrastructure would have supported that in various ways -- but it turns out that you still need enough expertise to understand how your infrastructure works, Cloud or no, or you just moved your single points of failure/vulnerability to another building. And of course the same considerations apply to all the other big Cloud hosting systems as well, as well as to simpler hosting like your favourite blogging platform or storing your code on GitHub.
Likewise in terms of finances, how often has someone who didn't fully understand the implications of a Cloud system or even just off-site hosting been hit with a huge bill they didn't expect, because the pricing model wasn't clear and they didn't really know what they were paying for and how much to budget?
There still ain't no such thing as a free lunch, and outsourcing critical business functions still ain't a silver bullet.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Love and coddle your Admin -- or else!
and this is obviously one of them. Criminals come from all walks of life, sysadmin isn't a position immune to containing the occasional bad apple.
So many questions of course, a lot of which boil down to "They must have had some serious lapse in procedure to have allowed this to happen." That's not really the case though. Back doors and logic bombs are serious threats when a person has been a trusted system administrator. Done "right", they can be extremely difficult to detect. It's a bit like the widely accepted advice of "Server was hacked? Don't try to clean it, you might miss something. You must wipe and reinstall it." (same really applies even to a home desktop) A departing admin (on bad OR good terms) is basically the exact same issue, a compromised system, but we only very rarely see such an extreme response. It's much less practical to nuke-n-pave when it's your entire network that is basically now classified as "compromised." Is this how we should respond? When you really stop and think about it, it starts to show itself as a really difficult question to answer. Rebuilding everything when an admin leaves when your system is large is just really hard to justify. But if your system is big, it's also more difficult to review it all and proclaim it "clean". It's just a bad position to be in, and that's why admin departures are such a headache. If you're big enough you have several admins and better compartmentalization of access, more robust isolation of systems, better logging, security software that's under the control of the CIO and not the admins, etc. They have a better chance, but it doesn't look like this one was big enough to have those benefits.
The lack of backups is the most troubling though. That's what stung the other recent post on the cleanout-from-inside. There's just no excuse for that.
I work for the Department of Redundancy Department.
I wouldn't hire a guy who copies all my data to his house.
An hour, if you happen to be awake and available.
I wouldn't hire a guy who copies all my data to his house.
Funny, it's data from *my* company. I'm the guy who *owns* the data. So why shouldn't back a copy up to my 12 TB storage array at my house?
If I worked for *your* company, I would back it up wherever *you* wanted it.
There's no place like
The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house..
So, you only have one backup at one place? You're flirting with desaster.
Could you explain "salt?" It's new to me but from your context I should know it. (A link would be fine).
Yup--as others posted below, 'salt stack'. It's pretty much like 'Puppet', 'Chef', or 'Ansible'. Set up a 'salt mater' and point all your 'salt minions' to the master. Then you can define the way you want your systems to be configured from the master. i.e. things like disabling SSH password auth, deploying authorized SSH keys, configuring firewalls, cron jobs, packages installed, etc...
There's no place like
without a backup
Actually, when I worked at a.... major transportation organization, I once accidentally deleted the entire database. It wasn't my fault, my code worked and was tested through dev -> stage -> test and all that.. but at the last minute my boss was like "Hey, you didn't use this cutting edge new ORM technique, refactor now" WHILE I WAS PATCHING IT TO PRODUCTION!!!!! So I bowed my head and said "yessir..." Well, what was supposed to delete one record ended up cascading to every related model and.... BAM.
I won't go into too many details because it didn't get reported, but basically a lot of things were moving and all record of them disappeared. And that major infrastructure system didn't have proper backups.. Ended up pulling the database copy from test (which was, at the time, 2 days old) and restoring from that, within about 2 hours.
We did a good job shifting blame that it was somehow everyone else's fault, and I didn't get in trouble (because, like I said, my code worked. Bossman made it break with his "strongly suggested" refactoring). After that, I was never asked to refactor in such silly ways again.
Oh, and if you thought we had learned our lesson and started doing backups after that... you must not have worked in IT long :)
there isn't a single point of failure
We pay handsomely
Are you hiring?
So it is enough to blow up your house.
Only if you didn't read and understand the entire comment.
There's no place like
"Guys", yes. "Guy", no. Never.