Slashdot Mirror
Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider (bleepingcomputer.com)
An anonymous reader quotes BleepingComputer: Verelox, a provider of dedicated KVM and VPS servers based in The Hague, Netherlands, suffered a catastrophic outage after a former administrator deleted all customer data and wiped most of the company's servers. Details of what exactly happened aren't available, but according to posts on various web hosting forums [1, 2, 3], the incident appears to have taken place Thursday, when users couldn't access their servers or the company's website.
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."
Nobody with a brain stores important data on someone elses server.
Seems like a glitch in the matrix
This is why you change the passwords when an employee leaves. Hey, I'm smarter than the guys that work at Verelox!
Hire me, you dumb motherfuckers!!
Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.
Did they not remove the ex-admin's credentials, or what? I mean, regardless of how the ex-admin gained access to the data to wipe it, it's a crime. But I'd like to know if Verelox was stupid enough to not remove his credentials, or this happened some other way. (Like, did he do this his last day of work as a "fuck you" to management for firing him?)
Mr. Hu is not a ninja.
At least these two stories are from different perspectives: https://m.slashdot.org/story/3...
The story stays the same - don't fuck over your admins and have proper procedure and backup.
Custom electronics and digital signage for your business: www.evcircuits.com
Why no secure backups? Idiots.
Only if it involved sabotaging US nuclear weapons and then selling state-of-the-art ABM technology to the Russians, North Koreans, and the Iranians. With discounts to ISIS, the PLO, Al Qaeda, and Boko Haram. Just to be thorough about screwing you all for a perceived insult.
Executives also read the press release, though. The mighty Cloud was supposed to mean much easier administration so we didn't need to handle most IT matters in-house.
In actual $$$ terms, at both the low end and the high end the Cloud often works out more expensive than self-hosting, often by quite a wide margin. There's a zone in between where that doesn't always seem to be the case, but I'm not sure how wide it really is, and it's usually based on TCO rather than the hardware and connectivity expenses alone. The thing is, it turns out that you can't just delegate all responsibility and get good results.
How many times has a significant chunk of the Internet gone off-line because a major AWS data centre was knocked out for a while? Sure, all those beautifully-architected, Cloud-hosted web apps could have just switched over to a standby -- the AWS infrastructure would have supported that in various ways -- but it turns out that you still need enough expertise to understand how your infrastructure works, Cloud or no, or you just moved your single points of failure/vulnerability to another building. And of course the same considerations apply to all the other big Cloud hosting systems as well, as well as to simpler hosting like your favourite blogging platform or storing your code on GitHub.
Likewise in terms of finances, how often has someone who didn't fully understand the implications of a Cloud system or even just off-site hosting been hit with a huge bill they didn't expect, because the pricing model wasn't clear and they didn't really know what they were paying for and how much to budget?
There still ain't no such thing as a free lunch, and outsourcing critical business functions still ain't a silver bullet.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Love and coddle your Admin -- or else!
and this is obviously one of them. Criminals come from all walks of life, sysadmin isn't a position immune to containing the occasional bad apple.
So many questions of course, a lot of which boil down to "They must have had some serious lapse in procedure to have allowed this to happen." That's not really the case though. Back doors and logic bombs are serious threats when a person has been a trusted system administrator. Done "right", they can be extremely difficult to detect. It's a bit like the widely accepted advice of "Server was hacked? Don't try to clean it, you might miss something. You must wipe and reinstall it." (same really applies even to a home desktop) A departing admin (on bad OR good terms) is basically the exact same issue, a compromised system, but we only very rarely see such an extreme response. It's much less practical to nuke-n-pave when it's your entire network that is basically now classified as "compromised." Is this how we should respond? When you really stop and think about it, it starts to show itself as a really difficult question to answer. Rebuilding everything when an admin leaves when your system is large is just really hard to justify. But if your system is big, it's also more difficult to review it all and proclaim it "clean". It's just a bad position to be in, and that's why admin departures are such a headache. If you're big enough you have several admins and better compartmentalization of access, more robust isolation of systems, better logging, security software that's under the control of the CIO and not the admins, etc. They have a better chance, but it doesn't look like this one was big enough to have those benefits.
The lack of backups is the most troubling though. That's what stung the other recent post on the cleanout-from-inside. There's just no excuse for that.
I work for the Department of Redundancy Department.
If at all possible, don't blindly trust your admins. Always keep several backup plans in mind.
I can see that jerks smug face while he sits in prison for the next few years, oh the hours of inconvenience he caused! well worth it my friend, well worth it.
I was with a one-man ISP for 12 years, starting off with a dial-up UNIX account to hosting my websites. Unfortunately, in 2007, the two lines from different providers to the out of state data center got cut in separate backhoe accidents (what are the odds?). It took ten days for the providers to restore the lines and him to get a third line with a different provider installed. I've already moved my websites to a dedicated hosting provider by day eight. I haven't suffered an outage since then.
The ex-admin allegedly deleted all the data... Until is has been thoroughly investigated and it can be proven, the company has made a potentially libelous statement. I don't know how defamation laws work in Europe but no semi-competent General Counsel would not have let a US corporation make such a stupid statement in a press release.
Somebody could have used the ex-admin credentials, an external bad actor or someone within the company looking to cover something up. The company may very well be attempting to pull off an elaborate insurance fraud claim.
The thing is that it seems they haven't lost any data. I bet the cloud did no harm.
Yes they did. Almost everything is recovered.
Can't recover? What did he do, dd if=/dev/zero of=/dev/fs ? Or were they using something like NTFS? Or most likely: storing the data in the CLOUD.
Pretty well every linux filesystem has recovery tools. There's a reason the POSIX term for "delete file" is "unlink". Because you aren't clearing the data, you're just unlinking from the table.
Since pretty well every file has a MAGIC at its start, it becomes fairly dooable to recover.
A lot of service level agreements say provider isn't responsible for lost data. Managed VPS with backup is generally a premium.
In a certain sort of movie (e.g. Mad Max, The Crow) the difference between good guys and bad guys is the order in which they commit their atrocities. In these two stories, good guys delete the data and then get fired, bad guys get fired and then delete the data.
Quattuor res in hoc mundo sanctae sunt: libri, liberi, libertas et liberalitas.
Libel laws are generally much stronger in Europe compared to the US. So yet another Anonymous idiot writing crap. So surprising...
Oh no - the BOFH would have deleted the (rather, Yet Another) Pointy-Haired Boss or put them into the long term backup (lime pits) long before it reached the point of being an ex-Admin.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
https://developers.slashdot.or...