Ex-Admin Deletes All Customer Data and Wipes Servers of Dutch Hosting Provider (bleepingcomputer.com)

An anonymous reader quotes BleepingComputer: Verelox, a provider of dedicated KVM and VPS servers based in The Hague, Netherlands, suffered a catastrophic outage after a former administrator deleted all customer data and wiped most of the company's servers. Details of what exactly happened aren't available, but according to posts on various web hosting forums [1, 2, 3], the incident appears to have taken place Thursday, when users couldn't access their servers or the company's website.

Verelox's homepage came back online earlier Friday, but the website was plastered with a grim message informing users of the ex-admin's actions. Following the incident, the hosting provider decided to take the rest of its network offline and focus on recovering customer data. Verelox staff don't believe they can recover all data.
Saturday night the web site was advising customers that the network and hosting services "will be back this week with security updates," adding that "current customers who are still interested in our services will receive compensation."

Comments

Not a big deal

[Anonymous Coward]

Nobody with a brain stores important data on someone elses server.

Re:Not a big deal

[JustAnotherOldGuy]

Nobody with a brain stores important data on someone elses server.

...without a backup.

Re:Not a big deal

[Anonymous Coward]

I store important things on other's servers all the time. I don't have enough bandwidth available where I live to host my own severs, it is cost prohibitive to co-locate a physical machine compared to a VPS.

You should have said "Nobody with a brain trusts the only copy of important data to someone else".

Everything important I have hosted on a cloud service I also have copies of.

Re:Not a big deal

[darkpixel2k]

Nobody with a brain stores important data on someone elses server.

...without a backup.

You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt. The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house.

I know I can do it in an hour because when Digital Ocean was having trouble at one of their data centers a few years back, I spun up new VMs and migrated everything to another data center.

Re:Not a big deal

[JustAnotherOldGuy]

You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt. The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house.

You're the kind of guy I'd want running my IT department.

Re:Not a big deal

[Anonymous Coward]

I wouldn't hire a guy who copies all my data to his house.

Re:Not a big deal

[nicoleb_x]

An hour, if you happen to be awake and available.

Re:Not a big deal

[darkpixel2k]

I wouldn't hire a guy who copies all my data to his house.

Funny, it's data from *my* company. I'm the guy who *owns* the data. So why shouldn't back a copy up to my 12 TB storage array at my house?

If I worked for *your* company, I would back it up wherever *you* wanted it.

Re:Not a big deal

[Anonymous Coward]

All that and you're an asshole too.

Re:Not a big deal

[AlanObject]

Could you explain "salt?" It's new to me but from your context I should know it. (A link would be fine).

Re:Not a big deal

[darkpixel2k]

An hour, if you happen to be awake and available.

Perhaps you've never heard of 'automation'? It allows you to perform actions 'automatically'. You can do wonderful things like launch backup jobs...

Re:Not a big deal

[AlanObject]

I wouldn't hire a guy whom I couldn't trust to have all my data at his house.

Re:Not a big deal

[sml156]

Not much point of having a offsite backup when the admin can delete that too.

Re:Not a big deal

[bill_mcgonigle]

Could you explain "salt?" It's new to me but from your context I should know it. (A link would be fine).

Hey, did you try googling 'salt'? ;)

I kid ... try salt devops though - you'll get what you need.

Re:Not a big deal

[Anonymous Coward]

Just don't forget the offline backups. That wiping may be someone doing "rm -rf /" on your VMs not "rm -f vm_hdd.vdi" on the host.

Re:Not a big deal

[93 Escort Wagon]

You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt.

Any competently run IT department should be able to largely recover from the malicious actions of an external actor. But that's not really the question now, is it?

The question is: if you chose to destroy all of that data - including, I assume, your salt configurations - could someone else recover and rebuild those VMs - including their (reasonably recent) data?

 

Re:Not a big deal

[sodul]

https://saltstack.com/

Without going into the details SaltStack is similar to Ansible, Chef or Puppet.

Re: Not a big deal

[Anonymous Coward]

I run a hosting service for many of our clients as an add on service to complement our application development, and we use replication between two datacenter (Las Vegas, Southern California), and everyone laughs when I spend money upgrading our tape backup system to LTO6. Yet only myself and an independent admin have access to the tape solution, so whatever happens, we can always recover everything. Replication can replicate deletes, and the only truest safe copy is one that is both offline and in your own hands. Sure, your point in time to recover maybe be from last night, but I bet these Dutch hosting providers wish they had that now.

Re:Not a big deal

[tommeke100]

Maybe that's the guy they had as well ... with a vengeance.

Re:Not a big deal

[moronoxyd]

The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house..

So, you only have one backup at one place? You're flirting with desaster.

Re:Not a big deal

[athmanb]

Can you really guarantee that you'll be able to do that if another admin with equal rights to yours maliciously wipes data?

Because sure you have snapshots but a couple lines of Powershell/BASH and all of them are gone in 5 minutes. And you might have tape or cloud backups but another few commands and the tapes get zeroed overnight while cloud storage can be deprovisioned in seconds.

Re:Not a big deal

[darkpixel2k]

Could you explain "salt?" It's new to me but from your context I should know it. (A link would be fine).

Yup--as others posted below, 'salt stack'. It's pretty much like 'Puppet', 'Chef', or 'Ansible'. Set up a 'salt mater' and point all your 'salt minions' to the master. Then you can define the way you want your systems to be configured from the master. i.e. things like disabling SSH password auth, deploying authorized SSH keys, configuring firewalls, cron jobs, packages installed, etc...

Re:Not a big deal

[geoskd]

I wouldn't hire a guy whom I couldn't trust to have all my data at his house.

Unless you're willing to pay them C-suite rates, you will not find people you can trust with certainty...

Re:Not a big deal

[darkpixel2k]

You can wipe every single VM I have and I can restore everything within an hour because they are all configured using salt.

Any competently run IT department should be able to largely recover from the malicious actions of an external actor. But that's not really the question now, is it?

The question is: if you chose to destroy all of that data - including, I assume, your salt configurations - could someone else recover and rebuild those VMs - including their (reasonably recent) data?

Yeah--exactly. In my case, our salt configs are checked in to a git repo, so it's as 'simple' as spinning up a new salt master, cloning the config, configuring the master master from the salt config, then spinning up the other hosts and kicking off the config. Then restore the databases from backups. In my case they should be ~2 hours old at most. As long as you have access to the git repo, you can do it. Finding a competent admin that knows salt (or puppet or chef) with the skills required to spin up the architecture is another matter though... ;)

Re:Not a big deal

[darkpixel2k]

Can you really guarantee that you'll be able to do that if another admin with equal rights to yours maliciously wipes data?

Because sure you have snapshots but a couple lines of Powershell/BASH and all of them are gone in 5 minutes. And you might have tape or cloud backups but another few commands and the tapes get zeroed overnight while cloud storage can be deprovisioned in seconds.

Yes, because I have three personal policies related to this:

1. Only hire admins I feel comfortable absolutely trusting.
2. Follow the principle of 'least privilege' (I have backups on my storage NAS at home, and I am the only one with access to the data. A friend of mine has a similar storage setup at his house and he also has backups of the database that only *he* has access to.
3. Keep backups of your salt config somewhere where other malicious actors don't have access to it. (Salt config is stored on github where admins have access and could royally fsck it up, but I also 'git clone' it down to my NAS every 24 hours and back it up off-site using 'tarsnap').

Re:Not a big deal

[darkpixel2k]

The databases are snapshotted every hour and backed up using tarsnap as well as an rsync down to a NAS at my house..

So, you only have one backup at one place? You're flirting with desaster.

Nope. Backups happen in two ways. ZFS snapshot combined with a snapshot pull to my off-site NAS, and an 'autopostgresqlbackup' "snapshot" that gets backed up via 'tarnsnap' as well as rsynced to yet another off-site NAS.

So there's a copy on the actual DB server, a snapshot on the DB server, a snapshot on my local NAS, a copy on another NAS in a different location, plus a tarsnap backup. I'm confident in my ability to restore. I've tested it. ;)

Re:Not a big deal

[rvw]

They have backups and although they haven't recovered yet completely, the servers that are recovered haven't lost data.

Re:Not a big deal

[Opportunist]

You did a restore test, right?

Re:Not a big deal

[Opportunist]

Depends on the guy and what "his house" actually means.

In my case it means another data center. Yes, I live in one. The air condition is awesome!

Re:Not a big deal

[Opportunist]

Believe me, if I need that data, he WILL be awake and available! I do know where my coworkers live. And there isn't a single point of failure, meaning that there are always at least 2 people capable of doing any job, and one of them IS within reach.

No, we don't simply subject our workers to such conditions. We pay handsomely for that privilege.

Re: Not a big deal

[Anonymous Coward]

They laughed because backing up to tape is inferior to backing up to a hard disk for 99% of non archive situations. Usb, drive docks, whatever. All cheaper and simpler than tape.

Re:Not a big deal

[Anonymous Coward]

Nobody with a brain stores important data on someone elses server.

...without a backup.

... without multiple, redundant, backups, that are by their nature READ-ONLY, and not stored at the same site. In fact, not stored in the same COUNTRY, and naturally, heavily encrypted with lots of fake data intercalated to ensure even if someone breaks in, manages to decrypt it, they won't know which is the real data and which the fake. (The key to what's real and what's not should of course be stored elsewhere, and no one person but the owner(s) of the company should be trusted to know about EVERYTHING.)

Re:Not a big deal

[Opportunist]

I can't delete anything anyway. But nobody really can. You see, our backups cannot be deleted by the admins. "Deleting" here means that you mark it for deletion which is executed at a later moment by the storage ... thingamajig (don't ask me, storage really isn't my strong side). Now, marking a recent backup for deletion pretty much instantly hits some of the storage upper echelons in the face because that isn't proper procedure and he'll ask 5 minutes after the mark gets set and about 5 days before it actually gets deleted what's going on.

Re:Not a big deal

[fisted]

They're talking about the restore job. I'm not sure I'd want to automatically launch that...

Re:Not a big deal

[Anonymous Coward]

Nobody with a brain stores important data on someone elses server.

Everyone stores important data on other people's servers, they just don't generally have any choice, or often, knowledge they're doing it.

For example, all my personal data was compromised when some gang of fuckups in the federal government let some assholes hack into the OPM database, a perfect example of why our federal government shouldn't be allowed to track us, or even really know who we are, since they're a bunch of goddamned fucking fuckups who can't do shit. Can't do shit RIGHT, anyway.

Then to add insult to injury, we get this mealy-mouthed fucking apology, and an offer for some brief period of complimentary credit monitoring.

GEE, Uncle Sam, THANKS! Now that you've lost all my personal data you DEMANDED I give you, to some local miscreants and crooks or some overseas criminals, you offer me a brief window of protection!

WHAT THE FUCK ABOUT THE DECADES AFTER THAT? What about the several more decades, (according to actuarial tables,) I'm likely to live, assuming that pile of orange dogshit in the Offal Office, (or is it Oafful Office, or Open Orifice Office, or Awful Office, or Oh Full of Shit Office, or simply the Toilet Bowlful Office, now?) doesn't fuck around, fuck up, and fucking fuck us all the fuck over by starting a war with a REAL adversary, (which we haven't had since 1945)?

Just saying.

Re:Not a big deal

[Anonymous Coward]

I also 'git clone' it [...] every 24 hours

You're doing it wrong.

Re:Not a big deal

[Interfacer]

Because C level guys' behavior is always beyond repoach. True upstanding citizens?
LOL :)

Re:Not a big deal

[barbariccow]

without a backup

Actually, when I worked at a.... major transportation organization, I once accidentally deleted the entire database. It wasn't my fault, my code worked and was tested through dev -> stage -> test and all that.. but at the last minute my boss was like "Hey, you didn't use this cutting edge new ORM technique, refactor now" WHILE I WAS PATCHING IT TO PRODUCTION!!!!! So I bowed my head and said "yessir..." Well, what was supposed to delete one record ended up cascading to every related model and.... BAM.

I won't go into too many details because it didn't get reported, but basically a lot of things were moving and all record of them disappeared. And that major infrastructure system didn't have proper backups.. Ended up pulling the database copy from test (which was, at the time, 2 days old) and restoring from that, within about 2 hours.

We did a good job shifting blame that it was somehow everyone else's fault, and I didn't get in trouble (because, like I said, my code worked. Bossman made it break with his "strongly suggested" refactoring). After that, I was never asked to refactor in such silly ways again.

Oh, and if you thought we had learned our lesson and started doing backups after that... you must not have worked in IT long :)

Re:Not a big deal

[darkpixel2k]

They're talking about the restore job. I'm not sure I'd want to automatically launch that...

Restoring from tarsnap is a bit slow at times, but that's an "everything else has failed" contingency plan. I have monitoring and alerting, so if the DB mysteriously went away, I'd wake up, coordinate with the other ops guys, and start a restore.

Re: Not a big deal

[barbariccow]

Tape is great for backups-of-backups. In my ideal system, you have incremental backups to a disk, and routinely (weekly, monthly) do a full backup to tape, and store tape offsite. There are actually some really nice tape implementations out there which let you use it just like a filesystem (as oppose to just TAR-like). It will store changes to disk, and after a certain amount of time passes it writes a complete, incremental, and compressed record to tape. That allows you to do things like rsync (which creates temporary files) without eating up huge amounts of tape, and get very quick turnaround. Tape is generally very very slow, so being able to dump data quick, then possibly fix it or something, and during off-hours the slow work happens in the background.

Re:Not a big deal

[barbariccow]

there isn't a single point of failure

We pay handsomely

Are you hiring?

Re: Not a big deal

[DimitarDimitrov6048]

So it is enough to blow up your house.

Re: Not a big deal

[darkpixel2k]

So it is enough to blow up your house.

Only if you didn't read and understand the entire comment.

Re: Not a big deal

[Anonymous Coward]

They'll laugh as long as someone drops the single existing backup drive on the floor. Needing to restore is stressful and people make more mistakes while being in stress.

Re:Not a big deal

[Anonymous Coward]

Nobody with a brain stores important data on someone elses server.

...without a backup.

I'd tend to argue that you probably need for important stuff at least, a couple offline backups with different dates in different locations that you own or control, plus an online backup from your own equipment that only your trusted people have access to.

The offline backups probably need, in this day and age to look for things that don't change, and verify that they haven't changed, via cryptographic checksums. Basically for less frequently accessed data, it would suck to have malware corrupt it, then keep backup up the corrupt data until you have no way to restore the uncorrupted data.

So to summarize, if you really want to keep things safe, there is no substitute for talented in house people. The cloud doesn't replace those, and even if you could come up with a scheme with enough cloud providers and such to be safe, you still, as others have mentioned need some local talent to keep on top of it and make sure it is doing what you want.

Re: Not a big deal

[Anonymous Coward]

One nice thing about tape is after it is created, it is offline, and can be physically marked as read only. Most disk-based solutions don't do that.

Re:Not a big deal

[rossz]

Bossman made it break with his "strongly suggested" refactoring.

Sometimes you have to tell your boss "no". That's what written policies are for, to give you the power to say no. If he insists, you ask for it in writing (at least an email trail). I have told a boss no, but I explained why. He didn't insist so it didn't get to the "in writing" demand.

Re:Not a big deal

[Anonymous Coward]

What sort of setup would you need to have something this resilient? Are your backups encrypted? What do you use to check their hashes?

I'm thinking I'd need at least another hard drive (with an encrypted partition) to attempt a robust backup solution. If you have any resources that helped you set this up, I'd be glad to read them.

Re:Not a big deal

[Anonymous Coward]

1. If it's your company, then you can do what you like. Most people aren't in such a situation.
2. Lots of people have to deal with pesky things like regulatory requirements. Copying data onto a USB drive might be grounds for dismissal, never mind doing wholesale dumps to their private unsecured systems.
3. It all fits in 12 TB? How nice. That's pretty manageable even for amateurs (not saying you're an amateur). Come back when you're dealing with 2,000 servers and have individual databases that wouldn't fit in 12 TB.

Re:Not a big deal

[Anonymous Coward]

He won't start a war, it's always those retards in Europe that start the real wars.

Re:Not a big deal

[Opportunist]

Actually we are. You'd be surprised, though, how hard it is to find suitable people. "No police record" and "years of IT security experience" is a combo that seems to be really rare...

Re:Not a big deal

[Anonymous Coward]

that's the idea some crazy guys are comming with
https://www.oneworld.nl/groen/nerdalize-computer-verwarmt-gratis-je-huis

Re:Not a big deal

[Anonymous Coward]

No, we don't simply subject our workers to such conditions. We pay handsomely for that privilege.

Are you hiring? :-D
I am working in similar elastic conditions for couple years.

last time, 2 month ago during "emergency meeting" our business users were a bit anxious (you know, New York delicate flowers) when I had to step into shoes of my "buddy" on call. He landed in hospital after car crash and i was away - what was frightening our dear business users was machine gun fire in the background.
(that was my NG weekend night out).

Re:Not a big deal

[quetwo]

Wait. So, now two people have full backups at their houses? I really, really hope that you don't deal with any data or servers that touch PCI-DSS, HIPPA, FERPA or other controlled data sets. I also hope you and your employee are never in a situation where you get laid off or fired. Having ALL that data in a personal residence will cause nothing but problems and could be extremely damaging to the company.

Re:Not a big deal

[Jason Levine]

Years ago, I had a website on a shared hosting environment at a company. One day, their servers all went down and their support lines weren't responding. Eventually, they came back enough to explain that they had gotten hit with a worm that was going around and that their servers should be back up in a week or so. This raised major red flags with me because I knew that the fix for this work was 1) take the servers offline, 2) apply the patch, 3) reboot the server, 4) done. Even a large data center doing this manually should be able to do this quickly - especially if they were pulling people from the support lines to assist.

I had all my files backed up but hadn't backed up my database recently. I was able to connect to the database server and quickly made a backup. Days turned into weeks and they still weren't online again. Every so often, they would pop back to announce that they were definitely coming back. Meanwhile, a forum of affected users (not on the company's servers) began to assemble with a lot of people lamenting that they lost all of the data for tons of clients.

In the end, the company vanished, only to re-appear under another name with the same management staff. (I'm not sure if anyone sued them or if the management repeated this scam.) A lot of people lost a lot of data, but I was spared. Since then, I've learned to always have a backup copy just in case. (I also wouldn't wait weeks for my host to restore my site anymore and don't use shared hosting. This was a long time ago.)

Re:Not a big deal

[Anonymous Coward]

So you run a tiny business. Good for you. This is about real stuff and the broader scope. For a self-proclaimed business and tech genius, it's surprising you are unable to see beyond you own teeny tinny world.

Re: Not a big deal

[rpstrong]

I just use write-protect tabs to cover the notch on the corner of the disc sleeve.

Re: Not a big deal

[Anonymous Coward]

Eh. I suppose burglars or thieves won't know what it is and won't have the equipment to read it either (unless they steal the drive)

Re:Not a big deal

[sorcaitis]

If you really own the company, what you do with it is your business. If you are working for someone else, moving data to your own home is potentially unethical and an option of last resort after you've presented better options for data archiving. The instance for the source article is a former admin was the one who wiped all the data. So the better question is, at your company, are you alone completely capable of wiping all of your data, backups, and archiving? What is the company's procedure for data security if you leave or are terminated? This wasn't a regular employee, it was an admin with access privileges which weren't revoked in time, or had set up a killswitch prior to leaving. It's not the first time it's happened, and won't be the last. So for me, while I'd like to hire someone I could trust to house the data at his own home, but even better, I'd prefer to hire someone who would tell me the reasons why it isn't a good idea.

Re:Not a big deal

[darkpixel2k]

Wait. So, now two people have full backups at their houses? I really, really hope that you don't deal with any data or servers that touch PCI-DSS, HIPPA, FERPA or other controlled data sets. I also hope you and your employee are never in a situation where you get laid off or fired. Having ALL that data in a personal residence will cause nothing but problems and could be extremely damaging to the company.

Nope. I'm not a doctor, I don't run any sort of medical clinic, and I don't touch PCI-DSS data. Remembver the original topic? Wiping and spinning up new VMs in the cloud? Why would someone be storing PCI-DSS data 'in the cloud' on virtual machines?

Re:Not a big deal

[darkpixel2k]

If you really own the company, what you do with it is your business. If you are working for someone else, moving data to your own home is potentially unethical and an option of last resort after you've presented better options for data archiving.

Agreed.

The instance for the source article is a former admin was the one who wiped all the data. So the better question is, at your company, are you alone completely capable of wiping all of your data, backups, and archiving? What is the company's procedure for data security if you leave or are terminated?

At my company (that I started and own), no one is capable of wiping all the backups.
If I leave or get 'terminated' (not sure how that works in a private business...death maybe?), it's probably in a situation where I've sold the company. It's up to the new company to implement their own backup and security policies and procedures.

This wasn't a regular employee, it was an admin with access privileges which weren't revoked in time, or had set up a killswitch prior to leaving. It's not the first time it's happened, and won't be the last.

Yup. A company I worked for 20 years ago had the admins start locking you out the moment you were pulled in to the HR meeting. They made sure the meeting lasted at least an hour. Gave us plenty of time to revoke access.

So for me, while I'd like to hire someone I could trust to house the data at his own home, but even better, I'd prefer to hire someone who would tell me the reasons why it isn't a good idea.

Meh. I'd rather have someone I can trust. Being told why it isn't a good idea sorta goes hand-in-hand with that.

Re: Not a big deal

[stoatwblr]

"In my ideal system, you have incremental backups to a disk, and routinely (weekly, monthly) do a full backup to tape, "

Why? A decent tape backup system(*)(**) will let you do nightly incrementals/differentials etc and ONLY restore the files you need if the time ever comes to do a full restore in anger.

Being able to do synthetic full backups is also handy when you can't afford the backup window on a particular server.

(*) Decent tape backup systems are backed with a database that keeps hashes of the files - that way you know what changed and when. It's a useful little basis for an IDS amongst other advantages (like being able to seek to the exact point on a tape where any version of any given file is, to restore that file and nothing else.) If the database gets trashed you can still do restores but it takes longer.

(**) The backup server should be entirely separate to everything else. That helps minimise exposure. I use Bacula plus a multidrive robot and am _very_ happy with it.

Re:Not a big deal

[stoatwblr]

"And you might have tape or cloud backups but another few commands and the tapes get zeroed overnight "

1: The backup server should be its own physical and administrative space, with its own security policies and be entirely separate from the backup clients (ie: only the backup admins needs access to the backup server.)

2: Tapes should be rotated frequently enough that the last full backup is ALWAYS in a data safe along with as close to current as you can get on differentials/incrementals. it's hard to zero a tape that isn't in the robot - and if you're really paranoid you can use WORMs or password the ability to write to the tape, so that zeroing can't happen without a good deal of insider knowledge.

3: LTO drives have encryption. Use it. That saves the embarassment if tapes are stolen or otherwise go missing on their way to Iron Mountain or whatever other offsite storage you may use (I spent $9k each on a pair of decent data safes and the building can't burn for more than an hour without dropping an airliner full of fuel on it.). In some countries that embarassment comes with a side helping of large fines from your data protection office should the backups happen to contain anything resembling personal information.

Re: Not a big deal

[Anonymous Coward]

Does tape and offline disk survive big EMP hits. I think tape last better under mechanical stress like a roof collapse or simple explosion. I think tape last longer under reasonable conditions than the high density disk drive. Also I think the amazon glacier at least used to be just tape.

Re:Not a big deal

[Anonymous Coward]

It would be hard to upload the 100+ TB backup requirement that I have onto new servers.

Just getting enough hardware in place will take days.

Re:Not a big deal

[RockDoctor]

their servers all went down and their support lines weren't responding. [...] do this quickly - especially if they were pulling people from the support lines to assist.

Hang on ... think that through for a second. That suggests either that they're routinely using senior IT Admins on their tier-1 phone lines ("Hello customer, what can I help you with? ... HAND!"), OR they're using tier-1 phone line staff to do important admin work, OR ... does anyone else smell bullshit? (From the company involved, not the OP.)

Re: Not a big deal

[bigmacx]

Yes to this. Tapes as offline backups are great protection from ransomware and intentional admin abuse. If possible, however, I'd rather have a rack of USB external hard drives to push to and then take those offline.

All-in-all, hard drives better than tape, but I think they are more susceptible to shock damage than tape, but that's offset because hard drives can handle more extreme temps compared to tape.

And then there's always the part where a lot of us have tape drive systems in libraries that are not really offline because we're too lazy rotate tapes offsite for our 3rd-tier backups.

Statistically, you're more likely to suffer an accidental delete as a need for a restore than any damage from disaster or mischief. That's where online backups excel.

Re:Not a big deal

[david_thornley]

1. Only hire admins I feel comfortable absolutely trusting.

I know people I absolutely trust. I also know they're going to screw up now and then. I'm also aware that I can be mistaken in my absolute trust, but I'm willing to take the chance.

Re:Not a big deal

[Anonymous Coward]

This is so NOT how it happens in the real world!

In my experience, even if the hosting contract states in huge flashing neon "We will NOT backup your machine in anyway", the EU still believes it will be backed up regardless, by someone else, so they don't really need to backup themselves. Not even the databases! It's usually when they hose their own VM by doing something stupid, and they then call support to "urgently restore" their insanely critical VM, that they truly start to understand what NOT being backed up means.

Re:Not a big deal

[darkpixel2k]

1. Only hire admins I feel comfortable absolutely trusting.

I know people I absolutely trust. I also know they're going to screw up now and then. I'm also aware that I can be mistaken in my absolute trust, but I'm willing to take the chance.

Yeah--part of that should be dealt with by policy and procedure. While admins should have access to destructive DB operations, they shouldn't be accessing the database with those special privileges if they are writing code for example.

Re: Not a big deal

[Anonymous Coward]

You sound like a jerk to work with. And salt n tarsnap are Trash.

Deja vu

[Anonymous Coward]

Seems like a glitch in the matrix

And...

[Anonymous Coward]

This is why you change the passwords when an employee leaves. Hey, I'm smarter than the guys that work at Verelox!

Hire me, you dumb motherfuckers!!

Re:And...

[Anonymous Coward]

Hey dumb motherfucker, ever heard of a logic bomb? Or backdoors? If this guy went and deleted everything, what exactly makes you think that he didn't also backdoor everything or planted a logic bomb to delete it all.

Sounds like you are just as stupid as the guys who work at Verelox who think that just removing a account/passwords solves all security issues related to firing a sysadmin.

Firing a sysadmin is perhaps one of the most dangerous things a company can do.

Re: And...

[Anonymous Coward]

It wasn't a guy, it was a female.

Re: And...

[Anonymous Coward]

English doesn't have a gender neutral term for a singular person when the gender isn't known. I didn't see anything in the story regarding the gender of the ex-admin.

Also how the hell does that change anything I said in my post?

Re: And...

[Anonymous Coward]

Erm....person works fine for a person when the gender isn't known.

Re: And...

[thundercattt]

They were probably having him teach his supreme Indian replacement.

Re: And...

[rholtzjr]

Wow, and some people just wake up in the morning and decide, "hey, I feel like pissing off half the planet today". Go figure.

Re: And...

[Anonymous Coward]

'Guy' is often used colloquially to refer even to explicitly female individuals these days. Colloquial language is common on this website.
Whether the sex of the admin matters is another matter entirely...

Re: And...

[Anonymous Coward]

calm down snowflake, not everyone is out to offend you

Re: And...

[Anonymous Coward]

Not even, more like, "I DESERVE a woman because I'm SOOO NICE!" all while jerking off into his mother's panties.

Re: And...

[sabri]

And so we come to the crux of the problem: FEMALES.

This has nothing to do with male vs female admins. If anything, it's the other way around. In my experience, females 'accept' more abuse than male admins. And by that I mean, that the women I've worked with let it slide more often and longer and only decide to look for another job when really needed.

This whole case is nothing more than amateur hour of a three-man "hosting shop" without proper procedures. Just look at the whois information for verelox.com. The phone number listed is a Dutch cell phone number.

They're amateurs. Nothing more and nothing less. And they made a rookie mistake.

Re: And...

[Anonymous Coward]

When I was a kid in NYC, I grew up in a neighborhood that was immigrant heavy. (I was immigrant light...), and one of the common confusions in communication was the fact that English doesn't have common Gender formations. "You" and "Youse" were a common workaround. There was even "Theyse". You'll see many examples in the writings of Damon Runyan and themse. "Guys" were specifically Male, as in "Guys and Dolls". Backward formed into the derisive "Guidos" fairly recently.
Later when out in California, it was all "Hey Guys" and "You Guys"; Gender Neutral, and routine in Girlspeak. I'm not sure how it all started; perhaps it was Surfer's Lingo. "Guy" was old Norman French for "Guide", and later adopted as a minor form of Noble Address: "Guy de Maupassant". In Nautical terms, which Surfers had some familiarity with, a "Guy Wire" is a "Guide Wire"- Standing Rigging.
There is a lot of misinformation going around that "Guy" originated from "Guy Fawlkes". It was already an old and common name before his gunpowder, treason, and plot.
And now you can all get back to your collective misogyny.

Re: And...

[Brockmire]

"Guys", yes. "Guy", no. Never.

Re: And...

[rholtzjr]

You know, /. really aught to prohibit modding and posting AC in the same article. Really? Modded down for sticking up for a female? I hope you and your other butt-buddy AC have a fun time tonight while packing each other.

Re: And...

[Anonymous Coward]

Sociopathic silverspoon nepotist

Re: And...

[Anonymous Coward]

This whole case is nothing more than amateur hour of a three-man "hosting shop" without proper procedures. Just look at the whois information for verelox.com. The phone number listed is a Dutch cell phone number.

They're amateurs. Nothing more and nothing less. And they made a rookie mistake.

Does a cell phone number suggest they're amateurs? I find land line numbers to be really really scarce nowdays, at least here in Finland.
Even if there is a "land line"-number, it probably is a virtual number for rerouting the call to mobile.

Not to worry

[Anonymous Coward]

It's Shirley backed up to the cloud someplace.

Re:Not to worry

[Anonymous Coward]

It's Shirley backed up to the cloud someplace.

No it isn't. And don't call me Shirley.

Loving the cloud

[Anonymous Coward]

All right, sweethearts, what are you waiting for? Breakfast in bed? Another glorious day in the Cloud! A day in the Cloud is like a day on the farm. Every meal's a banquet! Every paycheck a fortune! Every formation a parade! I *love* the Cloud!

Re:Loving the cloud

[rvw]

The thing is that it seems they haven't lost any data. I bet the cloud did no harm.

Good

[110010001000]

Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.

Re:Good

[Brett Buck]

What is particularly idiotic is that everyone didn't understand it from the beginning. But clearly, they don't.

Re:Good

[Lisandro]

That's a bit condescending of a position. Most cloud users do know that; you pay for the convenience not to fund and support a datacenter yourself.

Re:Good

[toonces33]

Hardly surprising. People look at the bottom line. Ohhh, we can save $$$ by outsourcing nearly all of our IT to some cloud company!

Re:Good

[Anonymous Coward]

That's why its always on YOU to do your own backups, that's never changed.

Re:Good

[Anonymous Coward]

Of course, this kind of thing couldn't happen to a company that hosted its data on prem with its own IT staff.

Re:Good

[XXeR]

Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.

Hosting was around long before cloud, and for some reason never garnered the amount of haters that cloud currently endures. VPS is hosting, not cloud...please direct your hate appropriately.

Re:Good

[Anonymous Coward]

It's "wipe your production data clean" news day, isn't it? Second story for today!

Re:Good

[Anonymous Coward]

No, most cloud users really don't. In fact, the cloud providers have an incentive to make sure that you are kept locked in. Sure, you can back up your data, but that's a whole lot different than a disaster recovery plan that's been tested.

Re:Good

[Anonymous Coward]

The Cloud is like Windows. A good clean install is needed every couple of years, to wipe out the cruft.

Re:Good

[Anonymous Coward]

But, but, but, disaster recovery!

Re:Good

[Anonymous Coward]

As a cloud customer we evaluated how much $$$ it would cost if our cloud company lost our data, and we picked a cloud company with a >$$$ insurance against data loss. Not sure what people are complaining about. In fact our business isn't doing very well right now so we're actually hoping they'll lose our data so we make some money.

Re:Good

[Anonymous Coward]

Maybe people will start realizing

Well congrats you just figured that out, I guess. Welcome to 1970!

You're in for a real treat in a couple years when you figure out you can buy a computer for use at home too!

Re:Good

[dissy]

Hosting was around long before cloud

That is very likely not true, unless you count being best friends with one of the admins at uni who gave you a shell account for free on a server.

"Cloud" is a term from the 1970s.
"Hosting", as performed by a company specifically offering such a thing, came about in the 1990's after the commercialization of the Internet.

The 90's did not come before the 70's

Re:Good

[maestroX]

Maybe people will start realizing that the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.

You're barking at the wrong tree. Customers just buy a service based on price, uptime and retention, as advertised.

If you don't feel respected due to underpayment, lack of resources or management, it's your job to step up and take a stand.

In any case, the customers are most likely not at fault so don't fuck them. It looks bad on all of us.

If you can't deal with responsibility, stay the fuck away from power.

Re:Good

[radarskiy]

'the Cloud is just "someone elses servers" and you have no idea how they manage them or back them up.'

The Internet is is just "someone elses networks" and you have no idea how they manage them or back them up.

Re:Good

[Anonymous Coward]

Abstractly speaking, if management doesn't understand how IT works, or just pays IT personnel and doesn't pay attention to anything they do, that's no different than outsourcing everything to a services company, at least from an operational readiness and security perspective. In both cases, the "not technical people" are just blindly paying "tech people" to do work that they don't understand, which gives the tech people almost complete control over how work is actually done and leaves management completely impotent when something goes wrong.

Re: Good

[Ash-Fox]

But I own my cloud? So, how could it someone else's?

Re:Good

[raftpeople]

Hosted solutions started back in the 60's, typically mainframe. The earliest I encountered it was mid-80's, one of our clients used a hosted financials application.

Re:Good

[Anonymous Coward]

Service bureau was the term used by financial institutions in the 70s and 80s for hosted mainframe applications.

Re:Good

[iCEBaLM]

Hosting is Cloud. It's like saying cars were around long before automobiles.

VPS is hosting is cloud. There is no difference. It's a marketing term to synergize the verticals.

Re:Good

[Anonymous Coward]

Wrong, incorrect, and you are on the wrong site. It's a separate servers you buy/rent and have running in huge dedicated buildings that provide massively more reliable internet connects in environments most businesses will no longer pay for.

Re:Good

[david_thornley]

That's why you keep local copies.

Re:Good

[Anonymous Coward]

Most insurance policies have a "get out" clause if the insured party is professionally incompetent. If they come along and say you need several monthly tested offline backups separated by certain geological distances, you may be left with nothing.

Hell hath no fury..

[Anonymous Coward]

..like a BOFH scorned.

Re:Hell hath no fury..

[RockDoctor]

Oh no - the BOFH would have deleted the (rather, Yet Another) Pointy-Haired Boss or put them into the long term backup (lime pits) long before it reached the point of being an ex-Admin.

So...

[Kierthos]

Did they not remove the ex-admin's credentials, or what? I mean, regardless of how the ex-admin gained access to the data to wipe it, it's a crime. But I'd like to know if Verelox was stupid enough to not remove his credentials, or this happened some other way. (Like, did he do this his last day of work as a "fuck you" to management for firing him?)

Re:So...

[Anonymous Coward]

Its not like a jaded administrator couldn't have put some backdoors into the systems, planted some sort of logic bomb that wiped out everything automatically etc.

Not everything is as simple as just removing credentials.

Re:So...

[Kjella]

Did they not remove the ex-admin's credentials, or what?

They should... but if you're sitting with the keys to the kingdom you might have the domain administrator account password, root passwords, various service accounts set up for particular purposes including but not limited to integration with external access... Yes, all could be done with the proper procedures in place. But very often the responsible for such IT procedures is the admin and the admin is the one keeping tabs on what everyone else has access to. Plus you often have the rights to create undocumented loopholes that you might reasonably excuse as being a test account and an oversight if discovered. Not to mention the setting you'd bring this up, either you're basically questioning the loyalty of one of the most trusted men in the system or it looks like you're setting him up to be fired.

Re:So...

[toonces33]

They may well have done this, but an admin is well placed to create a backdoor that nobody else knows about.

Re:So...

[Anonymous Coward]

There is also the issue of offsite backups...why wouldn't this company have them?

Even though the admin's actions were criminal, failing to take basic steps to protect yourself from a rogue admin, is negligent.

Re:So...

[Anonymous Coward]

He just made sure that his admin access was disabled properly.

Re: So...

[Anonymous Coward]

Any company with a single point of failure, like no backup admin or no data loss insurance policy, is run by a fool. Poorly.

Re:So...

[ausekilis]

Then it's a matter of strong procedures. There should never be a single point of failure, and all those passwords should be written down and sealed away somewhere accessible by the appropriate people.

The CEO/President may not *need* the passwords, but if he/she (and/or his/her admin) have the passwords saved somewhere then should the Admin get hit by a bus the company can keep moving forward. Any sane company would have the hand-off of the keys to the kingdom as part of the out-processing procedures. This includes updating POCs for any external accounts, such as cloud storage.

It's really common sense and not a huge burden to keep things up to date, even for small companies.

Re:So...

[RockDoctor]

and all those passwords should be written down and sealed away somewhere accessible by the appropriate people.

And your procedure for updating the locked-away passwords is?

And your procedure for checking that the password hasn't been changed without notification is ?

Conceptually, it's simple. In practice, it's not so simple. That's why screens have bezels - for sticking the post-it to.

At least link the articles together, geeze!

[Anonymous Coward]

https://developers.slashdot.org/story/17/06/10/0450234/developer-accidentally-deletes-production-database-on-their-first-day-on-the-job

CAP === 'tenant'

Wiped with a cloth or something?

[Anonymous Coward]

Seems like these incidents happen more frequently now, or maybe they are just reported more. You must be able to trust your sysadmins. I wish they had more details about why the ex-admin did this.

Re: Wiped with a cloth or something?

[Anonymous Coward]

I found details in a Dutch newspaper. The sysadmin was a Muslim fellow and was disgruntled when his boss told him that instead of taking a break for prayers five times per day he should write a cron job to do it for him. The police decided not to treat this as an act of terrorism so as not to invite islamophobia, but they suggested if your sysadmin is named Mohammed you should hire an outside firm to ensure your backup plan is effective.

Re: Wiped with a cloth or something?

[Anonymous Coward]

Why the fuck are all Islams named Mohammed, Mohamad, Mohmet or the like? Do their parents have no imagination or is this another one of those things the Koran forces them to do?

Slashdot wipes memory of recent posts?

[guruevi]

At least these two stories are from different perspectives: https://m.slashdot.org/story/3...

The story stays the same - don't fuck over your admins and have proper procedure and backup.

Re:Slashdot wipes memory of recent posts?

[Anonymous Coward]

The story stays the same - don't fuck over your admins and have proper procedure and backup.

Or alternatively -- don't trust admins, they can be stupid and / or assholes, and don't care about your business.

Re:Slashdot wipes memory of recent posts?

[Anonymous Coward]

Are these two stories the same incident or just happened at the same time?

Re:Slashdot wipes memory of recent posts?

[Sqr(twg)]

The first story does not identify the company, other than saying that it's fairly large ("40+ devs"), but if that story is actually true, then it seems likely that the two are the same. Otherwise I think we would have heard from the customers of a second company affected by an outage.

I can't imagine any company being honest enough to put out a press release saying "We lost our customers' data because we put the root credentials of the production database in a training document, and we didn't have working backups." And calling the new guy an "ex admin" would technically be true, since they fired him.

Re:Slashdot wipes memory of recent posts?

[93 Escort Wagon]

Whether or not these are the same story - the CTO should be fired (in both cases - this case).

Re:Slashdot wipes memory of recent posts?

[jafiwam]

The story stays the same - don't fuck over your admins and have proper procedure and backup.

Or alternatively -- don't trust admins, they can be stupid and / or assholes, and don't care about your business.

How do you "not trust" your admin?

You have to put a second admin there, who knows as much, and then trust THEM.

How do you trust the second admin? Maybe that person is the problem instead!

You could put a complicated system of checks and balances in place and reward following draconian rules and reward informing on someone who calls out incorrectness... But you still have to trust them.

The only practical way to increase the reliability of the meat based "admin system" is to keep the meat informed, happy, well paid and motivated to share the goals of the company.

Re: Slashdot wipes memory of recent posts?

[Anonymous Coward]

NO ONE cares about your business. That's why you pay them. Pay your admins a good wage and treat them like real human beings, and you'll have no problems. Pay them chump wages and treat them like subhuman dirt, and no one will have any sympathy when they hose your systems.

Re:Slashdot wipes memory of recent posts?

[jandersen]

The story stays the same - don't fuck over your admins and have proper procedure and backup.

There's a lot of lessons that management somehow never seem able to learn. Like, never have only 1 sysadmin, even if having more seems like overkill. Or, how about nurturing a relationship based on mutual respect and trust? The sysadmin has after all been trusted with one of the most important resource in any company, in most cases: the data. Yet, in a previous job I had an experience that I think is not uncommon: I had been there for over 10 years, I was regularly praised by my colleagues, but I was 'old' (ie >50); so my manager tried several ways to get rid of me - unreasonable demands, vague targets with moving goal-posts, etc, and when that didn't work, I was made "redundant" and asked to train some poor, unexperienced soul. On my last day, I wiped my workstation disk (very easy to do, not easy to recover from: using the 'dd' command to copy from /dev/zero to /dev/sda overwrites the partition table and then everything else, and the system doesn't report an error until it needs to load something that isn't there); nothing criminal in that, but just imagine I had decided to be creative and incorporated some discrete time bombs of that sort months later, when I was well away? It isn't hard to do, and it isn't hard to see how it could be made to look like the new guy had screwed up in a major way. Fortunatey I am both honest and mature enough to not do this kind of shit; but that company could have suffered significant losses if I hadn't been, and I think complacent, idiot managers, who are incapable of inspiring trust and loyalty and treat people like cattle, should be flayed and rolled in salt - after being given a drug to heighten their pain sensitivity, not that I want to sound harsh or anything.

Re: Slashdot wipes memory of recent posts?

[Anonymous Coward]

NO ONE cares about your business. That's why you pay them. Pay your admins a good wage and treat them like real human beings, and you'll have no problems. Pay them chump wages and treat them like subhuman dirt, and no one will have any sympathy when they hose your systems.

That is incredibly naive.

Re:Slashdot wipes memory of recent posts?

[RockDoctor]

How do you "not trust" your admin?

You have to put a second admin there, who knows as much, and then trust THEM.

How do you trust the second admin?

So, Juvenal is a candidate for the Patron Saint of InfoSec? Since he was aware of this problem around 100AD. (He's the guy who came up with the question "Quis custodiet ipsos custodes?" ("Who watches the watchers themselves?")

Backups?

[edibobb]

Why no secure backups? Idiots.

Re:Backups?

[QuietLagoon]

Why no secure backups?...

The article(s) seem to indicate that most, but not all, customer data can be recovered. So it seems there were working backups. But in a hosting environment, not everything is backed up continuously, and that may be where some of the data will be lost.

Re:Sounds Familiar

[Jzanu]

Only if it involved sabotaging US nuclear weapons and then selling state-of-the-art ABM technology to the Russians, North Koreans, and the Iranians. With discounts to ISIS, the PLO, Al Qaeda, and Boko Haram. Just to be thorough about screwing you all for a perceived insult.

They didn't make backups?

[Anonymous Coward]

What were they thinking?

Re:They didn't make backups?

[rvw]

Yes they did. Almost everything is recovered.

Procedures

[Anonymous Coward]

How is it possible that one guy deletes the storage, snapshots, backups, off-site backups, off-line backups with an account that was disabled after he left ?
Or did the company not have their procedures in order ? :D

It's time we did more deletions

[Anonymous Coward]

You outsource my job, I delete your data. I hope next cryptocurrency data gets deleted off of exchanges to burst the current bubble.

Re: It's time we did more deletions

[Anonymous Coward]

Then you go to jail. Have fun getting raped!

Re:Sounds Familiar

[Anonymous Coward]

Only if it involved sabotaging US nuclear weapons and then selling state-of-the-art ABM technology to the Russians, North Koreans, and the Iranians. With discounts to ISIS, the PLO, Al Qaeda, and Boko Haram. Just to be thorough about screwing you all for a perceived insult.

Oh, so you mean Obama's team then?

It's not just the about $$$

[Anonymous Brave Guy]

Executives also read the press release, though. The mighty Cloud was supposed to mean much easier administration so we didn't need to handle most IT matters in-house.

In actual $$$ terms, at both the low end and the high end the Cloud often works out more expensive than self-hosting, often by quite a wide margin. There's a zone in between where that doesn't always seem to be the case, but I'm not sure how wide it really is, and it's usually based on TCO rather than the hardware and connectivity expenses alone. The thing is, it turns out that you can't just delegate all responsibility and get good results.

How many times has a significant chunk of the Internet gone off-line because a major AWS data centre was knocked out for a while? Sure, all those beautifully-architected, Cloud-hosted web apps could have just switched over to a standby -- the AWS infrastructure would have supported that in various ways -- but it turns out that you still need enough expertise to understand how your infrastructure works, Cloud or no, or you just moved your single points of failure/vulnerability to another building. And of course the same considerations apply to all the other big Cloud hosting systems as well, as well as to simpler hosting like your favourite blogging platform or storing your code on GitHub.

Likewise in terms of finances, how often has someone who didn't fully understand the implications of a Cloud system or even just off-site hosting been hit with a huge bill they didn't expect, because the pricing model wasn't clear and they didn't really know what they were paying for and how much to budget?

There still ain't no such thing as a free lunch, and outsourcing critical business functions still ain't a silver bullet.

Re: It's not just the about $$$

[Anonymous Coward]

Um, no? It's much cheaper. If you can't set it up right, you should be fired.

Re:It's not just the about $$$

[CmdrTamale]

The thing is, it turns out that you can't just delegate all responsibility and get good results.

Just to be a little bit pedantic, you can delegate authority, but you cannot actually delegate responsibility.
--
What would Loki do?

Re:It's not just the about $$$

[Anonymous Coward]

In actual $$$ terms, at both the low end and the high end the Cloud often works out more expensive than self-hosting, often by quite a wide margin. There's a zone in between where that doesn't always seem to be the case, but I'm not sure how wide it really is, and it's usually based on TCO rather than the hardware and connectivity expenses alone

Variability of workload is a big factor there. If your loading is very uneven, it can be cheaper to offload to something like AWS than to keep enough servers around to handle peak loads.

Likewise in terms of finances, how often has someone who didn't fully understand the implications of a Cloud system or even just off-site hosting been hit with a huge bill they didn't expect, because the pricing model wasn't clear and they didn't really know what they were paying for and how much to budget?

That sort of thing happened long before we started calling it the Cloud. I got hit with it more than 40 years ago, when somebody was sold on the idea of moving one of our biggest production jobs to an off-site machine and downsizing our own. They didn't at all understand how the characteristics of that job interacted with the vendor's pricing algorithm, and the costs were way higher than predicted.

Re:It's not just the about $$$

[Anonymous Coward]

err, you're not pedantic, just annoying. Of course you can delegate responsibility.

Re:It's not just the about $$$

[Anonymous Brave Guy]

If your loading is very uneven, it can be cheaper to offload to something like AWS than to keep enough servers around to handle peak loads.

That is true, but how many web apps really have such highly variable demands that this works out in their favour?

That sort of thing happened long before we started calling it the Cloud.

True enough, but that doesn't make the current Cloud-related incarnation any better.

Re: It's not just the about $$$

[Anonymous Brave Guy]

Of course it's not much cheaper. At the low end, if you only need a handful of servers, it's far cheaper to just buy them and colocate them somewhere, and the management overheads are low anyway. At the high end, it's still cheaper to just buy the dedicated hardware and arrange your own facilities and IT team, because that's all you're effectively doing with Cloud hosting, you're just paying someone else's profit margin on top. In between, where there is a chance to make savings on things like rapid scaling up/down without having to buy the maximum level of hardware you ever need or through reduced management overheads, the higher direct costs of Cloud infrastructure might be outweighed by those savings. As I said before, it's not clear how wide that zone is. There are a lot of businesses who think they're in it, but in my experience that's not always the case when you run the numbers.

As for not setting it up right, many of the most high-profile web sites and apps in existence have suffered downtime as a direct result of "not setting it up right". If they can't do it reliably, even with their modern tools and Bay Area salaries and IT staff who have been using these technologies since their inception, maybe the problem isn't (only) with the clients using these services.

Re:It's not just the about $$$

[tlhIngan]

That is true, but how many web apps really have such highly variable demands that this works out in their favour?

Easy - e-commerce does. Every person consumes a certain percentage of server resources and you can handle the average load pretty easily.

But on special days like Black Friday (or Boxing Day or others), your server load and customer count can easily be 2-5x as much, or more. Just for a single day of shopping. And maybe if there's a big day of sales, it may peak at 2x or so. While a 2x peak might be able to be handled by existing resources (people may have to wait to check out), peaks of 5x or more are insanely difficult to architect for.

And it's not just the web front end that displays the webpage - companies suddenly realized when traffic spiked, their database load went up - it's not just 5x the front end web servers, now you need to beef up the database server as well to handle all those new requests. But then another part of their infrastructure failed - the payment side! People can buy and view products, but checking out is an exercise in frustration as the payment servers collapsed under the load. So they beef that up as well, but then their payment processor collapses!

Indeed. some retailers realized this and had to add slow down queues so their payment processors wouldn't get overwhelmed with the sudden strain. (And upgrading those servers is nigh-impossible because you can't easily spin up an AWS instance due to regulations and security). Heck, some retailers start pushing it off to Paypal and other processors to help with the load - either use our processor and wait, or use Paypal and check out instantly (even though we hate Paypal as much as you do, they can at least handle the load).

Re:It's not just the about $$$

[Anonymous Brave Guy]

That is a good example. If you're running a large online store and have very seasonal buying patterns, sure, the on-demand scalability of Cloud-style architecture makes sense.

I think it's also fair to say that not all e-commerce is like that. I've worked with a few smaller or niche market stores, and they'd love to have enough demand that hosting everything on a handful of servers and delegating payment processing to an external service wasn't sufficient. If you're a big national chain and dealing with thousands of orders per minute, your situation is different to, say, a local furniture store that would be happy to reach a thousand sales in a month.

Re: It's not just the about $$$

[nasch]

At the low end, if you only need a handful of servers, it's far cheaper to just buy them and colocate them somewhere, and the management overheads are low anyway.

You can get a very small (t2.micro) Linux AWS server with 1GB storage for $106 a year. Is that really more expensive than colocation (let alone far more expensive as you claimed)? Then consider that you can set up an identical server on the opposite coast that costs almost nothing until you need to spin it up for disaster recovery. There are also ways to set up replication and automatic failover though I'm less familiar with that.

Re: It's not just the about $$$

[Anonymous Brave Guy]

You can get a very small (t2.micro) Linux AWS server with 1GB storage for $106 a year. Is that really more expensive than colocation (let alone far more expensive as you claimed)?

That's quite an apples-to-oranges comparison, don't you think? A single t2.micro instance is a far lower specification than even the most basic server you'd colo, and surely no-one is running a substantial online service on a single t2.micro instance and nothing else even in AWS world. I confess that I've never got my head around all the modern AWS options, but it looks like even a moderately loaded web server could exceed the bursty CPU behaviour of such a small instance, and you need to allow for storage and data transfer as well. If anything, this is closer to the general purpose admin account we have on a shared hosting box with our colo/hosting provider, which costs a lot less than $106+storage+transfers per year.

Re: It's not just the about $$$

[nasch]

t2.medium is $412 per year, plus about a buck per gig of storage, and $0.09 per GB data transfer out up to 10 TB per month (up to 1 GB is free). How does that compare? For apples to apples it should be somewhere with redundant everything since I'm sure Amazon data centers have that. Also, if t2.micro (which isn't even the smallest available) is smaller than anything you would colo, then that is a plus for AWS. That means something that would be run off some developer's desktop machine, or shoehorned onto another server where it doesn't really belong could be put on its own virtual server instead for peanuts. Not to mention that there's a whole category of usage called spot instances that they say are 50-90% cheaper than normal pricing. This is not suitable for something like a web server, but if your usage is suitable for it, could be a really cheap way to get a lot of server.

This account on a shared hosting box sounds a lot like running your process on someone else's server, which I thought is what you wanted to avoid. Or do you not really care about that and your argument is just about price? A lot of people think AWS and the like is a bad idea because of relying on "someone else's computer".

BTW I'm not trying to be a dick, just interested. I worked somewhere with the colo solution but I didn't see anything about the costs. Now I work somewhere that uses AWS, and the biggest difference I can think of offhand is that if something goes really badly wrong, I can reboot a server from a webpage, whereas at my old job someone had to drive to the colo. Also if the colo got taken out by a hurricane or a backhoe or something, we would have been pretty well screwed, whereas now I might be able to get a new server going in a different region fairly quickly. It would probably depend on whether the database backups were available. If we were really serious about DR, we'd have those copied to a storage service run by some other company.

Re:It's not just the about $$$

[RockDoctor]

--

What would Loki do?

Laugh ?

Re: It's not just the about $$$

[Anonymous Brave Guy]

It's tough to do a fair comparison, because I'm in the UK so pricing for a lot of things will be slightly different, but I can have a go in general terms.

As an example, a colo package we've looked at for one of my projects would place boxes in major London data centres with proper redundancies etc. The entry level has 1TB/month of data transfer included and enough power and rack space for an entry-level server, and would cost about the same as the data transfer costs alone with AWS, with a similar rate for any extra bandwidth required. So at that level the colo aspect is heavily dominated by bandwidth, although of course it's a bit more expensive if you need extra space or power for a larger server.

The next question is probably how the cost of an equivalent server stacks up against a t2.medium instance. Since I don't know anyone who sells colo servers with 40% of a CPU and 4GB of RAM, I can't meaningfully compare these, so I'll have to use another example. If I've correctly interpreted what a vCPU actually is, a fairly basic server that we might use for the project I was thinking about looks roughly equivalent to an m4.2xlarge instance in terms of processing power and RAM. Amazon will rent you one of those for a bit over £3k/year in London. Reserving for a year with full up-front payment seems to save just over a third of that AWS fee. Buying a server outright probably falls somewhere around the full (non-reserved) price, depending on the rest of the spec, so you're looking at say 1-1.5 years of AWS EC2 instance fees to buy a roughly equivalent server.

To be realistic, we should then allow for significant inefficiency with having a permanent server that probably won't be 100% utilised, but that only seems to make a difference if you've got one of the smaller, burst-friendly EC2 instances, since if we're looking at the prices for reserved instances on AWS they have analogous inefficiency.

We should also do some sort of calculations regarding storage, but this is probably the hardest area of all to do a fair comparison. The price for a single GB-month of storage on EBS is already far more than the price of buying equivalent storage in a server hard drive outright, or about 3-4 months' worth of AWS EBS storage buys you the equivalent in local SSD capacity. We'd have to consider things like redundancy/backups through EBS and inefficient use of resources with local storage to get any sort of meaningful comparison here, but assuming 50% capacity usage of any given drive and let's say only 2/3 efficiency due to whatever RAID setup or other redundancy we have in our servers, you're probably still looking at buying fast, local, redundant storage within a year of the equivalent EBS storage capacity.

I'm certainly not claiming that pricing is the whole story, because even these basic back-of-envelope calculations show up obvious differences in terms of how the pricing models work and redundancy and efficiency issues that need to be taken into account to decide which approach is best for any given situation. And of course if you really do only need a very low level of resources, AWS will do that just like any other level it offers, while you'd be looking at a shared hosting account rather than a dedicated hosted server or colo arrangement.

However, for pricing, I think it's quite hard to find a situation where the cost for the actual hardware resources and connectivity doesn't work out quite a bit cheaper for self-hosted (or colo) arrangements over the long term unless you have sufficient variation in demand that your usage is going to be highly inefficient and the inherent scalability of AWS becomes the deciding factor.

Beyond that, there are the issues of management to consider, but my argument there was that the Cloud emperor often has no clothes, because it turns out that you just need experts in the Cloud infrastructure and how to deploy efficient and resilient systems in that environment instead of experts in hosting or colocating self-managed equipment.

Re: It's not just the about $$$

[nasch]

Interesting, thanks.

Takeaway

[UnixUnix]

Love and coddle your Admin -- or else!

Re:Takeaway

[Anonymous Coward]

Love and coddle your Admin -- or else!

That's naive. It's like "pay politicians lots of money and they will not be susceptible to corruption". It doesn't work that way. Once you get more than you need, you figure out a rationale for why you deserve it and you figure out better things to do with it than letting it stack up in a savings account and suddenly your needs and feels of deserve are open-ended.

You treat your staff fair and upfront, and you have the best chance at being treated correspondingly even in the face of a layoff. "Best chance" still is no guarantee.

Re:Takeaway

[RockDoctor]

The BOFH has a cattle-prod with your name on it. The PFY will hold you down and line up the welding rods. It's a helluva cattle prod (unsurprisingly).

some admins belong in prison

[v1]

and this is obviously one of them. Criminals come from all walks of life, sysadmin isn't a position immune to containing the occasional bad apple.

So many questions of course, a lot of which boil down to "They must have had some serious lapse in procedure to have allowed this to happen." That's not really the case though. Back doors and logic bombs are serious threats when a person has been a trusted system administrator. Done "right", they can be extremely difficult to detect. It's a bit like the widely accepted advice of "Server was hacked? Don't try to clean it, you might miss something. You must wipe and reinstall it." (same really applies even to a home desktop) A departing admin (on bad OR good terms) is basically the exact same issue, a compromised system, but we only very rarely see such an extreme response. It's much less practical to nuke-n-pave when it's your entire network that is basically now classified as "compromised." Is this how we should respond? When you really stop and think about it, it starts to show itself as a really difficult question to answer. Rebuilding everything when an admin leaves when your system is large is just really hard to justify. But if your system is big, it's also more difficult to review it all and proclaim it "clean". It's just a bad position to be in, and that's why admin departures are such a headache. If you're big enough you have several admins and better compartmentalization of access, more robust isolation of systems, better logging, security software that's under the control of the CIO and not the admins, etc. They have a better chance, but it doesn't look like this one was big enough to have those benefits.

The lack of backups is the most troubling though. That's what stung the other recent post on the cleanout-from-inside. There's just no excuse for that.

Re:some admins belong in prison

[Anonymous Coward]

Even with backups, depending on the role of the departing admin, they may have had the ability to wipe/poison those as well.

Re:some admins belong in prison

[Anonymous Coward]

I worked for a company that locked an programmer's account when he left the company (amicably), and multiple background services & jobs stopped working. Since production was down, they tried to re-activate his account but with a different password. It still didn't work because the saved password didn't match.

They had to call him up, ask for his password, and set it back to the way it was. Management heads rolled after everyone worked through the weekend to set things to use a proper system account.

Re:some admins belong in prison

[Anonymous Coward]

From just a few stories back...

https://developers.slashdot.org/story/17/06/10/0450234/developer-accidentally-deletes-production-database-on-their-first-day-on-the-job

They could be related.

Re:some admins belong in prison

[Anonymous Coward]

and this is obviously one of them. Criminals come from all walks of life, sysadmin isn't a position immune to containing the occasional bad apple.

What's so obvious about it? I can think of several alternative scenarios, including sociopathic management that pushes their employees to and over the limit, or someone looking for a scapegoat and blaming the situation on someone who isn't part of the team anymore. I don't know what actually happened there and neither do you. "Obvious" is not the right word for describing situations you have very little information about.

Re:some admins belong in prison

[Anonymous Coward]

> The lack of backups is the most troubling though.

I'm not aware of any unmanaged VPS service that keeps backups of customer data. The nature of a cheap and unmanaged service is that the client is expected to perform OS management including backups.

Now if they lack backups of their own infrastructure, that's harder to excuse. As someone who has used several VPS providers over the years, and currently has several VPS for personal projects, I know it's entirely on me to make sure my data is backed up.

Sadly far too many people assume that because someone else is hosting it, they're safe to ignore best practices.

Re: some admins belong in prison

[Reverend Green]

Feed the Gulag!

Re:some admins belong in prison

[tmjva]

Even with backups.  Some admins don't know how to restore.  They routinely run backups and file the tapes away or send them to off site storate.  (Mostly they don't even run backups anymore but the automation runs it, they just pack the tapes.)

But ask them to do a restore, you get apoplectic fits as though it you were asking for Form 27b stroke 6 from the movie "Brazil".

Re:some admins belong in prison

[v1]

We have a little bit of that going on here right now. One of the higher admins (that's about due to early retire) has several systems using his account because the system accounts he should be using for that don't work properly. Fortunately he's not the only user that has that access though, so while it can be a rare inconvenience when he's gone (to track down one of the other admins that also has access) it's still not a good idea.

If your admin leaves on amicable terms, have them change their password when they leave. I don't expect them to tell you what their current password is (as many will reuse passwords etc) but they should be able to get on their computer before they walk out, make sure all personal information is out of their keychain (personal info on company computers being another rant for another time) and then start the password change process. Then their manager can sit down and enter a new password he has created.

This has several benefits. You don't have to worry about what the user's password may be locking you out of after they exit the building, AND it works like an account-disable because they don't have their new password. (just make sure you note on the account not to let the user do a password reset by calling the helpdesk!) There may still be some things that need to be individually addressed though such as SSH public key logins that don't rely on the user's directory password. If you're worried about that you can disable the account entirely, and then activate it momentarily if you find you need it for something - get it migrated, and disable it again.

That way, if something DOES end up being locked down under his login, the manager can get at it. It also allows for good tracking of what information required the new password, so they can review handovers and reduce/eliminate future needs with policy changes. This is a better idea than calling the employee a week after they've departed. It's more considerate for the company as well as the employee, and is also more reliable. (there may be hard feelings you were unaware of that give you the finger when you try to call them back)

Seen this happen with several companies now

[charronia]

If at all possible, don't blindly trust your admins. Always keep several backup plans in mind.

Re:Seen this happen with several companies now

[minstrelmike]

When we first built-up our it department, we were all developers but one of us was also sysadmin and I was the dba. And I had other folks back me up as dba and a different developer backed up the sys admin. Then a few years go by and people leave and no one new gets hired and suddenly the sysdamin is the backup dba and I'm the backup sysadmin.

And now my buddy just quit. So now I'm sysdamin, dba, developer, tester, etc.
Be very careful when you tell me I need to train my H1-B replacement.

Re: Seen this happen with several companies now

[Anonymous Coward]

Leave immediately. You know it's the right decision, so what's keeping you?

Re: Seen this happen with several companies now

[HornWumpus]

Seconded.

And never forget: NEVER accept counteroffers!

Re: Seen this happen with several companies now

[Anonymous Coward]

Legit counter offers can be worth accepting. Legit means in writing, specifying, in writing, that you get paid out even if fired for cause., and that the business and boss are jointly and severally liable.

Re: Seen this happen with several companies now

[HornWumpus]

Nope.

Anytime a counteroffer is on the table, you are faced with two choices:

1. A new job, with a brand new basis for all future raises.
2. The same old job, where you just 'extorted' a raise they didn't want to give you.

The best case for scenario #2 is they won't give you another decent raise until the next time you have an offer on the table. The things that really made you want to quit won't change.

This could have happened to me

[Anonymous Coward]

At first I thought this was crazy. But then I started thinking about where I have worked for several years. I'm a Drupal administrator on a site configured so I can write PHP in the body of pages or blocks. Sometimes I get asked to get data out of the DB with db_query() in such a page. So I suppose it's not quite out of the realm of possibility that I could delete the production DB, though it would be hard to make it look like an accident. And last time the site was potentially hacked, people acted like the data in that DB was irreplaceable, although I know they have backups now.

Wait, you only get compensated if

[Anonymous Coward]

you're willing to risk paying them for additional service?

Last time I checked, consumer law usually requires some kind of penalty when you fail to deliver a service people paid for.

Re: Wait, you only get compensated if

[Brockmire]

A lot of service level agreements say provider isn't responsible for lost data. Managed VPS with backup is generally a premium.

huh

[Anonymous Coward]

did somebody piss in his corn flakes?

Re:huh

[Anonymous Coward]

Yep. That was me. Right after six bottles of Ranger IPA and an asparagus salad.

Man, he showed them!

[Hentai007]

I can see that jerks smug face while he sits in prison for the next few years, oh the hours of inconvenience he caused! well worth it my friend, well worth it.

Re:Man, he showed them!

[Anonymous Coward]

Prison is much too harsh a punishment for something like the destruction of data. Also, he had to have known the backup systems were in place, meaning all he intended to do was inconvenience to begin with. At most, he should be charged for the manhours that went into reversing what he did. No more. Go beyond that, and you're just punishing to get yourself off. That mentality is toxic to society.

A new hire deleted everything...

[Anonymous Coward]

Is this related to that earlier reddit news where a new hire deleted everything with rm -rf / and the backups won't work?

captcha: unaware

This won't go will for hosting provider...

[__aaclcg7560]

I was with a one-man ISP for 12 years, starting off with a dial-up UNIX account to hosting my websites. Unfortunately, in 2007, the two lines from different providers to the out of state data center got cut in separate backhoe accidents (what are the odds?). It took ten days for the providers to restore the lines and him to get a third line with a different provider installed. I've already moved my websites to a dedicated hosting provider by day eight. I haven't suffered an outage since then.

Re:This won't go will for hosting provider...

[Anonymous Coward]

This won't go will [sic] for hosting provider

"Company that loses all customer data and disrupts service to customers will probably lose a substantial number of customers."

Gee, ya think? But really, thanks for jumping in with your irrelevant anecdote, Captain Obvious.

I've already moved my websites to a dedicated hosting provider by day eight. I haven't suffered an outage since then.

Too bad. I'd laugh my ass off if your waste of perfectly good electrons disappeared from the internet.

Re:This won't go will for hosting provider...

[__aaclcg7560]

But really, thanks for jumping in with your irrelevant anecdote, Captain Obvious.

You're welcome.

I'd laugh my ass off if your waste of perfectly good electrons disappeared from the internet.

The ad revenues I get from Slashdot readers visiting my websites over the last three months is enough to pay for six months of web hosting. But I'm putting the money towards three months of The Wall Street Journal instead.

Re:This won't go will for hosting provider...

[R3d M3rcury]

Yeah, I gotta admit I saw the, "current customers who are still interested in our services will receive compensation." and thought, "Yeah. The both of them."

Re:This won't go will for hosting provider...

[Anonymous Coward]

To be fair, only 2 days of data appears to have been lost from this admin's stunt. If their customers honestly cared about their data, the hosting provider would only be one of several mirrors or other types of backups. If their work was important, it'd be backed up daily to multiple locations, on- and off-site.

It's great for this company to offer some sort of compensation to their customers. It'll probably be X months of hosting free, or an extra feature for a year or something like that.

You're right though, some may see this as unacceptable and take their business elsewhere. But at least they're owning the problem. That's more than I can say about most business.

Re:This won't go will for hosting provider...

[eWarz]

Slashdot must really be going down hill then. I remember when a link from Slashdot meant a windfall. A HUGE windfall.

Re:This won't go will for hosting provider...

[__aaclcg7560]

Slashdot must really be going down hill then. I remember when a link from Slashdot meant a windfall. A HUGE windfall.

If I was doing this ten years ago, ad revenues from Slashdot traffic would have been in the hundreds of dollars.

Re:This won't go will for hosting provider...

[Anonymous Coward]

"towards three months of The Wall Street Journal instead."

Aren't there public libraries where you are?

Yup.

https://www.google.com/maps/d/viewer?mid=14KVDs8w0_7nvkCF8VdrPN3wcWCY&hl=en_US&ll=37.225867831442486%2C-121.86616100000003&z=10

Re:This won't go will for hosting provider...

[__aaclcg7560]

Aren't there public libraries where you are?

I read the WSJ on my iPhone on the express bus to my tech job in Palo Alto.

Re:This won't go will for hosting provider...

[Anonymous Coward]

...libraries can loan electronically... At least they do in North Canuckistan.

There is one KEY word missing in the Press Release

[KingRatMass]

Allegedly

The ex-admin allegedly deleted all the data... Until is has been thoroughly investigated and it can be proven, the company has made a potentially libelous statement. I don't know how defamation laws work in Europe but no semi-competent General Counsel would not have let a US corporation make such a stupid statement in a press release.

Somebody could have used the ex-admin credentials, an external bad actor or someone within the company looking to cover something up. The company may very well be attempting to pull off an elaborate insurance fraud claim.

Can't Recover

[barbariccow]

Can't recover? What did he do, dd if=/dev/zero of=/dev/fs ? Or were they using something like NTFS? Or most likely: storing the data in the CLOUD.

Pretty well every linux filesystem has recovery tools. There's a reason the POSIX term for "delete file" is "unlink". Because you aren't clearing the data, you're just unlinking from the table.

Since pretty well every file has a MAGIC at its start, it becomes fairly dooable to recover.

Not a big deal-salting the wound.

[Anonymous Coward]

Could you explain "salt?" It's new to me but from your context I should know it. (A link would be fine).

Yup--as others posted below, 'salt stack'. It's pretty much like 'Puppet', 'Chef', or 'Ansible'. Set up a 'salt mater' and point all your 'salt minions' to the master. Then you can define the way you want your systems to be configured from the master. i.e. things like disabling SSH password auth, deploying authorized SSH keys, configuring firewalls, cron jobs, packages installed, etc...

Sort of every bot master's dream tool.

Good-Home businesses

[Anonymous Coward]

No local expertise, especially for small, and home-based businesses. The cloud is perfect for them.

Re:There is one KEY word missing in the Press Rele

[Anonymous Coward]

No, this hasn't happened in the paranoid CYA litigious States, but in a small country in Europe where we don't care about that crap, and call the things what they are.

Good guys and bad guys

[Michael Woodhams]

In a certain sort of movie (e.g. Mad Max, The Crow) the difference between good guys and bad guys is the order in which they commit their atrocities. In these two stories, good guys delete the data and then get fired, bad guys get fired and then delete the data.

Re:Good guys and bad guys

[RockDoctor]

In a certain sort of movie (e.g. Mad Max, The Crow) the difference between good guys and bad guys

There was a good guy in Mad Max? That's not the impression I got from seeing bits of it. Oh, I see - that's your point. OK : quod erat demonstratum.

Re:There is one KEY word missing in the Press Rele

[Megol]

Libel laws are generally much stronger in Europe compared to the US. So yet another Anonymous idiot writing crap. So surprising...

spanking clean

[Anonymous Coward]

What? Like with a cloth?

Sounds familiar...

[iq145]

https://developers.slashdot.or...