...Not to mention a firmware update that bricked 1.6TB of my data, in a completely unrecoverable way. I was running a Drobo v2 with v.1.31 and had rebooted it hundreds of times in the last 2 years. There was a new firmware update that claimed to fix some performance issues. I upgraded the firmware (using the approved Windows method).
The device never booted again. It's been bricked for months now, and my data, while still striped across the platters, is held hostage by the Drobo device. Downgrading the firmware isn't possible, because the moment the firmware is updated (before the initial reboot of the device), the disk pack itself is upgrade to that same firmware revision.
There is no going back... and DRI openly states that I'm screwed, and there's nothing they can do. They can't even give me the "Last Resort Firmware" that they hand out in cases just like this. I've opened plenty of cases with them about it, and their response is "Sorry, you should have had your data on another Drobo as a backup."
I will never use a proprietary, black-box data storage solution again, ever.
10 years of digital photos, dozens of system backups, thousands of scanned documents long since gone, my entire music collection, etc. all stored on a device that claimed to be completely safe to store it.
Avoid Drobo at all costs, if you care about your data.
If you have a real solution to those two issues, I'd love to hear about it.
Well first, some of those Firefox add-ons actually send an encrypted blob through the web-based email system... so if you don't have the add-on, you get ascii-armored jibberish, with legible headers.
Also, the systems you mention all support IMAP and POP3, so you can use the mail client of your choice to interact with them (Evolution, Thunderbird, Outlook, OE, Mail.app, etc.)
If you send your email to somebody (the "third party") that somebody can choose to hand it over to anyone.
This is PRECISELY why you encrypt emails to recipients... there is absolutely no doubt that there was an expectation of privacy, when the receiver has to decrypt the email using a private key, to read it.
I've said this many times here before, and I'll say it again... don't let them see anything other than the delivery envelope (headers) of your email. They can't legally open your postal mail, so treat it the same: gpg/PGP-encrypt your emails; all of them.
If a recipient you email frequently doesn't know how to use encryption, teach them. There are plugins for Firefox, Gmail, Thunderbird, Mail.app, and dozens of other mail clients.
If it's someone you don't converse over email with often, then it's probably not worth protecting anyway.
Learn to create, protect and use your gpg keys and your keychain. It's not that hard, and the benefits far outweigh the minutes of work and learning it takes to incorporate it into your daily workflow.
The FBI has no need or right to know about my IT skills, but I would certainly like to know more about their IT skills, given all of the obvious and avoidable breaches in silicon-based security, document controls and methodologies lately.
I appreciate your concern, but sendmail is definitely not the issue causing the load to skyrocket. I've already tested that by shutting down sendmail right before the window when I know it's going to happen, and it happens anyway. Besides, if sendmail is already refusing connections due to the load, then sendmail itself isn't the problem.
I'm already 30GB into the rsync of the data. I run rsnapshot on an hourly basis to my Drobo here for near-line backups, but the Drobo just ate 1.5TB of my data (dangerous device to rely on, and the company line is to always back up your Drobo to... another Drobo). So I have to pull the data clean from the server to another storage array here, otherwise the backup would have been incremental.
Once that is done, I'll be terminating my contract. These rampant, unexplained outages are infuriating, and my users, clients and customers are pissed off, and so am I.
"When there is a problem, you call the landlord to come fix it. Yes, you have the right of sole use of the property, but you can't reasonably expect him to fix the problem without allowing him access to the property."
To continue with your analogy, this is like telling the landlord that there's a problem with the lawn, and he demands to have a copy of the keys to my home office desk drawers to fix it.
Yes, they "rent" a KVM to customers for $35.00/USD for a 24-hour period, unfortunately...
In this case, to break the standoff between myself and the hosting provider, I yielded and had them invoice me for the $35 so I could get the server up, rip the data off of it, terminate my services with them and go after them for financial compensation for the damages, downtime (12 day outage 2 months ago without an apology), etc.
It's both hobby, personal and business. The server hosts ~300 public websites, as well as source code repositories, mail and mailing lists for about a dozen of those projects.
"In any event, tuning should be able to prevent that from knocking the box over completely, allowing you to stay logged in and see what's going on."
If absolutely nothing changed other than the IP and physical datacenter the hardware was located in, and the problems every other Sunday only started after the physical machine was relocated, how could it possibly be the OS or applications?
The network graphs clearly show external activity flooding the machine with connections that never complete. I'd show you the graphs, but my server is down at the moment.:(
I understand where you're coming from, but knowing that it's every other Sunday, I've even shut down any and all apps, cron, etc. and just watched it, and it still happens, consistently. It's like clockwork, and it's not the OS or its configuration, because this never happened when I was in the old dc.
"Stop being a jerk and cooperate with the owners of the machine you are renting or take your data elsewhere."
Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.
Basically I am leasing a physical server from company (A) who is leasing it from company (B), and that too may not be the end of the line. (B) may not own it either, and they may be colocating hardware from company (C) or (D) somewhere in there.
So whose TOS am I subject to here? Who is violating whose laws? It gets curiouser and curiouser the more I dig into it.
"If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on."
This IS an unmanaged plan. All the provide is ping and power, I do the rest. I manage the OS, the configuration and everything else. This is not VPS, I lease a physical server, and they don't touch it.
"I trust the techs of the company I'm hosting with so I don't mind giving up root access to chase this problem down. What I do after that is change the root pass again and I'm done."
How am I expected to change the root password to let them in, when they've denied me access to the server unless I hand over the current root password? They're not asking for logs, they're demanding the root password; those are two very-different issues entirely.
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
"Give them an account with limited sudo access to view your logs."
I can't do that, since they are now prohibiting me access to my server unless I hand over the root password. They're not asking for logs, they're asking for the password to the 'root' account.
"If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for."
I pay several thousand dollars a year for their disgusting service, and am going to be migrating away ASAP. Again, they're not asking for logs, they're demanding root. Two very different things.
Right now, they won't give me KVM access so I can log in remotely and fix the networking they broke, so I can get my server back online. It's been down 2 days now because of this.
"Switch providers. Plenty offer remote reboot and serial console or KVM for both VMs or physical servers, which would allow you to go crazy with custom encrypted partitions etc."
They offer KVM access, at $35.00/day, which in this case I refuse to pay to fix what they broke, outside of the context of the server. They migrated me from one chassis to another with completely different hardware, causing my machine to go offline. They want me to pay $35.00 for 24-hours of KVM access to reconfigure the network to support the hardware they moved things to.
Alternately, they want me to hand over the root password (not a privileged account, but THE root password), so they can do it themselves. Since I installed, configured and manage the OS entirely on this machine, and they've demonstrated their ineptitude before, I'm not giving them root. Ever.
"I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?"
Because this server has been running 24x7 for about 3 years without a single outstanding issue. When they migrated it from Savvis to some datacenter in Dallas 2 months ago, I've had no less than 20 separate outages, while the underlying OS and application stack itself has not changed in any way to facilitate those outages.
In every single case, they demand that I give them the root password, so they can diagnose the issues on the machine. In every single case, I've shown them nagios, ntop, hotsanic, sar, etc. logs demonstrating that the OS itself is not the cause of the outages.
For example, since this migration to Dallas, every other Sunday between 7:00am and 8:00am EST, my server's load goes over 100 as incoming connections spike over 700/sec., sendmail refuses connections due to the load, and the box seizes up. The logs show that the connections are established and then hang. NOTHING on the machine triggers every other Sunday between these hours that would cause that.
Only a few days ago, they indicated that the NIC on the server may be causing the issues. I'm down 2-3 hours every other Sunday because of this.
They're not asking for the logs, they're asking for root. That's a completely separate (and unacceptable) solution to their own problems outside of the box itself.
"How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access?"
They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.
"I would assume any reasonable host would be willing to get you a similar sort of hookup."
In this case, it appears the PSU failed, and they moved my drive to a different chassis, with completely different hardware, and are asking for the root password so they can reconfigure everything to coincide with that hardware change.
They want to charge me $35.00/24-hour acccess to a KVM, so I can go in and fix the networking they broke by changing the hardware around the leased server in the dc. I flatly refused to take ownership of that, since they did not tell me beforehand that they'd be swapping out the entire physical chassis, and I don't think I should have to pay $35.00 for 24-hours of KVM use when it'll take me less than 2 minutes to fix it.
They caused the problem, they "downgraded" the hardware to a different chassis, and they're holding my data hostage until I either give them root to go poking around (which I flatly refuse to do, as it violates my company policy), or pay them to fix what they broke.
...except setting the important data partitions to be dm-crypt, which means they can root the machine all they want, but without the passphrase to the dm-crypt partitions, they won't get to any client, customer or confidential data (i.e. transactions in the SQL db)
"As the above poster said, either create a limited account for them with only log file access, or else man up and just give them a full login."
I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.
Slashdot Mirror
...Not to mention a firmware update that bricked 1.6TB of my data, in a completely unrecoverable way. I was running a Drobo v2 with v.1.31 and had rebooted it hundreds of times in the last 2 years. There was a new firmware update that claimed to fix some performance issues. I upgraded the firmware (using the approved Windows method).
The device never booted again. It's been bricked for months now, and my data, while still striped across the platters, is held hostage by the Drobo device. Downgrading the firmware isn't possible, because the moment the firmware is updated (before the initial reboot of the device), the disk pack itself is upgrade to that same firmware revision.
There is no going back... and DRI openly states that I'm screwed, and there's nothing they can do. They can't even give me the "Last Resort Firmware" that they hand out in cases just like this. I've opened plenty of cases with them about it, and their response is "Sorry, you should have had your data on another Drobo as a backup."
I will never use a proprietary, black-box data storage solution again, ever.
10 years of digital photos, dozens of system backups, thousands of scanned documents long since gone, my entire music collection, etc. all stored on a device that claimed to be completely safe to store it.
Avoid Drobo at all costs, if you care about your data.
Windows-only, yecch!
Try KeePassX... much better, cross-platform, free, secure and has a great generator built right into it.
http://www.keepassx.org/
Yep, and I still have my 'hacker' username too! :)
What a coincidence... 10+ years of my collected Palm gadgets are up for sale too.. make me an offer :)
There, fixed that for you :)
Well first, some of those Firefox add-ons actually send an encrypted blob through the web-based email system... so if you don't have the add-on, you get ascii-armored jibberish, with legible headers.
Also, the systems you mention all support IMAP and POP3, so you can use the mail client of your choice to interact with them (Evolution, Thunderbird, Outlook, OE, Mail.app, etc.)
This is PRECISELY why you encrypt emails to recipients... there is absolutely no doubt that there was an expectation of privacy, when the receiver has to decrypt the email using a private key, to read it.
And what happens when systems go down? Power goes out? Electronic transactions are blocked/denied/lost for any reason? What then?
No, no, I'm afraid paper money (currency, checks, notes) will be here to stay, for many, many decades to come.
I've said this many times here before, and I'll say it again... don't let them see anything other than the delivery envelope (headers) of your email. They can't legally open your postal mail, so treat it the same: gpg/PGP-encrypt your emails; all of them.
If a recipient you email frequently doesn't know how to use encryption, teach them. There are plugins for Firefox, Gmail, Thunderbird, Mail.app, and dozens of other mail clients.
If it's someone you don't converse over email with often, then it's probably not worth protecting anyway.
Seriously...
Learn to create, protect and use your gpg keys and your keychain. It's not that hard, and the benefits far outweigh the minutes of work and learning it takes to incorporate it into your daily workflow.
The FBI has no need or right to know about my IT skills, but I would certainly like to know more about their IT skills, given all of the obvious and avoidable breaches in silicon-based security, document controls and methodologies lately.
I appreciate your concern, but sendmail is definitely not the issue causing the load to skyrocket. I've already tested that by shutting down sendmail right before the window when I know it's going to happen, and it happens anyway. Besides, if sendmail is already refusing connections due to the load, then sendmail itself isn't the problem.
I'm already 30GB into the rsync of the data. I run rsnapshot on an hourly basis to my Drobo here for near-line backups, but the Drobo just ate 1.5TB of my data (dangerous device to rely on, and the company line is to always back up your Drobo to... another Drobo). So I have to pull the data clean from the server to another storage array here, otherwise the backup would have been incremental.
Once that is done, I'll be terminating my contract. These rampant, unexplained outages are infuriating, and my users, clients and customers are pissed off, and so am I.
To continue with your analogy, this is like telling the landlord that there's a problem with the lawn, and he demands to have a copy of the keys to my home office desk drawers to fix it.
Yes, they "rent" a KVM to customers for $35.00/USD for a 24-hour period, unfortunately...
In this case, to break the standoff between myself and the hosting provider, I yielded and had them invoice me for the $35 so I could get the server up, rip the data off of it, terminate my services with them and go after them for financial compensation for the damages, downtime (12 day outage 2 months ago without an apology), etc.
It's both hobby, personal and business. The server hosts ~300 public websites, as well as source code repositories, mail and mailing lists for about a dozen of those projects.
If absolutely nothing changed other than the IP and physical datacenter the hardware was located in, and the problems every other Sunday only started after the physical machine was relocated, how could it possibly be the OS or applications?
The network graphs clearly show external activity flooding the machine with connections that never complete. I'd show you the graphs, but my server is down at the moment. :(
I understand where you're coming from, but knowing that it's every other Sunday, I've even shut down any and all apps, cron, etc. and just watched it, and it still happens, consistently. It's like clockwork, and it's not the OS or its configuration, because this never happened when I was in the old dc.
Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.
Basically I am leasing a physical server from company (A) who is leasing it from company (B), and that too may not be the end of the line. (B) may not own it either, and they may be colocating hardware from company (C) or (D) somewhere in there.
So whose TOS am I subject to here? Who is violating whose laws? It gets curiouser and curiouser the more I dig into it.
This IS an unmanaged plan. All the provide is ping and power, I do the rest. I manage the OS, the configuration and everything else. This is not VPS, I lease a physical server, and they don't touch it.
How am I expected to change the root password to let them in, when they've denied me access to the server unless I hand over the current root password? They're not asking for logs, they're demanding the root password; those are two very-different issues entirely.
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
I can't do that, since they are now prohibiting me access to my server unless I hand over the root password. They're not asking for logs, they're asking for the password to the 'root' account.
I pay several thousand dollars a year for their disgusting service, and am going to be migrating away ASAP. Again, they're not asking for logs, they're demanding root. Two very different things.
Right now, they won't give me KVM access so I can log in remotely and fix the networking they broke, so I can get my server back online. It's been down 2 days now because of this.
They offer KVM access, at $35.00/day, which in this case I refuse to pay to fix what they broke, outside of the context of the server. They migrated me from one chassis to another with completely different hardware, causing my machine to go offline. They want me to pay $35.00 for 24-hours of KVM access to reconfigure the network to support the hardware they moved things to.
Alternately, they want me to hand over the root password (not a privileged account, but THE root password), so they can do it themselves. Since I installed, configured and manage the OS entirely on this machine, and they've demonstrated their ineptitude before, I'm not giving them root. Ever.
Because this server has been running 24x7 for about 3 years without a single outstanding issue. When they migrated it from Savvis to some datacenter in Dallas 2 months ago, I've had no less than 20 separate outages , while the underlying OS and application stack itself has not changed in any way to facilitate those outages.
In every single case, they demand that I give them the root password, so they can diagnose the issues on the machine. In every single case, I've shown them nagios, ntop, hotsanic, sar, etc. logs demonstrating that the OS itself is not the cause of the outages.
For example, since this migration to Dallas, every other Sunday between 7:00am and 8:00am EST, my server's load goes over 100 as incoming connections spike over 700/sec., sendmail refuses connections due to the load, and the box seizes up. The logs show that the connections are established and then hang. NOTHING on the machine triggers every other Sunday between these hours that would cause that.
Only a few days ago, they indicated that the NIC on the server may be causing the issues. I'm down 2-3 hours every other Sunday because of this.
They're not asking for the logs, they're asking for root. That's a completely separate (and unacceptable) solution to their own problems outside of the box itself.
They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.
In this case, it appears the PSU failed, and they moved my drive to a different chassis, with completely different hardware, and are asking for the root password so they can reconfigure everything to coincide with that hardware change.
They want to charge me $35.00/24-hour acccess to a KVM, so I can go in and fix the networking they broke by changing the hardware around the leased server in the dc. I flatly refused to take ownership of that, since they did not tell me beforehand that they'd be swapping out the entire physical chassis, and I don't think I should have to pay $35.00 for 24-hours of KVM use when it'll take me less than 2 minutes to fix it.
They caused the problem, they "downgraded" the hardware to a different chassis, and they're holding my data hostage until I either give them root to go poking around (which I flatly refuse to do, as it violates my company policy), or pay them to fix what they broke.
...except setting the important data partitions to be dm-crypt, which means they can root the machine all they want, but without the passphrase to the dm-crypt partitions, they won't get to any client, customer or confidential data (i.e. transactions in the SQL db)
I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.
I find these to be unacceptable terms.