Believe me, you can be sued for telling the truth. I've been sued for telling the truth. Read the first few entries of http://thespamdiaries.blogspot.com/ for the whole miserable story.
The only thing the truth does for you is help you win if you have good lawyers, the money to pay them, and an unbiased judge.
I was lucky in that I had all three so it only cost me a year's salary. My co-defendant wasn't so lucky.
Hoo boy, the stories I could tell. Actually, I can't, because the hosting provider threatened to sue us if we named them publicly.
OK, first, if there's more than a couple of servers involved, and your business depends on it, use two or more different providers. If you only have one provider, it puts them in a position to screw you. When we terminated our relationship with our provider, they held our data hostage until we paid them an additional $15,000 to put our servers on line again long enough for us to copy our data.
Which brings us to: DO YOUR OWN BACKUPS. Service providers either don't do them, or they don't do them right. The world is full of horror stories of customers paying the data center extra for backups, and then finding the backups were never actually done. And even if they do do backups, they maintain control of them, which puts them in a good position to extort you.
Remember, the practice of holding your data hostage goes back a long way. Happened to my father's company back in the 70's mainframe days. It still happens.
Most important of all: have a professional go over your data center contract with a fine-tooth comb. The default contract they'll give you (or at least the one they gave us) is highly abusive.
For instance: if you don't explicitly terminate a contract at the end of its period, it's automatically renewed for another 18 months. You need to give 2 months notice before the end of the term before canceling. There is no early termination. If you so much as upgrade a single disk drive, the contract is automatically renewed for another 18 months.
Here's a doozy: our contract specified that if a server went down, they would either fix or replace it within two hours of determining the problem. The catch: they merely have to say that they haven't determined the problem yet, and then they don't have to replace anything. Our main server was kept off-line for a month this way.
I think that once your server farm exceeds 100 servers, it's time to start seriously thinking of re-architecting in C. That's probably about the tipping point where your cost of build-out will exceed the engineering costs of switching technologies.
I invested in a small startup a couple of years ago. We based everything on Ruby-on-Rails, php, mysql, and I forget what else. The bitter lesson we learned -- and what was the main cause of the company's failure -- was that this technology doesn't scale. We nursed it along to 30M records on two servers (the db admins must've been geniuses to coax it that far), but in the end, it fell over.
We could have saved the company if we could have afforded to expand our servers on our shoestring budget, but in the end, the software infrastructure we were using would have failed us one way or another.
I've also worked for companies that scaled successfully, and they were the ones that got it. Switched from scripting languages to C++. Switched from MySQL to a dedicated database engine.
Scripting languages are great for development and prototyping, but for serious production use, you really need to bite the bullet and switch to compiled languages.
You can go to Chilling Effects and read up on many, many cases of Google censorship. The bottom line is that Google gets something like 1000 C&D notices per week, and they just can't afford to fight them, so in most cases, Google just immediately rolls over and complies.
While everybody else is suggesting technical things to look at, I'm going to suggest that you look at the legal things instead.
Read the contract with a magnifying glass. In fact, have a professional with experience look at it for you. The contracts often have traps in them that it takes a trained professional to spot.
For example: one managed hosting provider had it in their contract that they would replace any dead servers within 2 hours of determining what the problem is. Catch is, all they had to do was say "we haven't determined what the problem is yet". Kept our main server off line for nearly a month while the night crew and the day crew argued over what needed to be done.
Contract said that if you so much as upgraded a disk drive, that re-upped the contract for eighteen months. Think about that for a bit. We couldn't afford to extend our contract for another 18 months and we couldn't do a thing about our overloaded server situation. You also had to give 60 days notice before the end of the contract or it was automatically renewed.
Also: never go with just one provider. Diversify. If all your data is in one data center, they can hold it hostage. Happened to my dad in the mainframe days, happened to me last year. Some things never change.
I could go on for hours, but I need to watch my blood pressure. Or my cholesterol. Something like that anyway.
I'm not happy to see more government interference in the internet, but I think the ISPs have this coming. Spam and online fraud exists because the ISPs choose to tolerate it. If they would do the right thing and get rid of their bad-actor customers, the government wouldn't need to get involved.
I'm not sure the problem is that the administrators enjoy their BOFH roles so much as that anyone who runs an anti-spam service is probably someone who's already really, really frustrated with the spam problem, and with dealing with lazy, greedy and incompetent ISPs.
Another thing to consider is that the anti-spam services work for the clients who want the spam stopped, and they don't work for the ISPs that are generating the spam. So they tend to get testy when those ISPs ask for help dealing with their own problem.
The anti-spam service's duty to the spamming ISPs ends with "fix it your own damn self". In the end, it may be counter-productive, but there you have it.
Wallace is the guy that invented mass email spam. If his ISP had shut him down on day 1, the world might be a different place today. Spam exists because ISPs tolerate it.
It is almost unheard of for a plaintiff to collect money from a spammer. They're either broke, or they've successfully hidden their money. The $711M judgment is purely symbolic. Facebook knows full well that they'll never collect a dime.
I agree that jail time is the only solution. Wallace is the recidivist's recidivist.
Absolutely. If it costs more than twice the price of 50% efficient cells, then its usefulness is limited to a few weight or surface area -critical applications.
Bruce Schneier examined a similar device a few years ago. Turns out that all you needed to do was take it apart and cut the red wire. Voila, no more self destruct.
You are correct. Applications still run as the logged-in user. The only vulnerability is if there was some sort of security hole in the server, and we all know that could never, ever happen.
I was a video driver developer for Sun for many years. The window system *always* ran as the logged-in user. When I started developing for Linux, I was appalled when I realized that Linux ran the windows server as root.
Here's how we did it at Sun: For every supported video card, there is a device driver. The driver provides basic services such as cursor and color-table management (there are advantages to doing this in the kernel), and additionally allows the user logged in at the console to map in the device registers. This means that the window system doesn't need any special privileges to run.
There are other advantages to having a device driver manage user-level hardware mapping. Not the least of which is that it allowed us to implement full-bore context switching at the device level. The advantages of this are enormous.
And yet Debian Gtk chose to recently arbitrarily rename the glib package, breaking binary compatibility. Why? Who knows? Will they ever fix it? Who knows?
Why does this Linux community have such a deep and abiding hatred of backwards compatibility? Library versions, device drivers, audio systems, hot-plugging, device naming, anything even remotely related to multimedia. This list goes on and on.
Until the Linux community decides to settle on some standards, it will never be ready for the end-user desktop.
I agree with almost every point in this article. I have been a dedicated Linux user since RH 4.2
My observations about why Linux is not ready for the desktop:
1) Lack of compatibility between versions.
I can't say enough about how frustrating this is. Every time I upgrade versions, something breaks. Usually audio. In fact, most multimedia functionality breaks every time I upgrade. I generally find that the/dev/cdrom symlink is broken at the very least, but I've frequently found that all of my CD writer scripts have to be modified.
Recently, Ubuntu arbitrarily renamed the "libglib1.2" package, breaking every application that links against the GTK+ library. Why? No answer.
It's as if Linux is actively hostile to the concept of backwards compatibility.
2) Lack of support for hot-plugging. (point 13 in the article)
I plug in a thumb drive or usb hard drive and maybe the OS will notice it and mount it for me, and maybe it won't. Usually it doesn't. Usually, I have to become super-user and perform actions to identify the drive and mount it that would be beyond the knowledge of the average end user. And even if the user does know how to do it, why should they have to? A 10-second task just got turned into a 5-minute task.
USB scanners are the same way. They used to work, now you have to become super-user to use them. Some script that detected scanner plugin events and change the permissions just stopped working.
Multi-card readers: Same thing. Sometimes they work, sometimes they don't.
Windows, Solaris, and MacOS all solved the hot-plugging problem years ago, why can't Linux?
3. Hardware support regression
Mentioned in the article, but worth repeating. I really hate upgrading my OS and discovering that some of my existing hardware is no longer supported. Recent discovery: you can have USB1 or USB2 enabled, but not both at the same time. If you want USB1, remove the ehci_hcd module. If you want USB2, install it.
See Bugzilla, Launchpad.net. It seems unlikely this bug will ever be fixed.
One of the best ideas I ever heard was that someone should generate fake tiger bones, rhinoceros horns and other artifacts of endangered species and flood the markets with them. This would cause the prices to collapse, making it no longer economical to be a poacher.
OK, seriously, what is it with the Linux community and arbitrarily renaming packages? It's like they *heard* of software compatibility, but decided they wanted no truck with it.
Recently they renamed "libglib1.2" to "libglib1.2dbl" in Debian (although the dev library was not renamed), breaking anything linked against glib. For what possible purpose was this done?
A CEO friend of mine once said "It's never too soon to fire someone".
A person like "Josh", no matter how brilliant they are, can actually produce negative work as a side-effect of the chaos they sew. They may be productive as hell, but their impact on the productivity of the people around them can cause more harm than good.
My friend had to fire a number of people over the years who caused big problems. He never regretted firing someone, but he often regretted not doing it sooner.
Oh, and by the way, writing "clever" code just produces code your peers (and you, six months later) can't maintain. I'll bet Josh's bosses regret not requiring code reviews.
Oh, wow, a real rebuke! That'll learn 'em a lesson.
How about awarding costs to the victims instead?
People complain about the overburdened legal system, but it seems to me that the courts like frivolous and harassing lawsuits because they mean more work for lawyers and judges.
If the courts really wanted to do something about the clogging of the legal system, they'd award costs to innocent defendants.
Slashdot Mirror
Believe me, you can be sued for telling the truth. I've been sued for telling the truth. Read the first few entries of http://thespamdiaries.blogspot.com/ for the whole miserable story.
The only thing the truth does for you is help you win if you have good lawyers, the money to pay them, and an unbiased judge.
I was lucky in that I had all three so it only cost me a year's salary. My co-defendant wasn't so lucky.
Lawsuits cost money, even if you're in the right.
Hoo boy, the stories I could tell. Actually, I can't, because the hosting provider threatened to sue us if we named them publicly.
OK, first, if there's more than a couple of servers involved, and your business depends on it, use two or more different providers. If you only have one provider, it puts them in a position to screw you. When we terminated our relationship with our provider, they held our data hostage until we paid them an additional $15,000 to put our servers on line again long enough for us to copy our data.
Which brings us to: DO YOUR OWN BACKUPS. Service providers either don't do them, or they don't do them right. The world is full of horror stories of customers paying the data center extra for backups, and then finding the backups were never actually done. And even if they do do backups, they maintain control of them, which puts them in a good position to extort you.
Remember, the practice of holding your data hostage goes back a long way. Happened to my father's company back in the 70's mainframe days. It still happens.
Most important of all: have a professional go over your data center contract with a fine-tooth comb. The default contract they'll give you (or at least the one they gave us) is highly abusive.
For instance: if you don't explicitly terminate a contract at the end of its period, it's automatically renewed for another 18 months. You need to give 2 months notice before the end of the term before canceling. There is no early termination. If you so much as upgrade a single disk drive, the contract is automatically renewed for another 18 months.
Here's a doozy: our contract specified that if a server went down, they would either fix or replace it within two hours of determining the problem. The catch: they merely have to say that they haven't determined the problem yet, and then they don't have to replace anything. Our main server was kept off-line for a month this way.
This happens all the time in the U.S. The government even shut down a Spanish travel agency that arranged tours to Cuba -- they were foolish enough to register their domain name here. http://thespamdiaries.blogspot.com/2008/03/dont-register-or-host-your-domain-in-us.html
I agree.
In my experience, the manager's primary -- and arguably only -- job is to act as a barrier between engineers and paperwork and other distractions.
I think that once your server farm exceeds 100 servers, it's time to start seriously thinking of re-architecting in C. That's probably about the tipping point where your cost of build-out will exceed the engineering costs of switching technologies.
I invested in a small startup a couple of years ago. We based everything on Ruby-on-Rails, php, mysql, and I forget what else. The bitter lesson we learned -- and what was the main cause of the company's failure -- was that this technology doesn't scale. We nursed it along to 30M records on two servers (the db admins must've been geniuses to coax it that far), but in the end, it fell over.
We could have saved the company if we could have afforded to expand our servers on our shoestring budget, but in the end, the software infrastructure we were using would have failed us one way or another.
I've also worked for companies that scaled successfully, and they were the ones that got it. Switched from scripting languages to C++. Switched from MySQL to a dedicated database engine.
Scripting languages are great for development and prototyping, but for serious production use, you really need to bite the bullet and switch to compiled languages.
You can go to Chilling Effects and read up on many, many cases of Google censorship. The bottom line is that Google gets something like 1000 C&D notices per week, and they just can't afford to fight them, so in most cases, Google just immediately rolls over and complies.
While everybody else is suggesting technical things to look at, I'm going to suggest that you look at the legal things instead.
Read the contract with a magnifying glass. In fact, have a professional with experience look at it for you. The contracts often have traps in them that it takes a trained professional to spot.
For example: one managed hosting provider had it in their contract that they would replace any dead servers within 2 hours of determining what the problem is. Catch is, all they had to do was say "we haven't determined what the problem is yet". Kept our main server off line for nearly a month while the night crew and the day crew argued over what needed to be done.
Contract said that if you so much as upgraded a disk drive, that re-upped the contract for eighteen months. Think about that for a bit. We couldn't afford to extend our contract for another 18 months and we couldn't do a thing about our overloaded server situation. You also had to give 60 days notice before the end of the contract or it was automatically renewed.
Also: never go with just one provider. Diversify. If all your data is in one data center, they can hold it hostage. Happened to my dad in the mainframe days, happened to me last year. Some things never change.
I could go on for hours, but I need to watch my blood pressure. Or my cholesterol. Something like that anyway.
I'm not happy to see more government interference in the internet, but I think the ISPs have this coming. Spam and online fraud exists because the ISPs choose to tolerate it. If they would do the right thing and get rid of their bad-actor customers, the government wouldn't need to get involved.
I'm not sure the problem is that the administrators enjoy their BOFH roles so much as that anyone who runs an anti-spam service is probably someone who's already really, really frustrated with the spam problem, and with dealing with lazy, greedy and incompetent ISPs.
Another thing to consider is that the anti-spam services work for the clients who want the spam stopped, and they don't work for the ISPs that are generating the spam. So they tend to get testy when those ISPs ask for help dealing with their own problem.
The anti-spam service's duty to the spamming ISPs ends with "fix it your own damn self". In the end, it may be counter-productive, but there you have it.
Wallace is the guy that invented mass email spam. If his ISP had shut him down on day 1, the world might be a different place today. Spam exists because ISPs tolerate it.
It is almost unheard of for a plaintiff to collect money from a spammer. They're either broke, or they've successfully hidden their money. The $711M judgment is purely symbolic. Facebook knows full well that they'll never collect a dime.
I agree that jail time is the only solution. Wallace is the recidivist's recidivist.
Absolutely. If it costs more than twice the price of 50% efficient cells, then its usefulness is limited to a few weight or surface area -critical applications.
Tarballs solve the problem nicely. Even on a linux-formatted drive, file attributes (especially timestamps) can be lost.
Bruce Schneier examined a similar device a few years ago. Turns out that all you needed to do was take it apart and cut the red wire. Voila, no more self destruct.
I hope this new device does a better job.
You are correct. Applications still run as the logged-in user. The only vulnerability is if there was some sort of security hole in the server, and we all know that could never, ever happen.
I was a video driver developer for Sun for many years. The window system *always* ran as the logged-in user. When I started developing for Linux, I was appalled when I realized that Linux ran the windows server as root.
Here's how we did it at Sun: For every supported video card, there is a device driver. The driver provides basic services such as cursor and color-table management (there are advantages to doing this in the kernel), and additionally allows the user logged in at the console to map in the device registers. This means that the window system doesn't need any special privileges to run.
There are other advantages to having a device driver manage user-level hardware mapping. Not the least of which is that it allowed us to implement full-bore context switching at the device level. The advantages of this are enormous.
And yet Debian Gtk chose to recently arbitrarily rename the glib package, breaking binary compatibility. Why? Who knows? Will they ever fix it? Who knows?
Why does this Linux community have such a deep and abiding hatred of backwards compatibility? Library versions, device drivers, audio systems, hot-plugging, device naming, anything even remotely related to multimedia. This list goes on and on.
Until the Linux community decides to settle on some standards, it will never be ready for the end-user desktop.
I agree with almost every point in this article. I have been a dedicated Linux user since RH 4.2
My observations about why Linux is not ready for the desktop:
1) Lack of compatibility between versions.
I can't say enough about how frustrating this is. Every time I upgrade versions, something breaks. Usually audio. In fact, most multimedia functionality breaks every time I upgrade. I generally find that the /dev/cdrom symlink is broken at the very least, but I've frequently found that all of my CD writer scripts have to be modified.
Recently, Ubuntu arbitrarily renamed the "libglib1.2" package, breaking every application that links against the GTK+ library. Why? No answer.
It's as if Linux is actively hostile to the concept of backwards compatibility.
2) Lack of support for hot-plugging. (point 13 in the article)
I plug in a thumb drive or usb hard drive and maybe the OS will notice it and mount it for me, and maybe it won't. Usually it doesn't. Usually, I have to become super-user and perform actions to identify the drive and mount it that would be beyond the knowledge of the average end user. And even if the user does know how to do it, why should they have to? A 10-second task just got turned into a 5-minute task.
USB scanners are the same way. They used to work, now you have to become super-user to use them. Some script that detected scanner plugin events and change the permissions just stopped working.
Multi-card readers: Same thing. Sometimes they work, sometimes they don't.
Windows, Solaris, and MacOS all solved the hot-plugging problem years ago, why can't Linux?
3. Hardware support regression
Mentioned in the article, but worth repeating. I really hate upgrading my OS and discovering that some of my existing hardware is no longer supported. Recent discovery: you can have USB1 or USB2 enabled, but not both at the same time. If you want USB1, remove the ehci_hcd module. If you want USB2, install it. See Bugzilla, Launchpad.net. It seems unlikely this bug will ever be fixed.
I was going to say make sure you have anonymous offshore registration, and offshore hosting, but I think Wikileaks is the better, simpler choice.
Google will almost certainly rat you out if given a court order to do so. They almost never fight a court document.
One of the best ideas I ever heard was that someone should generate fake tiger bones, rhinoceros horns and other artifacts of endangered species and flood the markets with them. This would cause the prices to collapse, making it no longer economical to be a poacher.
OK, seriously, what is it with the Linux community and arbitrarily renaming packages? It's like they *heard* of software compatibility, but decided they wanted no truck with it.
Recently they renamed "libglib1.2" to "libglib1.2dbl" in Debian (although the dev library was not renamed), breaking anything linked against glib. For what possible purpose was this done?
A CEO friend of mine once said "It's never too soon to fire someone".
A person like "Josh", no matter how brilliant they are, can actually produce negative work as a side-effect of the chaos they sew. They may be productive as hell, but their impact on the productivity of the people around them can cause more harm than good.
My friend had to fire a number of people over the years who caused big problems. He never regretted firing someone, but he often regretted not doing it sooner.
Oh, and by the way, writing "clever" code just produces code your peers (and you, six months later) can't maintain. I'll bet Josh's bosses regret not requiring code reviews.
This comic says it all. http://www.penny-arcade.com/comic/2009/1/2/
Oh, wow, a real rebuke! That'll learn 'em a lesson.
How about awarding costs to the victims instead?
People complain about the overburdened legal system, but it seems to me that the courts like frivolous and harassing lawsuits because they mean more work for lawyers and judges.
If the courts really wanted to do something about the clogging of the legal system, they'd award costs to innocent defendants.