Slashdot Mirror

Bennett Haselton writes: United Airlines announced that they will offer up to 1 million air miles to users who can find security holes in their website. I demonstrated a way to brute-force a user's 4-digit PIN number and submitted it to them for review, emailing their Bugs Bounty contact address on three occasions, but I never heard back from them. Read on for the rest. If you've had a different experience with the program, please chime in below.

United Airlines announced the program in May (also specifying rules which specifically prohibited hacking in-flight systems, but which included "[t]he ability to brute-force reservations, MileagePlus numbers, PINs or passwords".) I poked around on their website and discovered that on their "Forgot your MileagePlus number?" page, you can request a reset of your password by submitting your first and last name, AND any ONE of the following:

• your e-mail address
• your street address
• your phone number
• your PIN
• your password
• your "old MileagePlus number"

And after submitting your information, the page will tell you whether your information matched an existing MilagePlus customer record.

This means that if you know a user's first and last name, you can guess their PIN, and the MileagePlus site will tell you whether you got it right or not. If the site doesn't limit your number of guesses, you can write a script that iterates through all 10,000 possibilities for the PIN until it finds the right one.

I wrote a script that did exactly that, and brute-forced my own account's PIN in a few hours (submitting one guess at a time, and running at 2 a.m. so as not to impact any other users). This means that United's website is not limiting the number of guesses per IP address, or showing a CAPTCHA after some number of failed attempts, or limiting the number of guesses per hour on a particular account, or any other countermeasures that you might expect. (The Bugs Bounty Program rules state, "[W]e do not allow execution of brute-force attacks on other users," which I interpreted to mean that brute-forcing your own account ought to be fine.)

So, United, if you're reading this, the immediate fix should be to disable the "PIN" option on the "Forgot your MileagePlus Number?" page. Keep the option to retrieve your account number by submitting your password, since even weak passwords are far harder to guess than 4-digit PIN numbers. But get rid of the PIN option.

I mentioned other possible countermeasures, including limiting requests per IP address and showing a CAPTCHA, but I actually don't think either of these would be effective. If you limit requests per IP address, any serious adversary will have a botnet of machines that they can use to submit requests from different addresses. If you make the user type in a CAPTCHA to submit a request, an attacker can hire workers online to read and type in the CAPTCHAs for a penny apiece. If you limit the number of reset attempts per hour on a particular account, that will slow down the attacker's attempts to brute-force the PIN for a particular account. However, if the attacker has a database of 1000 customer names and wants to find PINs for all of them, on Day 1 they could try 10 PINs for customer 1, then 10 PINs for customer 2, and so on up to customer 1000, and then on Day 2 they could try the next set of 10 PINs on customer 1, customer 2, etc. The attacker can't find any particular customer's PIN quickly, but they will be able to recover all of the customers' PINs slowly -- even though they never did more than 10 PIN authentication attempts on any particular account in the same day. Without a safe countermeasure, then, simply getting rid of PIN authentication would be the best fix.

It's because of attacks like this that I would argue that 4-digit PINs should never be used by themselves for authentication, if there's any possibility of a brute-force attack. They should only ever be used (a) for authentication in conjunction with something else, like a password (for example, if you're already logged in to a financial services account, you could require an additional 4-digit PIN to transfer money to another user); or (b) in a scenario where a brute-force attack is infeasible (for example, if you call tech support and a live human operator asks you to authenticate yourself with a 4-digit PIN).

The same attack is probably possible on the MileagePlus login page, since you can log in using your 4-digit PIN as an alternative to your password. However, this is less of a glaring security hole, because to brute-force a someone's PIN number on that page, you would have to at least know their MileagePlus number. The "Forgot Your MileagePlus Number?" page, on the other hand, allows you to brute-force someone's PIN number when all you know is their name.

As is often the case with stolen PINs and passwords, the most harmful effect here would probably not be the compromising of the user's MileagePlus account. The biggest problem is that most users use the same PINs and passwords for multiple accounts, and the attacker now has the 4-digit PIN that the user probably uses for their voicemail password, their ATM card, their burglar alarm, and who knows what else.

I first sent sent two emails about this to United's bug bounty email address reporting the issue on May 23, a few hours apart, and then followed up on June 1 asking if anyone had seen the first messages. I still have not receive a response.

So why didn't United reply? Have they just been receiving too many submissions by email? About 18 months ago I wrote about a researcher who emailed a security hole to Google and never heard back from them, even after they fixed the issue (although Google apologized and paid him his reward after the article ran). I suggested that if email submissions sometimes get back-logged, it would be a more effective approach to have email submissions reviewed by a lower-paid, less-experienced team of interns than by senior security researchers. The principle is that while it takes experience to find and fix security holes, it only takes some simple logical reasoning skills to evaluate whether a particular discovery constitutes a security hole, so the work can be farmed out to interns who want to gain work experience. By having each submission reviewed by, say, 3 randomly chosen interns from your pool of evaluators, you can churn through the submissions faster and reduce the chances of a legitimate bug falling through the cracks.

I'm sure some of the submissions are crap, and it's not United's fault if they initially got behind because they got more mails than they expected. But as soon as they realized they were getting swamped, they should have put more people on it -- even if those extra people were IT interns with just enough computer experience to read a bug description and tell if it was legit.

And one of the interns could also proofread the submission guidelines. Currently, under "things we will pay 250,000 miles for", the program page lists: "Brute-force attacks." Under "things that will result in criminal prosecution," the same page lists: "Brute-force attacks." If United keeps both promises, I hope my air miles don't expire before I get out of jail.

Bennett Haselton writes: United Airlines announced that they will offer up to 1 million air miles to users who can find security holes in their website. I demonstrated a way to brute-force a user's 4-digit PIN number and submitted it to them for review, emailing their Bugs Bounty contact address on three occasions, but I never heard back from them. Read on for the rest. If you've had a different experience with the program, please chime in below.

United Airlines announced the program in May (also specifying rules which specifically prohibited hacking in-flight systems, but which included "[t]he ability to brute-force reservations, MileagePlus numbers, PINs or passwords".) I poked around on their website and discovered that on their "Forgot your MileagePlus number?" page, you can request a reset of your password by submitting your first and last name, AND any ONE of the following:

• your e-mail address
• your street address
• your phone number
• your PIN
• your password
• your "old MileagePlus number"

And after submitting your information, the page will tell you whether your information matched an existing MilagePlus customer record.

This means that if you know a user's first and last name, you can guess their PIN, and the MileagePlus site will tell you whether you got it right or not. If the site doesn't limit your number of guesses, you can write a script that iterates through all 10,000 possibilities for the PIN until it finds the right one.

I wrote a script that did exactly that, and brute-forced my own account's PIN in a few hours (submitting one guess at a time, and running at 2 a.m. so as not to impact any other users). This means that United's website is not limiting the number of guesses per IP address, or showing a CAPTCHA after some number of failed attempts, or limiting the number of guesses per hour on a particular account, or any other countermeasures that you might expect. (The Bugs Bounty Program rules state, "[W]e do not allow execution of brute-force attacks on other users," which I interpreted to mean that brute-forcing your own account ought to be fine.)

So, United, if you're reading this, the immediate fix should be to disable the "PIN" option on the "Forgot your MileagePlus Number?" page. Keep the option to retrieve your account number by submitting your password, since even weak passwords are far harder to guess than 4-digit PIN numbers. But get rid of the PIN option.

I mentioned other possible countermeasures, including limiting requests per IP address and showing a CAPTCHA, but I actually don't think either of these would be effective. If you limit requests per IP address, any serious adversary will have a botnet of machines that they can use to submit requests from different addresses. If you make the user type in a CAPTCHA to submit a request, an attacker can hire workers online to read and type in the CAPTCHAs for a penny apiece. If you limit the number of reset attempts per hour on a particular account, that will slow down the attacker's attempts to brute-force the PIN for a particular account. However, if the attacker has a database of 1000 customer names and wants to find PINs for all of them, on Day 1 they could try 10 PINs for customer 1, then 10 PINs for customer 2, and so on up to customer 1000, and then on Day 2 they could try the next set of 10 PINs on customer 1, customer 2, etc. The attacker can't find any particular customer's PIN quickly, but they will be able to recover all of the customers' PINs slowly -- even though they never did more than 10 PIN authentication attempts on any particular account in the same day. Without a safe countermeasure, then, simply getting rid of PIN authentication would be the best fix.

It's because of attacks like this that I would argue that 4-digit PINs should never be used by themselves for authentication, if there's any possibility of a brute-force attack. They should only ever be used (a) for authentication in conjunction with something else, like a password (for example, if you're already logged in to a financial services account, you could require an additional 4-digit PIN to transfer money to another user); or (b) in a scenario where a brute-force attack is infeasible (for example, if you call tech support and a live human operator asks you to authenticate yourself with a 4-digit PIN).

The same attack is probably possible on the MileagePlus login page, since you can log in using your 4-digit PIN as an alternative to your password. However, this is less of a glaring security hole, because to brute-force a someone's PIN number on that page, you would have to at least know their MileagePlus number. The "Forgot Your MileagePlus Number?" page, on the other hand, allows you to brute-force someone's PIN number when all you know is their name.

As is often the case with stolen PINs and passwords, the most harmful effect here would probably not be the compromising of the user's MileagePlus account. The biggest problem is that most users use the same PINs and passwords for multiple accounts, and the attacker now has the 4-digit PIN that the user probably uses for their voicemail password, their ATM card, their burglar alarm, and who knows what else.

I first sent sent two emails about this to United's bug bounty email address reporting the issue on May 23, a few hours apart, and then followed up on June 1 asking if anyone had seen the first messages. I still have not receive a response.

So why didn't United reply? Have they just been receiving too many submissions by email? About 18 months ago I wrote about a researcher who emailed a security hole to Google and never heard back from them, even after they fixed the issue (although Google apologized and paid him his reward after the article ran). I suggested that if email submissions sometimes get back-logged, it would be a more effective approach to have email submissions reviewed by a lower-paid, less-experienced team of interns than by senior security researchers. The principle is that while it takes experience to find and fix security holes, it only takes some simple logical reasoning skills to evaluate whether a particular discovery constitutes a security hole, so the work can be farmed out to interns who want to gain work experience. By having each submission reviewed by, say, 3 randomly chosen interns from your pool of evaluators, you can churn through the submissions faster and reduce the chances of a legitimate bug falling through the cracks.

I'm sure some of the submissions are crap, and it's not United's fault if they initially got behind because they got more mails than they expected. But as soon as they realized they were getting swamped, they should have put more people on it -- even if those extra people were IT interns with just enough computer experience to read a bug description and tell if it was legit.

And one of the interns could also proofread the submission guidelines. Currently, under "things we will pay 250,000 miles for", the program page lists: "Brute-force attacks." Under "things that will result in criminal prosecution," the same page lists: "Brute-force attacks." If United keeps both promises, I hope my air miles don't expire before I get out of jail.

"The United States Postal Service "Click-N-Ship" site suffered no outages or slowdowns during Christmas rush," writes Bennett Haselton. "It just has bugs that make the process more annoying than just standing in line at the post office, which defeats the purpose. The most frustrating part is that most of these bugs could have been fixed, just by having some testers run through the ordering process and make a note of anything that seems confusing or wrong. (Although I've included notes on how to work around all the bugs, so you really can print your own labels and skip the line.)" Read on for the rest; what other gripes do you have about the current package delivery regime, and how would you resolve them? This suggestion on the LifeProTips subreddit reminded me that I'd been meaning to try printing my own USPS mailing labels to skip the lines at the post office. I'd been putting it off because I knew that I'd be determined to find the most efficient way of doing everything through the site, and if the site didn't steer me towards exactly the best options, I'd end up forcing myself to reverse-engineer their whole algorithm in order to find the most efficient way myself. That's why I always appreciate it when a website just tells me the best option instead of making me second-guess them.

Right away, the USPS website failed that test because it does not allow you to print first-class mail labels, instead steering you towards the more expensive Priority Mail and Priority Mail Express options. Online users have complained about the lack of first-class-mail options on USPS.com for years, and users on several forums suggested using the PayPal Ship Now site instead, which does let you print first class mailing labels online, along with Priority Mail labels other options.

In my case it was a moot point because I had to use the Priority Mail labels in order for my packages to arrive by Christmas, but the deception was still hugely aggravating. Not just because of the thought of millions of people wasting money (and the finite resources of the postal system) due to the USPS site tricking them into a more expensive upgrade that they didn't need. But because it now meant I'd have to second-guess every recommendation they made, wondering if they were steering me toward something that was worse for me and better for them. The reason sites like Amazon are so stress-free to use is because, for the most part, they do display the options that are best for you, even at the expense of their own short-term profit. Some third-party merchant is selling a book for less than Amazon's list price? They'll let the seller list the book right on their site and undercut Amazon's own sales. The benefit to the user is not just the cost savings, but knowing that you don't have to feel like a chump for not wasting time on search engines trying to find a cheaper deal.

Once I realized the USPS site was concealing the cheaper options, in my determination to avoid getting ripped off by the USPS I almost ended up getting ripped off much worse by one of their "partners". I remembered an ad on a Google search mentioning Stamps.com, so I signed up for an account there and downloaded their software, which does in fact let you print first-class postage. It was only after reading a warning in the original subreddit that I realized I had unwittingly "agreed" to a $15.99/month charge. It turns out that the Stamps.com registration page says above the credit card form that your card info is "required to purchase postage", but this is misleading -- the fine print in the sidebar says you will be charged $15.99 per month if you don't cancel. (And neither the software nor the website gives you a link to cancel -- you have to call their customer service number.) Fortunately, I did call and cancel after realizing I'd been duped, but I was not surprised to learn on Wikipedia that the company had been the subject of over 1,000 Better Business Bureau complaints from users regarding the unauthorized monthly charges. (The part on Wikipedia about "long hold times" is out of date, though -- the automated prompts recognized my account by my phone number and let me cancel without any waiting.)

What does that have to do with USPS.com? Because it never would have happened if the USPS website had been on my side in the first place, giving me all the mailing options that I actually needed. It's bad enough when a private company does this, but the USPS works for us, don't they?

So that's not a "bug" in the traditional sense, but I'm counting it: #1: Not giving users all the mailing options they want to know about.

Most of the other bugs are not self-serving tricks; rather, they're just unclear directions where you have to pause and puzzle out what you're really supposed to do, which is different from what the site tells you to do. For example:

#2: Listing boxes as shipping options that don't fit the dimensions that you've already entered

On the label printing page (requires a USPS.com login if you don't have one) is the option to enter package dimensions. If you specify package details of 1 lbs and 13x5x6 inches, and click to calculate "available Services and Prices" based on the details you've entered, you're presented with a list of options that include 'Priority Mail Flat Rate Envelope 12-1/2" x 9-1/2"', 'Priority Mail Small Flat Rate Box 5-3/8" x 8-5/8" x 1-5/8"', 'Priority Mail Medium Flat Rate Box 11" x 8-1/2" x 5-1/2"', 'Priority Mail Medium Flat Rate Box 13-5/8" x 11-7/8" x 3-3/8"', and 'Priority Mail Padded Flat Rate Envelope 9-1/2" x 12-1/2"' -- all of which, of course, are too small to hold the package whose dimensions you just specified.

You could argue that it's the user's responsibility to make sure their package fits into the box they select, but a user could reasonably assume that the whole point of entering the length, width and height is so that the USPS can recommend only those boxes that will hold the item. Remember, the user usually doesn't have these boxes in front of them at the time they're printing the label. They could end up selecting a box option, printing the label, taking it all the way to the post office along with their package, only to find out that the package doesn't fit into the box that they printed the label for, and that they have to wait in line anwyay to pay for an alternate method.

It's a middle-school-level programming exercise to take the length, width, and height of a package as an input, take as a second input a list of boxes of varying lengths, widths, heights, and costs, and find the lowest-cost box that will hold the package (keeping in mind that the package can be rotated to different orientations so that the "height" becomes the "width", etc.). It's reasonable to expect the postal service to be able to do this too.

#3: Everything wrong with the "print your labels" page

Here's a screen grab of the "print your labels" page that appears after you've paid, which you can use to play the Highlights "What's Wrong?" game:

• The text at the top says "You'll have until 11:59 PM CST of the Ship Date to print these labels." OK, but if I print them at 11:59 PM, what good does it do if the post office closed at 6? Are the labels only valid on the ship date, or will they still work if I take them to the post office the next day? This should be more clear.

• Text says "A SCAN Form must be printed when taking packages to the Post Office." Fine, but there's a checkbox next to that sentence. If that sentence describes a postal regulation, what does it mean if I un-check the box? That the regulation no longer applies to me? Can someone tell me if the drug laws work that way as well?

• The next sentence says: "Close out and print your SCAN Form here." I have no idea what that sentence means. Close out of the browser? And where is "here"? When it's not hyperlinked, "here" means here.

• WHY IS THE "PRINT LABELS" BUTTON DISABLED?? I have the checkboxes checked for both labels. I want to print them. What else do you want me to DO? (My PC has a printer, which the Chrome browser is aware of -- it lets me print from other webpages with no problem.) I got it to work by saving the PDF and printing that, but I never figured out why the Print button was just sitting there, mocking me from behind its veil of grey.

• The "Schedule a Pickup" button at the bottom -- same problem as the "print until 11:59 PM" message at the top. Since I printed these labels with the ship date specified as today, it should be more clear if the labels will still be considered valid tomorrow, which is the soonest time that a pickup could be scheduled.

#4: Over an hour on hold and never got through.

As an adherent to the touchingly quaint notion that a reporter should talk to the subjects of their story before running it, and also because I just wanted clarification on some of these questions, I called the USPS help line and waited on hold for 30 minutes before their help line disconnected me. I called back and waited for another 40 minutes before I hung up this time. OK, strictly speaking that's not a "bug". They just suck.

In the end, after reverse-engineering their pricing options as I had vowed to do, I determined what appeared to be their rules, (applies only to domestic Priority Mail), which you may find handy:

• If you're shipping in a Flat Rate box, the weight of the package doesn't matter (up to the 70 lb limit), only the dimensions, to the extent that they determine which Flat Rate box you can fit it into, with the bigger ones being more expensive.
• On the other hand, if you pick the Priority Mail "Use your own box" option, then the dimensions don't matter (unless you exceed the allowed limits), only the weight -- a 5 lb, 3"x3"x3" package and a 5 lb, 21"x21"x21" package both ship for $15.22, but if you change the weight, that's when the price changes. (If you try to ship a 22"x22"x22" package, you get an error that you've exceeded the dimensions for a Click-N-Ship.)

Using this, I was able to strategically break my one shipment, which would have cost about $30, into two separate shipments which cost $12 and $8. All told, with the effort to reverse-engineer their pricing options and to document all of the bugs for posterity, it took me about an hour to figure out that $10 savings and to print labels that I could take to the post office and skip the line -- which, it turned out, looked only about 3 minutes long -- in order to experience what one redditor described as "feeling the hate from the people standing in line as I casually stroll up and drop my packages off at the front desk". But the important thing is, I did it efficiently.

"The United States Postal Service "Click-N-Ship" site suffered no outages or slowdowns during Christmas rush," writes Bennett Haselton. "It just has bugs that make the process more annoying than just standing in line at the post office, which defeats the purpose. The most frustrating part is that most of these bugs could have been fixed, just by having some testers run through the ordering process and make a note of anything that seems confusing or wrong. (Although I've included notes on how to work around all the bugs, so you really can print your own labels and skip the line.)" Read on for the rest; what other gripes do you have about the current package delivery regime, and how would you resolve them? This suggestion on the LifeProTips subreddit reminded me that I'd been meaning to try printing my own USPS mailing labels to skip the lines at the post office. I'd been putting it off because I knew that I'd be determined to find the most efficient way of doing everything through the site, and if the site didn't steer me towards exactly the best options, I'd end up forcing myself to reverse-engineer their whole algorithm in order to find the most efficient way myself. That's why I always appreciate it when a website just tells me the best option instead of making me second-guess them.

Right away, the USPS website failed that test because it does not allow you to print first-class mail labels, instead steering you towards the more expensive Priority Mail and Priority Mail Express options. Online users have complained about the lack of first-class-mail options on USPS.com for years, and users on several forums suggested using the PayPal Ship Now site instead, which does let you print first class mailing labels online, along with Priority Mail labels other options.

In my case it was a moot point because I had to use the Priority Mail labels in order for my packages to arrive by Christmas, but the deception was still hugely aggravating. Not just because of the thought of millions of people wasting money (and the finite resources of the postal system) due to the USPS site tricking them into a more expensive upgrade that they didn't need. But because it now meant I'd have to second-guess every recommendation they made, wondering if they were steering me toward something that was worse for me and better for them. The reason sites like Amazon are so stress-free to use is because, for the most part, they do display the options that are best for you, even at the expense of their own short-term profit. Some third-party merchant is selling a book for less than Amazon's list price? They'll let the seller list the book right on their site and undercut Amazon's own sales. The benefit to the user is not just the cost savings, but knowing that you don't have to feel like a chump for not wasting time on search engines trying to find a cheaper deal.

Once I realized the USPS site was concealing the cheaper options, in my determination to avoid getting ripped off by the USPS I almost ended up getting ripped off much worse by one of their "partners". I remembered an ad on a Google search mentioning Stamps.com, so I signed up for an account there and downloaded their software, which does in fact let you print first-class postage. It was only after reading a warning in the original subreddit that I realized I had unwittingly "agreed" to a $15.99/month charge. It turns out that the Stamps.com registration page says above the credit card form that your card info is "required to purchase postage", but this is misleading -- the fine print in the sidebar says you will be charged $15.99 per month if you don't cancel. (And neither the software nor the website gives you a link to cancel -- you have to call their customer service number.) Fortunately, I did call and cancel after realizing I'd been duped, but I was not surprised to learn on Wikipedia that the company had been the subject of over 1,000 Better Business Bureau complaints from users regarding the unauthorized monthly charges. (The part on Wikipedia about "long hold times" is out of date, though -- the automated prompts recognized my account by my phone number and let me cancel without any waiting.)

What does that have to do with USPS.com? Because it never would have happened if the USPS website had been on my side in the first place, giving me all the mailing options that I actually needed. It's bad enough when a private company does this, but the USPS works for us, don't they?

So that's not a "bug" in the traditional sense, but I'm counting it: #1: Not giving users all the mailing options they want to know about.

Most of the other bugs are not self-serving tricks; rather, they're just unclear directions where you have to pause and puzzle out what you're really supposed to do, which is different from what the site tells you to do. For example:

#2: Listing boxes as shipping options that don't fit the dimensions that you've already entered

On the label printing page (requires a USPS.com login if you don't have one) is the option to enter package dimensions. If you specify package details of 1 lbs and 13x5x6 inches, and click to calculate "available Services and Prices" based on the details you've entered, you're presented with a list of options that include 'Priority Mail Flat Rate Envelope 12-1/2" x 9-1/2"', 'Priority Mail Small Flat Rate Box 5-3/8" x 8-5/8" x 1-5/8"', 'Priority Mail Medium Flat Rate Box 11" x 8-1/2" x 5-1/2"', 'Priority Mail Medium Flat Rate Box 13-5/8" x 11-7/8" x 3-3/8"', and 'Priority Mail Padded Flat Rate Envelope 9-1/2" x 12-1/2"' -- all of which, of course, are too small to hold the package whose dimensions you just specified.

You could argue that it's the user's responsibility to make sure their package fits into the box they select, but a user could reasonably assume that the whole point of entering the length, width and height is so that the USPS can recommend only those boxes that will hold the item. Remember, the user usually doesn't have these boxes in front of them at the time they're printing the label. They could end up selecting a box option, printing the label, taking it all the way to the post office along with their package, only to find out that the package doesn't fit into the box that they printed the label for, and that they have to wait in line anwyay to pay for an alternate method.

It's a middle-school-level programming exercise to take the length, width, and height of a package as an input, take as a second input a list of boxes of varying lengths, widths, heights, and costs, and find the lowest-cost box that will hold the package (keeping in mind that the package can be rotated to different orientations so that the "height" becomes the "width", etc.). It's reasonable to expect the postal service to be able to do this too.

#3: Everything wrong with the "print your labels" page

Here's a screen grab of the "print your labels" page that appears after you've paid, which you can use to play the Highlights "What's Wrong?" game:

• The text at the top says "You'll have until 11:59 PM CST of the Ship Date to print these labels." OK, but if I print them at 11:59 PM, what good does it do if the post office closed at 6? Are the labels only valid on the ship date, or will they still work if I take them to the post office the next day? This should be more clear.

• Text says "A SCAN Form must be printed when taking packages to the Post Office." Fine, but there's a checkbox next to that sentence. If that sentence describes a postal regulation, what does it mean if I un-check the box? That the regulation no longer applies to me? Can someone tell me if the drug laws work that way as well?

• The next sentence says: "Close out and print your SCAN Form here." I have no idea what that sentence means. Close out of the browser? And where is "here"? When it's not hyperlinked, "here" means here.

• WHY IS THE "PRINT LABELS" BUTTON DISABLED?? I have the checkboxes checked for both labels. I want to print them. What else do you want me to DO? (My PC has a printer, which the Chrome browser is aware of -- it lets me print from other webpages with no problem.) I got it to work by saving the PDF and printing that, but I never figured out why the Print button was just sitting there, mocking me from behind its veil of grey.

• The "Schedule a Pickup" button at the bottom -- same problem as the "print until 11:59 PM" message at the top. Since I printed these labels with the ship date specified as today, it should be more clear if the labels will still be considered valid tomorrow, which is the soonest time that a pickup could be scheduled.

#4: Over an hour on hold and never got through.

As an adherent to the touchingly quaint notion that a reporter should talk to the subjects of their story before running it, and also because I just wanted clarification on some of these questions, I called the USPS help line and waited on hold for 30 minutes before their help line disconnected me. I called back and waited for another 40 minutes before I hung up this time. OK, strictly speaking that's not a "bug". They just suck.

In the end, after reverse-engineering their pricing options as I had vowed to do, I determined what appeared to be their rules, (applies only to domestic Priority Mail), which you may find handy:

• If you're shipping in a Flat Rate box, the weight of the package doesn't matter (up to the 70 lb limit), only the dimensions, to the extent that they determine which Flat Rate box you can fit it into, with the bigger ones being more expensive.
• On the other hand, if you pick the Priority Mail "Use your own box" option, then the dimensions don't matter (unless you exceed the allowed limits), only the weight -- a 5 lb, 3"x3"x3" package and a 5 lb, 21"x21"x21" package both ship for $15.22, but if you change the weight, that's when the price changes. (If you try to ship a 22"x22"x22" package, you get an error that you've exceeded the dimensions for a Click-N-Ship.)

Using this, I was able to strategically break my one shipment, which would have cost about $30, into two separate shipments which cost $12 and $8. All told, with the effort to reverse-engineer their pricing options and to document all of the bugs for posterity, it took me about an hour to figure out that $10 savings and to print labels that I could take to the post office and skip the line -- which, it turned out, looked only about 3 minutes long -- in order to experience what one redditor described as "feeling the hate from the people standing in line as I casually stroll up and drop my packages off at the front desk". But the important thing is, I did it efficiently.

"The United States Postal Service "Click-N-Ship" site suffered no outages or slowdowns during Christmas rush," writes Bennett Haselton. "It just has bugs that make the process more annoying than just standing in line at the post office, which defeats the purpose. The most frustrating part is that most of these bugs could have been fixed, just by having some testers run through the ordering process and make a note of anything that seems confusing or wrong. (Although I've included notes on how to work around all the bugs, so you really can print your own labels and skip the line.)" Read on for the rest; what other gripes do you have about the current package delivery regime, and how would you resolve them? This suggestion on the LifeProTips subreddit reminded me that I'd been meaning to try printing my own USPS mailing labels to skip the lines at the post office. I'd been putting it off because I knew that I'd be determined to find the most efficient way of doing everything through the site, and if the site didn't steer me towards exactly the best options, I'd end up forcing myself to reverse-engineer their whole algorithm in order to find the most efficient way myself. That's why I always appreciate it when a website just tells me the best option instead of making me second-guess them.

Right away, the USPS website failed that test because it does not allow you to print first-class mail labels, instead steering you towards the more expensive Priority Mail and Priority Mail Express options. Online users have complained about the lack of first-class-mail options on USPS.com for years, and users on several forums suggested using the PayPal Ship Now site instead, which does let you print first class mailing labels online, along with Priority Mail labels other options.

In my case it was a moot point because I had to use the Priority Mail labels in order for my packages to arrive by Christmas, but the deception was still hugely aggravating. Not just because of the thought of millions of people wasting money (and the finite resources of the postal system) due to the USPS site tricking them into a more expensive upgrade that they didn't need. But because it now meant I'd have to second-guess every recommendation they made, wondering if they were steering me toward something that was worse for me and better for them. The reason sites like Amazon are so stress-free to use is because, for the most part, they do display the options that are best for you, even at the expense of their own short-term profit. Some third-party merchant is selling a book for less than Amazon's list price? They'll let the seller list the book right on their site and undercut Amazon's own sales. The benefit to the user is not just the cost savings, but knowing that you don't have to feel like a chump for not wasting time on search engines trying to find a cheaper deal.

Once I realized the USPS site was concealing the cheaper options, in my determination to avoid getting ripped off by the USPS I almost ended up getting ripped off much worse by one of their "partners". I remembered an ad on a Google search mentioning Stamps.com, so I signed up for an account there and downloaded their software, which does in fact let you print first-class postage. It was only after reading a warning in the original subreddit that I realized I had unwittingly "agreed" to a $15.99/month charge. It turns out that the Stamps.com registration page says above the credit card form that your card info is "required to purchase postage", but this is misleading -- the fine print in the sidebar says you will be charged $15.99 per month if you don't cancel. (And neither the software nor the website gives you a link to cancel -- you have to call their customer service number.) Fortunately, I did call and cancel after realizing I'd been duped, but I was not surprised to learn on Wikipedia that the company had been the subject of over 1,000 Better Business Bureau complaints from users regarding the unauthorized monthly charges. (The part on Wikipedia about "long hold times" is out of date, though -- the automated prompts recognized my account by my phone number and let me cancel without any waiting.)

What does that have to do with USPS.com? Because it never would have happened if the USPS website had been on my side in the first place, giving me all the mailing options that I actually needed. It's bad enough when a private company does this, but the USPS works for us, don't they?

So that's not a "bug" in the traditional sense, but I'm counting it: #1: Not giving users all the mailing options they want to know about.

Most of the other bugs are not self-serving tricks; rather, they're just unclear directions where you have to pause and puzzle out what you're really supposed to do, which is different from what the site tells you to do. For example:

#2: Listing boxes as shipping options that don't fit the dimensions that you've already entered

On the label printing page (requires a USPS.com login if you don't have one) is the option to enter package dimensions. If you specify package details of 1 lbs and 13x5x6 inches, and click to calculate "available Services and Prices" based on the details you've entered, you're presented with a list of options that include 'Priority Mail Flat Rate Envelope 12-1/2" x 9-1/2"', 'Priority Mail Small Flat Rate Box 5-3/8" x 8-5/8" x 1-5/8"', 'Priority Mail Medium Flat Rate Box 11" x 8-1/2" x 5-1/2"', 'Priority Mail Medium Flat Rate Box 13-5/8" x 11-7/8" x 3-3/8"', and 'Priority Mail Padded Flat Rate Envelope 9-1/2" x 12-1/2"' -- all of which, of course, are too small to hold the package whose dimensions you just specified.

You could argue that it's the user's responsibility to make sure their package fits into the box they select, but a user could reasonably assume that the whole point of entering the length, width and height is so that the USPS can recommend only those boxes that will hold the item. Remember, the user usually doesn't have these boxes in front of them at the time they're printing the label. They could end up selecting a box option, printing the label, taking it all the way to the post office along with their package, only to find out that the package doesn't fit into the box that they printed the label for, and that they have to wait in line anwyay to pay for an alternate method.

It's a middle-school-level programming exercise to take the length, width, and height of a package as an input, take as a second input a list of boxes of varying lengths, widths, heights, and costs, and find the lowest-cost box that will hold the package (keeping in mind that the package can be rotated to different orientations so that the "height" becomes the "width", etc.). It's reasonable to expect the postal service to be able to do this too.

#3: Everything wrong with the "print your labels" page

Here's a screen grab of the "print your labels" page that appears after you've paid, which you can use to play the Highlights "What's Wrong?" game:

• The text at the top says "You'll have until 11:59 PM CST of the Ship Date to print these labels." OK, but if I print them at 11:59 PM, what good does it do if the post office closed at 6? Are the labels only valid on the ship date, or will they still work if I take them to the post office the next day? This should be more clear.

• Text says "A SCAN Form must be printed when taking packages to the Post Office." Fine, but there's a checkbox next to that sentence. If that sentence describes a postal regulation, what does it mean if I un-check the box? That the regulation no longer applies to me? Can someone tell me if the drug laws work that way as well?

• The next sentence says: "Close out and print your SCAN Form here." I have no idea what that sentence means. Close out of the browser? And where is "here"? When it's not hyperlinked, "here" means here.

• WHY IS THE "PRINT LABELS" BUTTON DISABLED?? I have the checkboxes checked for both labels. I want to print them. What else do you want me to DO? (My PC has a printer, which the Chrome browser is aware of -- it lets me print from other webpages with no problem.) I got it to work by saving the PDF and printing that, but I never figured out why the Print button was just sitting there, mocking me from behind its veil of grey.

• The "Schedule a Pickup" button at the bottom -- same problem as the "print until 11:59 PM" message at the top. Since I printed these labels with the ship date specified as today, it should be more clear if the labels will still be considered valid tomorrow, which is the soonest time that a pickup could be scheduled.

#4: Over an hour on hold and never got through.

As an adherent to the touchingly quaint notion that a reporter should talk to the subjects of their story before running it, and also because I just wanted clarification on some of these questions, I called the USPS help line and waited on hold for 30 minutes before their help line disconnected me. I called back and waited for another 40 minutes before I hung up this time. OK, strictly speaking that's not a "bug". They just suck.

In the end, after reverse-engineering their pricing options as I had vowed to do, I determined what appeared to be their rules, (applies only to domestic Priority Mail), which you may find handy:

• If you're shipping in a Flat Rate box, the weight of the package doesn't matter (up to the 70 lb limit), only the dimensions, to the extent that they determine which Flat Rate box you can fit it into, with the bigger ones being more expensive.
• On the other hand, if you pick the Priority Mail "Use your own box" option, then the dimensions don't matter (unless you exceed the allowed limits), only the weight -- a 5 lb, 3"x3"x3" package and a 5 lb, 21"x21"x21" package both ship for $15.22, but if you change the weight, that's when the price changes. (If you try to ship a 22"x22"x22" package, you get an error that you've exceeded the dimensions for a Click-N-Ship.)

Using this, I was able to strategically break my one shipment, which would have cost about $30, into two separate shipments which cost $12 and $8. All told, with the effort to reverse-engineer their pricing options and to document all of the bugs for posterity, it took me about an hour to figure out that $10 savings and to print labels that I could take to the post office and skip the line -- which, it turned out, looked only about 3 minutes long -- in order to experience what one redditor described as "feeling the hate from the people standing in line as I casually stroll up and drop my packages off at the front desk". But the important thing is, I did it efficiently.

"The United States Postal Service "Click-N-Ship" site suffered no outages or slowdowns during Christmas rush," writes Bennett Haselton. "It just has bugs that make the process more annoying than just standing in line at the post office, which defeats the purpose. The most frustrating part is that most of these bugs could have been fixed, just by having some testers run through the ordering process and make a note of anything that seems confusing or wrong. (Although I've included notes on how to work around all the bugs, so you really can print your own labels and skip the line.)" Read on for the rest; what other gripes do you have about the current package delivery regime, and how would you resolve them? This suggestion on the LifeProTips subreddit reminded me that I'd been meaning to try printing my own USPS mailing labels to skip the lines at the post office. I'd been putting it off because I knew that I'd be determined to find the most efficient way of doing everything through the site, and if the site didn't steer me towards exactly the best options, I'd end up forcing myself to reverse-engineer their whole algorithm in order to find the most efficient way myself. That's why I always appreciate it when a website just tells me the best option instead of making me second-guess them.

Right away, the USPS website failed that test because it does not allow you to print first-class mail labels, instead steering you towards the more expensive Priority Mail and Priority Mail Express options. Online users have complained about the lack of first-class-mail options on USPS.com for years, and users on several forums suggested using the PayPal Ship Now site instead, which does let you print first class mailing labels online, along with Priority Mail labels other options.

In my case it was a moot point because I had to use the Priority Mail labels in order for my packages to arrive by Christmas, but the deception was still hugely aggravating. Not just because of the thought of millions of people wasting money (and the finite resources of the postal system) due to the USPS site tricking them into a more expensive upgrade that they didn't need. But because it now meant I'd have to second-guess every recommendation they made, wondering if they were steering me toward something that was worse for me and better for them. The reason sites like Amazon are so stress-free to use is because, for the most part, they do display the options that are best for you, even at the expense of their own short-term profit. Some third-party merchant is selling a book for less than Amazon's list price? They'll let the seller list the book right on their site and undercut Amazon's own sales. The benefit to the user is not just the cost savings, but knowing that you don't have to feel like a chump for not wasting time on search engines trying to find a cheaper deal.

Once I realized the USPS site was concealing the cheaper options, in my determination to avoid getting ripped off by the USPS I almost ended up getting ripped off much worse by one of their "partners". I remembered an ad on a Google search mentioning Stamps.com, so I signed up for an account there and downloaded their software, which does in fact let you print first-class postage. It was only after reading a warning in the original subreddit that I realized I had unwittingly "agreed" to a $15.99/month charge. It turns out that the Stamps.com registration page says above the credit card form that your card info is "required to purchase postage", but this is misleading -- the fine print in the sidebar says you will be charged $15.99 per month if you don't cancel. (And neither the software nor the website gives you a link to cancel -- you have to call their customer service number.) Fortunately, I did call and cancel after realizing I'd been duped, but I was not surprised to learn on Wikipedia that the company had been the subject of over 1,000 Better Business Bureau complaints from users regarding the unauthorized monthly charges. (The part on Wikipedia about "long hold times" is out of date, though -- the automated prompts recognized my account by my phone number and let me cancel without any waiting.)

What does that have to do with USPS.com? Because it never would have happened if the USPS website had been on my side in the first place, giving me all the mailing options that I actually needed. It's bad enough when a private company does this, but the USPS works for us, don't they?

So that's not a "bug" in the traditional sense, but I'm counting it: #1: Not giving users all the mailing options they want to know about.

Most of the other bugs are not self-serving tricks; rather, they're just unclear directions where you have to pause and puzzle out what you're really supposed to do, which is different from what the site tells you to do. For example:

#2: Listing boxes as shipping options that don't fit the dimensions that you've already entered

On the label printing page (requires a USPS.com login if you don't have one) is the option to enter package dimensions. If you specify package details of 1 lbs and 13x5x6 inches, and click to calculate "available Services and Prices" based on the details you've entered, you're presented with a list of options that include 'Priority Mail Flat Rate Envelope 12-1/2" x 9-1/2"', 'Priority Mail Small Flat Rate Box 5-3/8" x 8-5/8" x 1-5/8"', 'Priority Mail Medium Flat Rate Box 11" x 8-1/2" x 5-1/2"', 'Priority Mail Medium Flat Rate Box 13-5/8" x 11-7/8" x 3-3/8"', and 'Priority Mail Padded Flat Rate Envelope 9-1/2" x 12-1/2"' -- all of which, of course, are too small to hold the package whose dimensions you just specified.

You could argue that it's the user's responsibility to make sure their package fits into the box they select, but a user could reasonably assume that the whole point of entering the length, width and height is so that the USPS can recommend only those boxes that will hold the item. Remember, the user usually doesn't have these boxes in front of them at the time they're printing the label. They could end up selecting a box option, printing the label, taking it all the way to the post office along with their package, only to find out that the package doesn't fit into the box that they printed the label for, and that they have to wait in line anwyay to pay for an alternate method.

It's a middle-school-level programming exercise to take the length, width, and height of a package as an input, take as a second input a list of boxes of varying lengths, widths, heights, and costs, and find the lowest-cost box that will hold the package (keeping in mind that the package can be rotated to different orientations so that the "height" becomes the "width", etc.). It's reasonable to expect the postal service to be able to do this too.

#3: Everything wrong with the "print your labels" page

Here's a screen grab of the "print your labels" page that appears after you've paid, which you can use to play the Highlights "What's Wrong?" game:

• The text at the top says "You'll have until 11:59 PM CST of the Ship Date to print these labels." OK, but if I print them at 11:59 PM, what good does it do if the post office closed at 6? Are the labels only valid on the ship date, or will they still work if I take them to the post office the next day? This should be more clear.

• Text says "A SCAN Form must be printed when taking packages to the Post Office." Fine, but there's a checkbox next to that sentence. If that sentence describes a postal regulation, what does it mean if I un-check the box? That the regulation no longer applies to me? Can someone tell me if the drug laws work that way as well?

• The next sentence says: "Close out and print your SCAN Form here." I have no idea what that sentence means. Close out of the browser? And where is "here"? When it's not hyperlinked, "here" means here.

• WHY IS THE "PRINT LABELS" BUTTON DISABLED?? I have the checkboxes checked for both labels. I want to print them. What else do you want me to DO? (My PC has a printer, which the Chrome browser is aware of -- it lets me print from other webpages with no problem.) I got it to work by saving the PDF and printing that, but I never figured out why the Print button was just sitting there, mocking me from behind its veil of grey.

• The "Schedule a Pickup" button at the bottom -- same problem as the "print until 11:59 PM" message at the top. Since I printed these labels with the ship date specified as today, it should be more clear if the labels will still be considered valid tomorrow, which is the soonest time that a pickup could be scheduled.

#4: Over an hour on hold and never got through.

As an adherent to the touchingly quaint notion that a reporter should talk to the subjects of their story before running it, and also because I just wanted clarification on some of these questions, I called the USPS help line and waited on hold for 30 minutes before their help line disconnected me. I called back and waited for another 40 minutes before I hung up this time. OK, strictly speaking that's not a "bug". They just suck.

In the end, after reverse-engineering their pricing options as I had vowed to do, I determined what appeared to be their rules, (applies only to domestic Priority Mail), which you may find handy:

• If you're shipping in a Flat Rate box, the weight of the package doesn't matter (up to the 70 lb limit), only the dimensions, to the extent that they determine which Flat Rate box you can fit it into, with the bigger ones being more expensive.
• On the other hand, if you pick the Priority Mail "Use your own box" option, then the dimensions don't matter (unless you exceed the allowed limits), only the weight -- a 5 lb, 3"x3"x3" package and a 5 lb, 21"x21"x21" package both ship for $15.22, but if you change the weight, that's when the price changes. (If you try to ship a 22"x22"x22" package, you get an error that you've exceeded the dimensions for a Click-N-Ship.)

Using this, I was able to strategically break my one shipment, which would have cost about $30, into two separate shipments which cost $12 and $8. All told, with the effort to reverse-engineer their pricing options and to document all of the bugs for posterity, it took me about an hour to figure out that $10 savings and to print labels that I could take to the post office and skip the line -- which, it turned out, looked only about 3 minutes long -- in order to experience what one redditor described as "feeling the hate from the people standing in line as I casually stroll up and drop my packages off at the front desk". But the important thing is, I did it efficiently.

With the holidays coming up, Bennett Haselton has updated his geek-oriented gift guide for 2014. He says: Some of my favorite gifts to give are still the ones that were listed in several different previously written posts, while a few new cool gift ideas emerged in 2014. Here are all my current best recommendations, listed in one place. Read on for the list, or to share any suggestions of your own.

Most annual gift guides would only list new items. It would be considered a mortal sin of click-baiting to tell the reader, "Well, the coolest stuff we could tell you about, was stuff that we mentioned this time last year, so first and foremost we're just going to direct you to that."

Well, my job in writing a gift guide is not to dazzle people with "all new hottest gift item" recommendations, my job is to recommend the things that I think you would most enjoy giving and the recipient would most enjoy receiving, and the fact of the matter is that most of the gifts I would most highly recommend, were listed in different previously written articles. I'll provide that list in a second (with links back to the older articles describing them in more detail), but first some criteria for how I make the recommendations.

First, I'm assuming you want to go inexpensive. If you have unlimited cash, you don't need my help finding cool presents -- although for the record, the online store of New York's Museum of Modern Art has the best collection of things that incorporate "visual puns" that I really like, but which are usually overpriced for what the item does. (Check out this image of a set of nesting tables, for example -- which isn't even that much of a "neat idea", by their standards -- and try to guess how much they cost, before looking at the answer on the product page.) If you don't mind spending the money, they also sell a dandelion encased in acrylic ($375), a lamp in the shape of an open book ($190), a necklace of small rectangular mirrors creating an interesting 3D effect ($190), a porcelain vase that kind of looks like a crinkled paper bag ($120), a pair of candleholders that interlock without touching ($170), a serving tray that looks like the splash from a drop of water ($130), a clock that evokes an M.C. Escher "infinite staircase" optical illusion ($80), and a vase that exists in the shape of an outline ($65, which at this point sounds cheap). At the end of this gift guide I list some MoMA items that are somewhat more reasonably priced.

Second, I'm assuming you don't need help finding branded merchandise. I'm sure literally every combination of [Star Wars / LOTR / Game of Thrones / Star Trek / Hunger Games] and [coffee mug / beach towel / earrings / Christmas tree ornaments / shot glass / cufflinks] is available somewhere. It's not that these are bad gifts for the hardcore fan, it's that all you need to find them is to Google "Game of Thrones Christmas tree ornaments" and you'll find something. And occasionally you'll will find something in this "branded" category that jumps out as a pretty cool idea, like the TARDIS Tea Infuser or the Game of Thrones Dragonclaw Goblet or the light-up lightsaber chopsticks.

Third, I look mostly for novelty or decorative items that confound your senses or demonstrate some interesting scientific principle (or both), but that can still fit in to a semi-elegant environment without garishly calling attention to themselves. These color changing beads are kind of neat, but it would look weird having them lying around on top of a living room dresser along with a UV flashlight to demonstrate what they do. On the other hand, a Galileo thermometer can blend in pretty well the decorations on a mantlepiece.

The following are my current most-recommended gift ideas:

Custom Photomosaic

In this December 2013 post I described how to create a photomosaic (a patchwork of smaller pictures that, when viewed from a distance, take on the appearance of a larger picture) using two free (donation-supported) programs, one to download and save pictures en masse from a friend's Facebook profile, and the other to create the photomosaic using those photos. I still think they make amazing gifts, and the only cost is the cost of printing and framing it. You can even give the digital-only version as a gift that costs nothing at all, making a photomosaic from a friend's photos and sending it to them on the other side of the world, where they can print it themselves or use it as a desktop background. Everyone that I've given one of these to, has loved it.

Strandbeest kit

With this $35 kit, which I recommended as a Christmas gift last year, you can assemble a tabletop version of the legendary full-size Strandbeests, the eerily lifelike creatures created by Dutch artist Theo Jansen which walk across the beaches of Holland powered only by the wind. Assembly of the creature takes about 90 minutes, less if you make a family activity out of it and share the labor.

In the last year, a second model has been released, dubbed the "Rhinoceros Mini-Beest (technically, it was available a year ago, but the assembly instructions were only printed in Japanese; now it's available with English directions). I haven't assembled one of these myself yet, but it looks fine in the video.

There is also now a pre-assembled, motorized, remote-control version of the Strandbeest, although honestly, where's the fun in that? Part of the effect of the Strandbeest assembly kit is the feeling that you've breathed life into an inanimate object by putting it together from static parts. A remote-control toy that moves forwards and backwards on the ground is a little underwhelming when you can get an RC helicopter for the same price.

Levitron Revolution and Levitron Cherrywood

The Levitron Revolution ($70) consists of a circular magnetic disc that levitates about half an inch above an electrically powered square base, and can support up to a pound of weight on top of it while maintaining levitation. The Levitron Cherrywood ($35) consists of a top that has to be spun by hand, which levitates almost a full two inches above the cherrywood base containing an embedded magnet. As described in the first gift guide, the Levitron Cherrywood is more visually impressive because of the extra height of levitation, but the top almost always falls if you touch the top or move the base while the top is spinning. The Levitron Revolution only levitates the disc by half an inch, but you can embellish the appearance by placing other objects on top of it, like the pyrite crystal levitating in this video. (Also, the Levitron Revolution will continue levitating as long as power is supplied to it, making it a good decorative item; the Levitron Cherrywood has to be spun by hand and levitates for only about two minutes before air friction slows it down, so it works better as a toy or party activity.) Both of them take some practice to operate (the Levitron Cherrywood takes considerably more), but they're worth it.

Spare batteries for your friend's phone

I mentioned this in January 2013 as a life hack for smartphones and got pilloried for promoting what people called an "obvious" idea. But two years later, almost nobody that I know is carrying around fully charged extra batteries for their phones. It's easy, it works, and the spare batteries in my jacket pockets have gotten me out of a jam multiple times. If you know what type of phone your gift recipient uses, get them some extra batteries. (T-Mobile sent me extra batteries for my LG Optimus for free.)

The iPhone is the only phone I'm aware of that does not support this, because the battery is not meant to be removed or replaced by the user.

And yes, I know about the portable external battery products that can be used to charge a phone. These aren't as big or expensive as they used to be, but you still have to leave them plugged in to your phone while they're charging it, which is awkward if you're using your phone or carrying it in your pocket (compared to the 10 seconds it takes to swap out the battery).

Heat Wave car heater

This $40 device from Canadian company "Heat & Clean" sits on your dashboard and turns itself on at a pre-determined time, blowing hot air into the interior of the car for 20 minutes, all without starting the car. If all goes well, this means your car will be warm (or at least not as cold as it would have been) when you first get into it in the morning, and you don't have to wait several minutes for the air ventilation system to heat up. The Heat Wave is powered by the car's 12V charger, although Heat & Clean states that "has built-in circuitry to ensure that the health of the vehicle's battery is never compromised."

I can't vouch for the device's respect for your battery (the device unfortunately isn't sold on Amazon, which is usually where I go to find out if something does what it's supposed to), but I ordered one and verified that it works, then gave it to my aunt for her birthday. (If it kills her car, I will promptly make it up to her by taking it out of the gift guide.) Even though I tested the functionality, I didn't get the chance to see if the device actually helps much, because it's not cold enough yet here in Seattle to really feel the bite of the cold when you get into your car in the morning. (The other reason I personally won't get much benefit from this, is that I work from home and leave the house at a different time every day, so I never know in advance what time I'll be getting into my car the next morning.)

Most cars can be modified so they can be started remotely, so that they're warmed up by the time you go outside and get into them, but that modification usually costs a few hundred dollars. You might as well try the Heat Wave first to see if it does the job almost as well.

(To me, the obvious question is: Why not make a version of the Heat Wave that can be turned remotely, as well? For people like me who usually don't know the night before what time they'll be getting into their car the next day, but who often do know at least 20 minutes in advance, so they can turn it on remotely and start heating the car. The company does make a very different-looking product called the Heat Stick which can be activated remotely -- but at $300, that's more than it costs to add remote start to the actual car.)

It's Nuts 3D puzzle

Distributed by Grand Illusions in the UK, the It's Nuts 3D puzzle consists of a bolt and a pair of nuts threaded onto the bolt. As you can see in the video, when you rotate one nut, it moves in the direction that you'd expect, following the threading on the bolt -- but when you rotate the other nut the same way, it moves in the opposite direction. There are no hidden moving parts to make the illusion work, and you can unscrew both nuts right off of the bolt and examine them.

Having acquired one, I can say that the secret is a little bit easier to figure out when you're holding it in your hand and looking at it closely, than when you're watching the video. But it still makes a nice novelty conversation piece.

As a brain-tickler, this feels a bit overpriced at $40 - costing more than the aforementioned Levitron Cherrywood, which actually levitates. I bought It's Nuts mainly to fill out my collection off oddities since I already owned most of the other items in this list. Of course, unlike the Levitron, this takes no skill to operate (only a little bit of smarts to figure out the secret). I wouldn't foist the Levitron on my grandfather, but he would probably enjoy this one.

(Note that if you buy from the Grand Illusions website to ship to the United States, you'll pay the non-VAT rate -- but then you'll have to add about $14 in air mail shipping to the U.S. So you might want to combine this order with some other items from Grand Illusions -- search this article for "Grand Illusions" for the other recommended items, or browse their site and pick your own.)

Inverter Magnet

The Inverter Magnet, from Grand Illusions (also available from Amazon at a slightly higher price), consists of one disc magnet encased in rubber, which holds a second disc magnet in a permanent "force field" a few millimeters away when the two are slid across a table.

At $40, this might be more of a "collection filler-outer", since especially as a magnetic toy it compares unfavorably with the Levitron Cherrywood, which, to repeat, can actually fly. But again, the Inverter Magnet also takes no skill to operate.

Magna Nails

This nail polish (about $7 used on Amazon) forms a stripe pattern when you hold a magnet near your fingernail while the polish is trying. (There are many similar products on the market, searchable under "magnetic nail polish.") The appeal to young science geeks is that the iron filings in the nail polish align themselves along magnetic field lines in the vicinity of the magnet, forming the stripes.

I don't wear glittery nail polish (at least not as "Bennett Haselton"...), but I took one for the team to see if this works. It does. That's my thumb. (Apparently it looks better if you apply a "top coat" after the nail polish dries, but I didn't have any.) If you plan on testing it out before giving it as a gift, remember to pick up some nail polish remover to get it off.

A few things from MoMA that we can actually afford

• An appetizer serving dish complete with toothpick holder that looks like a porcupine ($28). This item probably has the highest price-to-cool-factor ratio of anything on this list, just barely within the limit of what I'd recommend, but it's elegant in addition to being funny, and the recipient would probably use it.
• A faux-wood cube clock ($38). This is interesting mostly for looking like a visual impossibility -- how can the digital numbers appear on the side of a block of wood, even fake wood? Unfortunately I think the photo is doctored, because this youtube video shows an undoctored shot of the cube clock, and you can easily see the un-illuminated LEDs on the side, which don't quite blend in with the wood. But it still makes for an elegant optical illusion.

Miscellaneous "Visual Puns"

• The ambiguous vase ($33) from Grand Illusions -- a real-life version of the Rubin vase optical illusion, where a vase suddenly takes on the appearance of two faces in profile.
• The glass water faucet ($50)
• The sliced grandfather clock ($36)
• Ulexite "television stones" ($10) - a rock that, when placed on a flat surface, will cause the markings on that surface to re-appear on the top face of the rock, due to the naturally occurring fiber optics in ulexite.

Of course, if you're now craving one of these items for yourself, order one and try it out before re-gifting, or set up an Amazon wish list in the last two weeks before Christmas. And remember to be good!

Bennett Haselton writes: My last article garnered some objections from readers saying that the sample sizes were too small to draw meaningful conclusions. (36 out of 47 survey-takers, or 77%, said that a picture of a black woman breast-feeding was inappropriate; while in a different group, 38 out of 54 survey-takers, or 70%, said that a picture of a white woman breast-feeding was inappropriate in the same context.) My conclusion was that, even on the basis of a relatively small sample, the evidence was strongly against a "huge" gap in the rates at which the surveyed population would consider the two pictures to be inappropriate. I stand by that, but it's worth presenting the math to support that conclusion, because I think the surveys are valuable tools when you understand what you can and cannot demonstrate with a small sample. (Basically, a small sample can present only weak evidence as to what the population average is, but you can confidently demonstrate what it is not.) Keep reading to see what Bennett has to say.

The smallest sample I've ever used to make an argument was when I submitted some legal briefs, each no longer than five pages, in the anti-spam cases that I'd been filing in Washington State small claims court. Since I suspected the judges were not taking the cases seriously, I filed the briefs with the third and fourth pages stuck together in the center, by a tiny thread of paper joining the back of the third page to the front of the fourth page. (If someone were to turn the pages and actually readthe brief, the thread would break.) I did something similar in six different cases, and when the motions were all rejected, I went to the courthouse to look at the paper motions still in the file. In three out of six cases, the judge had rejected the motion without reading it first.

Now, the point was not to make any accurate estimation of the actual proportion, in the total population of small claims court judges, who would reject a brief in an anti-spam case without reading it. There's no basis for saying that the proportion of such judges is close to 50%. But we can still probably reject any contention that the proportion of such judges is very low. If only 10% of judges were rejecting motions without reading them, then there is only about a 1.4% chance of taking a random sample of six rejected motions and finding that in three or more cases, the judge did not read the motion. Even if 20% of judges were doing so, for an event with a probability of p=0.20 you would still only see it occur in three out of six cases, about 8.2% of the time. (If an event has probability p, the exact probability of that event occuring three or more times in six trials is given by 20*(p^3)*((1-p)^3) + 15*(p^4)*((1-p)^2) + 6*(p^5)*((1-p)^1) + 1*(p^6)*((1-p)^0).) So we can say that the proportion of such judges is quite probably more than 20%. I did this repeatedly because even after I had "caught" the first judge, I wanted to head off any objection that this was just an isolated case of rare behavior.

And, as always, it's important not to generalize too much about the behavior whose probability we're estimating. I don't think that 20% or more of judges, even in small claims court, are throwing most types of cases without reading or listening to the arguments. My impression was that most judges see view small claims court as a place to redress injustices, and that they see anti-spam and anti-telemarketer plaintiffs as just trying to "make money" at it, so they take those suits less seriously. I disagreed with this stance because (1) anti-spam plaintiffs usually really have been harmed and are not just "whining about one email" which they are trying to "cash in" (I still get so much spam that it interferes somewhat with the operation of my server and with my ability to get through my daily email); and (2) the law is intended after all as a deterrent, with disproportionate damages in order to discourage spammers from spamming in the first place. However, the charitable reading of the results is to assume that judges are merely biased against anti-spam plaintiffs -- but at least they probably don't treat all cases as casually as they treat anti-spam suits!

Back to the issue of small samples. My previous article was prompted by an editorial about the online response that had been elicited by two different photos -- one showing a black woman breastfeeding, and a nearly identical photo showing a white woman breastfeeding. The author asserted that the photos had received vastly different responses, which she attributed to racism. I presented a survey to a sample if users recruited from Amazon's Mechanical Turk, randomly showed each survey-taker one of the two photos, and asked:

Our academic department has asked everyone to submit a "fun" photo of themselves, so that our photos can be displayed together on the department home page. One of our employees submitted a photo that has caused some internal debate about whether the photo is inappropriate. I wanted to do a poll to get the opinion of a random sample of Internet users of different backgrounds.

Do you think this is an appropriate picture to be used in a photo collection on our academic department home page?

Out of 47 respondents who saw the black woman's photo, 36 of them (77%) said it was inappropriate. Out of 54 respondents who saw the white woman's photo, 38 of them (70%) said it was inappropriate.

As before, these samples are to small to say precisely what the relevant proportions in the background populations are, but we can probably reject certain statements about the populations -- for example, that the percentage of users offended by the black woman's photo is 20 percentage points higher than the percentage of users offended by the white woman's photo. This is where the counterintuitive part comes in. Suppose that in the background population, 81% of respondents would find the black woman's photo offensive, but only 61% would be offended by the white woman's photo. What are the odds of getting 77% or less "yes that's offensive" responses from a sample of 47 users shown the black woman's photo, and getting 70% or more "yes that's offensive" responses from a sample of 54 users shown the white woman's photo? It doesn't sound unlikely at all, because the percentages are quite close to the originals -- but you can verify, either with statistical calculations or with a quickly written computer program, that the odds are only about 2.5%.

Two main factors contribute to this counterintuitive result. First, even with a sample size of a few dozen, the frequency of an event starts to tend very closely to the frequency in the background population (if 80% of your population has some trait, and you take a sample of size 50, there's about a 95% chance that the number with that trait in your population will be between 34 and 46). Second, to find the odds of seeing both of these deviations at the same time (deviating from an assumed 81% in the background population down to 77% in the first sample, and deviating from an assumed 61% in the background population up to 70% in the second sample), you have to mutiply the probabilities of these two unlikely events. The probability of the first deviation is about 19%, the probability of the second is about 13%, and so the probability of them both occurring is about 2.5%.

The reason I calculated the odds of getting 77% or less "offended" responses for the black woman's photo while also getting 70% or more "offended" responses for the white woman's photo, is that in calculating the "unlikeliness" of a statistical result, it's customary to calculate the odds of getting "this result or a more extreme one". For example, suppose you want to know if a company's hiring process is gender-balanced (assuming a 50/50 gender split in the population), and you notice that in a random sample of 100 recent hires, 61 were men. You wouldn't ask "What are the odds of there being exactly 61 men in this sample?", because the odds of getting any particular number, are small. You'd ask, "What are the odds of getting this result or a more extreme one -- i.e. the odds of getting 61 or more men out of a random sample of 100, if the population were truly gender-balanced? As this calculation tool shows, the odds are only about 1.7%.

Similarly, in the case of the two populations being measured, the author of the original editorial hypothesized that there was some significant gap between the percentages of the population that were offended by the two photos, which I arbitrarily assumed to be 20 percentage points. Under that assumption, showing the two pictures to two different groups and having them be offended at similar rates, is the unexpected, "extreme" result, and the closer the rates are to each other, the more extreme the result is. That's why I calculated "77% of less" for the first group vs. "70% or more" for the second group.

And out of the pairs of numbers that I tested which were separated by 20 percentage points, 81% and 61% were the numbers which made the given result the least unlikely. 80/60 and 79/59 give odds of about 2.5% and 2.4%; 82/62 and 83/63 give odds of 2.4% and 2.2%.

You can do the statistical calculations directly, but in case you won't believe it unless you see the results unfold with your own eyes, you can run this perl script, which iterates through a million trials of the experiment, counting the number of times that the unexpected result occurs.

Why did I assume a 20-point gap? That was the most subjective leap that I made. Looking through the original editorial, I figured that on the basis of inflammatory statements like

"Only one woman was called 'adorable' by the media and portrayed with girlish innocence, and it wasn't the black one. It never is."

and

"The contrast in headlines is so stark, it deserves to be examined" [I assume here she meant the contrast in responses]

the author meant to imply a difference in people's attitudes that was at least that large. But the results suggest that it isn't.

For all of this effort, of course, I could have just expanded the original experiment to a sample of several hundred and mollified some people's concerns. But I wanted to argue for what you can show, even with small samples, because I would like to try (and would like others to try) similar experiments in the future, and do not think people should be discouraged if they can't afford to pay a thousand Amazon Mechanical Turk workers to take their survey. I paid my 100 respondents $0.25 each; naturally, one experiment I'd like to do soon is to figure out what's the lowest I can get away with paying them.

Bennett Haselton writes: A editorial with 24,000 Facebook shares highlights the differences in public reaction to two nearly identical breastfeeding photos, one showing a black woman and one showing a white woman, each breastfeeding an infant. The editorial decries the outrage provoked by the black woman's photo compared to the mild reaction elicited by the white woman's photo, and attributes the difference to racism. I tried an experiment using Amazon's Mechanical Turk to test that theory. Read on to see the kind of results Bennett found.

You can see the side-by-side pictures in the November 10 editorial by Ruby Hamad. My first thought, upon seeing the pictures, was that this is not a controlled experiment -- the woman on the left is breastfeeding in public, while the woman on the right is breastfeeding against a blank wall inside a presumably private room. While I think breastfeeding in public should be completely normalized, it's not the same thing as breastfeeding in private, and so that might have accounted for the difference in reactions, if there was any.

My second thought was that the data on people's reactions was not collected in a systematic way. According to the editorial, the black photo of the black mother, Karlesha Thurman, was posted on the Facebook page Black Women Do Breastfeed, and "[w]hile Karlesha received many supportive comments, the backlash was so severe, she eventually deleted the photo." The photo of the Australian woman, Jacci Sharkey, was posted by the University of the Sunshine Coast on their Facebook page, where it received 275,000 Facebook "likes", but also, according to the editorial, "more than a few detractors, proving that breastfeeding in public is (still!) a contentious issue for women of all races." There's no apples-to-apples comparison gauging people's reactions to the two photos under similar conditions.

But just because the methodology was imprecise, doesn't mean that the underlying phenomenon might not be real. Maybe Internet users really do have different gut reactions to pictures of black women and white women breastfeeding.

One quick way to get a rough answer is Amazon's Mechanical Turk service, where you can pay legions of workers some small amount of money per person to complete some menial task that can't be automated by a computer. I've used it dozens of times for surveys (such as gauging whether people would strongly prefer slideout keyboard phones) and for amateur psychological experiments (including one experiment which suggested that people who answered a math problem correctly were more likely to disagree with an attorney general's dubious legal argument). So I created a poll on Mechanical Turk, limited to U.S. users and with a payout of 25 cents for each person who answered. The poll asked:

Our academic department has asked everyone to submit a "fun" photo of themselves, so that our photos can be displayed together on the department home page. One of our employees submitted a photo that has caused some internal debate about whether the photo is inappropriate. I wanted to do a poll to get the opinion of a random sample of Internet users of different backgrounds.

Do you think this is an appropriate picture to be used in a photo collection on our academic department home page?

Since the original photos had been published in different contexts anyway, I tried to find a middle ground for the wording of the survey question, to emphasize that the photos were going to be published in a "fun" setting, but still integrated into the women's professional environments. The survey-takers were then (randomly) shown either the black woman's photo or the white woman's photo, and answered "Yes, the image is fine" or "No, the image is inappropriate". Then respondents were asked to fill in their age, gender, ethnicity, and education level.

(One thing that I've found with all of my previous surveys on Mechanical Turk, is that there is strong evidence that survey-takers are not answering randomly. Strong correlations often occur where you would expect them to -- for example, in a survey about what are the greatest causes of global strife, the same people tend to select "Energy shortages" and "Environmental damage" above other options, whereas another subgroup will tend to select both "Atheism" and "Decline of traditional values". And any survey where I've added a textbox for users to enter "more thoughts", most users enter something reasonably thoughtful which corresponds to the multiple-choice answers they've selected. Formal research by the psychologist Samuel Gosling has similarly found that Internet surveys can be useful for psychological research and are not plagued with bot-responders or random answers. So I'm working under that assumption.)

The results: Out of 47 respondents who saw the black girl's picture, 36 said the image was inappropriate (77%). Out of 54 respondents who saw the white girl's picture, 38 said the image was inappropriate (70%). For such a small sample, that's not enough to definitively say whether the small difference is due to random chance, or due to small differences in opinion in the population being surveyed. What it does show, even with such a small sample, is that in the underlying population there's almost certainly no huge gap between people's opinions of black women vs. white women breastfeeding in photos.

In both surveys, both male and female respondents voted the photos "inappropriate" with about the same frequency. For the black woman's photo, 22 out of 26 men (86%) and 14 out of 21 women (67%) voted the photo inappropriate; for the white woman's photo, 19 out of 30 men (63%) and 19 out of 24 women (79%) voted it inappropriate. There also didn't appear to be any correlation between the age of the respondents and their responses. (You can view the breakdown of answers in terms of respondent demographics here for the black woman's picture and here for the white woman's picture; the crummy layout is because I just copied-and-pasted the output from my own custom-written survey-taking tool, where I usually just view the results for myself.) As for the gap between black and white survey-takers, in the case of the black woman's photo, 24 out of 34 white survey-takers (70%) and 5 out of 6 black survey-takers (83%) voted it inappropriate, while for the white woman's photo, 25 out of 36 white survey-takers (69%) and 4 out of 4 (100%) of black survey-takers voted it inappropriate -- but those discrepancies probably don't mean much, since the population of self-identified black respondents was too small in both cases to draw any conclusions.

Even with small samples, though, I would argue that this is a better way to answer the question of latent racism than to draw fuzzy conclusions based on the trolling comments posted on a Facebook photo. My guess is that even if there was an underlying difference in the frequency of negative comments posted to the two photos, part of it could have been due to the photo being posted in a Facebook group titled "Black Women Do Breastfeed", a group name that is practically begging for trolls to wait for a chance to try and provoke an outraged response. The white woman's photo, on the other hand, was posted on the University of the Sunshine Coast Facebook page, which is not the kind of place that maladjusted nitwits hang out trying to start a flame war. And for the trolls who did post on the white woman's photo, their natural inclination would be to make some immature comment about b00bs; whereas for the trolls posting on the black woman's photo, the easiest cheap shot would be to make it about race. But that doesn't mean that there is actually a racially motivated difference in people's reactions to the photos.

Besides, if you want to use Facebook to raise awareness of racism, there are properly controlled scientific experiments that have demonstrated the extent of prejudice, such as the infamous 2003 resume callback experiment which showed that resumes with white-sounding names on them received about 50% more callbacks than resumes with black-sounding names. A viral story with 24,000 Facebook shares, about two isolated incidents under different circumstances, is not necessarily evidence of racism. It might be. But you have to do some kind of controlled experiment to check first.

Slashdot regular (and Couchsurfing.org volunteer) Bennett Haselton writes with a report that an anonymous prankster hacked the Couchsurfing.org website and sent spam to about 1 million members, snarkily advertising their commercial arch-rival Airbnb as "the new Couchsurfing." (Read on below for more on the breach.) As of now, the spam's been caught, but not the spammer.

I've been a volunteer host on Couchsurfing.org for 16 months. Despite the ongoing controversies surrounding the site's changes in recent years, I've always found it to be a great way to meet travelers with fascinating stories and to make new friends, not to mention a way to force a deadline upon yourself to clean up your house before the next guest arrives.

On August 15, I received an email sent from "Couchsurfing <noreply@couchsurfing.org>" with the subject "Site Improvements", which read:

Hi!

We have some exciting news. Find out more about the new CouchSurfing here.

The CouchSurfing team

but the hyperlink on the word "here" did nothing when I clicked on it. So I looked at the HTML source code of the message and saw that the source code of the link was: We have some exciting news. Find out more about the new CouchSu= rfing <a href=3D=E2=80=9Chttps://www.airbnb.com/signup_login=E2=80=9D> her= e </a>.

So... the email from Couchsurfing was promoting a link to their commercial arch-rival, Airbnb.

At that point I assume the message was spam that had been sent from some third-party server and simply forged a return address from couchsurfing.org, but the message headers clearly showed that the message really had been sent from Couchsurfing: Received: from messaging3.couchsurfing.com (messaging3.couchsurfing.com. [54.236.187.135]) by mx.google.com with ESMTP id v7si15118226qay.99.2014.08.15.21.30.16 for <bennetthaselton@gmail.com>; The complete message headers and message source are here.

I sent a message to Couchsurfing tech support asking if they knew what had happened, and I started a thread on the Seattle Couchsurfing page, where several other users chimed in that they had received the same email. Couchsurfing support replied to me on August 18th:

Hello Bennett,

Thanks for your patience while we have been looking into this. As you saw yourself, some Couchsurfing members received an email in error on Friday night -- we apologize.

The part of Couchsurfing’s system that sends email to members was breached Friday night and an email was sent to approximately 1 million members. We take this very seriously, and we will continue to investigate and take all appropriate action until this situation is resolved.

There is no action you need to take to secure your account. Once we have further information, we will be sure to send out updates.

Warm Regards,

Then on August 19th, I received an email from Couchsurfing (presumably along with all or most other Couchsurfing users) with the subject "Incorrect email -- our apologies":

Dear Bennett Haselton:

We're writing because you may have received an odd email from Couchsurfing in the last few days titled "Site Improvements."

We apologize for any confusion this may have caused -- it should not have been sent.

-- The Couchsurfing Team

Want more details? Find them here

where the "here" link further explains: "The message was sent by an unauthorized user of our email system. No other systems were compromised, and we've addressed the circumstances that led to this unauthorized use."

So, kudos to Couchsurfing for at least alerting users that something had gone wrong. (Judging from the reactions in the thread that I started, most users who received the email simply deleted it without a second thought after seeing that the link didn't work, so Couchsurfing probably could have said nothing to their users at all, and gotten away with it. As of this writing, a Google News search for "couchsurfing hacked" turns up no other articles about the incident, so it's not as if there was a mob clamoring for answers that they had to respond to.)

On the other hand, I hope Couchsurfing is more forthcoming in the next few days about how much they know about what actually happened. When they say "We've addressed the circumstances that led to this unauthorized use," that probably means that they at least know whether the email was sent by (a) a disgruntled employee (or recently fired employee whose credentials still enabled them to access the server); or (b) someone who used an unpatched security hole to break in from the outside; or (c) something else. (I replied to the tech support ticket asking as much, but as of this writing I have not received a reply. I wasn't naive enough to think that they were probably going to tell me everything they knew, but it's one of those rituals that quasi-journalists engage in so that we can say "as of this writing I have not received a reply".)

Obviously I think it's unlikely that anyone at the real Airbnb would actually risk jail time by hacking Couchsurfing's servers to send out spam advertising the Airbnb website; it seems more like the actions of someone being snarky, possibly a former employee or an outsider with an axe to grind. Couchsurfing's apology email said "Once we have further information, we will be sure to send out updates." Hope so.

Bennett Haselton writes: I can't stand switching from a slideout-keyboard phone to a touchscreen phone, and my own informal online survey found a slight majority of people who prefer slideout keyboards even more than I do. Why will no carrier make them available, at any price, except occasionally as the crummiest low-end phones in the store? Bennett's been asking around, of store managers and users, and arrives at even more perplexing questions. Read on, below.

In my rant about the sucky LG Optimus phone that I got from T-Mobile, I admitted that I stuck with it anyway and let them keep my money, because I couldn't stand switching away from the slideout keyboard on the phone. Same reason that I kept the Stratosphere from Verizon for so long, despite the other features of that phone sucking too. But after failing to find even one true smartphone with a slideout keyboard after visiting the local AT&T, Verizon, Sprint and T-Mobile stores, I started to wonder if I was just an old fud who couldn't get with the times.

(The slideout keyboards are usually called "QWERTY keyboards" in the marketing, but I'm using "slideout keyboard" in order to distinguish them from phones like Blackberries that have a physical QWERTY keyboard and screen all on the outer surface of the phone, since that forces the keyboard and the screen to be much smaller.)

Slideout keyboards have always felt more natural to me in a couple of ways. You can let your finger or thumb center on the correct key, and then press the key in a separate action, resulting in far fewer typos then if you're required to land your fingertip on the correct spot on the screen. (Fewer typos also means you can turn off autocorrect and worry about fewer idiotic auto-corrections.) A slide-out keyboard also makes it easier to hold the phone in a relaxed grip -- with the keyboard out, you can rest the phone on your other fingers while using your thumb to keep it in place, rather than having to grip the phone around the edges with your fingers to keep the screen uncovered. The relaxed thumb-centered grip makes it much easier to tilt the phone at different angles and even hold above your head without dropping it (handy for the first texts you answer before getting out of bed), all while hardly having to tense your fingers at all.

I mentioned this to the Sprint sales guy and he shook his head and said, "Oh, no, everybody wants touchscreen phones now." When I mentioned later to the AT&T store manager that I felt I must be in a shrinking minority, he said that he preferred slide-out keyboards, most other people preferred slide-out keyboards, and the industry was just moving away from them regardless. Who was right? Skeptical as ever about people's claims that they've "heard lots of people saying so-and-so," I posted a survey on Amazon's Mechanical Turk ( which I have used in the past for all kinds of weird stuff), seeking out respondents who had used both a phone with a slideout keyboard and a phone with a virtual keyboard, and asking which one they preferred, and why.

Out of 49 respondents, 27 said they preferred slideout keyboards and 22 said they preferred virtual keyboards. And I know the Internet survey-takers weren't just clicking answers at random, because most of them gave details as to the reason for their preference (even though this was not enforced by the survey form). Obviously that's too small of a sample to be very precise about the percentage of users that prefer slide-out keyboards (apart from the fact that Mechanical Turk users are unrepresentative of the general population in several ways), but it does mean that the near-extinction of slideout-keyboard phones in retail stores is probably not in proportion to what people actually want.

You can download the raw survey data here; some of the highlights from people who said they preferred slideouts:

"I preferred using an actual keyboard because I can actually feel the keys. After my hands get used to the keyboard, I could type very fast. Using a virtual one is much harder because you don't actually feel the keys you are typing."

"I can put my fingers on the actual keys just like a typewriter and know they won't slip off and hit the wrong key. I was heartbroken when then got rid of almost all qwerty keyboards in the new phones. They are now almost impossible to find."

"The slide-out keyboard offers more accuracy and feedback than a virtual keyboard. I can easily tell if I'm pressing the wrong letter key on a physical keyboard than a virtual one. I also prefer my keyboard to be off of the screen so I can easily see what I'm typing."

"I think its easier to type on a slide out keyboard. With the virtual ones I'm always spending half the time correcting the mistakes."

"I preferred slide-out keyboards because you could actually feel the crevices that separate each letter on the keyboard, and this allowed you to type much more efficiently. There's just something more beautiful and human about physically touching something rather than using the heat in your fingers to make unreal letters type on a screen."

On the other side of the aisle, the most common reasons that people gave for preferring virtual keyboards were that slideouts were too flimsy or bulky:

"Virtual keyboards are sturdier than slide out keyboards."

"The decreased overall weight of the device due to the lack of physical keyboard is the biggest benefit to me. Plus the added benefit is that virtual keyboard technology has come a long way in the last few years and offers unique features such as swiping words whereas a physical keyboard still limits you to typing and switching between buttons and the screen in order to select or correct words."

"A virtual keyboard is faster and less cumbersome than a slide out keyboard."

"I liked the tactile feeling of the slide out keyboard. I found the keyboard slide to be more bulky however. I like the virtual keyboard because it allows me to use a larger amount of screen space on my phone when I am not typing. You can also do cool keyboard gestures with the virtual keyboard, such as sliding the finger to type. The virtual keyboard also has an auto correct feature built in which is handy. My old slide out keyboard phone was cool at the time but lacks the features modern virtual keyboard have. Also, real keyboards make clicky noises, which can prevent you from sending texts out under your desk during meetings, haha."

(That last guy's right -- I've been out of the workforce long enough that I forgot you can't get away with texting in a meeting on a slideout, unless other people in the room are covering your noise by "taking notes" typing on their laptops.)

So - not everyone wants slideout keyboards, but a lot of people really, really want them, and the stores refuse to stock them. What gives?

The AT&T store manager simply said that they were more expensive to make, and people return them more often because they break more easily. Well of course it makes sense that the extra component costs more, but it seemed counterintuitive that the slideout keyboards are usually only found on the cheapest phones in the store (which don't qualify as true smartphones). It's odd for an expensive extra component to be found only in the cheapest models of a product line, as if Ford had announced that their self-parking technology would only come bundled with the Fiesta.

More importantly, it seems strange that a more expensive or even a more fragile component, cannot be made available at any price when so many people want it. If it costs more, surely they could just charge more. I'd pay at least an extra $100-$200 for a phone with a slideout keyboard (which is more than the entire retail cost of a dumbphone with a slideout keyboard, so the price increase on a real phone should be less than that). If it makes the phone more fragile and more likely to be returned, surely that could just be reflected in a higher monthly "insurance" fee to cover the cost of exchanging damaged phones (which is only about $5 per month anyway). Is this another example of market failure, even in a competitive industry? It's easy for Facebook to force changes down our throats, since we have nowhere else to go, but how did Verizon, AT&T, T-Mobile and Sprint all end up abandoning such a sizable portion of their customers, even while locked in a cutthroat battle with each other?

Maybe this can be the next big thing that T-Mobile does to differentiate themselves from everybody else (like when they broke ranks and decided to sell all phones at retail price with no long-term contracts) -- everybody knows their network is spottier, but it's usable, and if they're doing one thing right that you really care about, and everyone else is doing it wrong, that's reason enough to switch. Their pink-shirted CEO certainly likes making waves with his colorful metaphors about the other carriers screwing you over. If T-Mobile sold me a real phone with a slideout keyboard, I'm sure I'd stay with them for years, even though yesterday the rain (a fairly common phenomenon here in Bellevue, where T-Mobile U.S. is headquartered) caused the reception on the phone to go from 4G to 2G and then down to "G," which I didn't even know was a thing.

Bennett Haselton writes: I can't stand switching from a slideout-keyboard phone to a touchscreen phone, and my own informal online survey found a slight majority of people who prefer slideout keyboards even more than I do. Why will no carrier make them available, at any price, except occasionally as the crummiest low-end phones in the store? Bennett's been asking around, of store managers and users, and arrives at even more perplexing questions. Read on, below.

In my rant about the sucky LG Optimus phone that I got from T-Mobile, I admitted that I stuck with it anyway and let them keep my money, because I couldn't stand switching away from the slideout keyboard on the phone. Same reason that I kept the Stratosphere from Verizon for so long, despite the other features of that phone sucking too. But after failing to find even one true smartphone with a slideout keyboard after visiting the local AT&T, Verizon, Sprint and T-Mobile stores, I started to wonder if I was just an old fud who couldn't get with the times.

(The slideout keyboards are usually called "QWERTY keyboards" in the marketing, but I'm using "slideout keyboard" in order to distinguish them from phones like Blackberries that have a physical QWERTY keyboard and screen all on the outer surface of the phone, since that forces the keyboard and the screen to be much smaller.)

Slideout keyboards have always felt more natural to me in a couple of ways. You can let your finger or thumb center on the correct key, and then press the key in a separate action, resulting in far fewer typos then if you're required to land your fingertip on the correct spot on the screen. (Fewer typos also means you can turn off autocorrect and worry about fewer idiotic auto-corrections.) A slide-out keyboard also makes it easier to hold the phone in a relaxed grip -- with the keyboard out, you can rest the phone on your other fingers while using your thumb to keep it in place, rather than having to grip the phone around the edges with your fingers to keep the screen uncovered. The relaxed thumb-centered grip makes it much easier to tilt the phone at different angles and even hold above your head without dropping it (handy for the first texts you answer before getting out of bed), all while hardly having to tense your fingers at all.

I mentioned this to the Sprint sales guy and he shook his head and said, "Oh, no, everybody wants touchscreen phones now." When I mentioned later to the AT&T store manager that I felt I must be in a shrinking minority, he said that he preferred slide-out keyboards, most other people preferred slide-out keyboards, and the industry was just moving away from them regardless. Who was right? Skeptical as ever about people's claims that they've "heard lots of people saying so-and-so," I posted a survey on Amazon's Mechanical Turk ( which I have used in the past for all kinds of weird stuff), seeking out respondents who had used both a phone with a slideout keyboard and a phone with a virtual keyboard, and asking which one they preferred, and why.

Out of 49 respondents, 27 said they preferred slideout keyboards and 22 said they preferred virtual keyboards. And I know the Internet survey-takers weren't just clicking answers at random, because most of them gave details as to the reason for their preference (even though this was not enforced by the survey form). Obviously that's too small of a sample to be very precise about the percentage of users that prefer slide-out keyboards (apart from the fact that Mechanical Turk users are unrepresentative of the general population in several ways), but it does mean that the near-extinction of slideout-keyboard phones in retail stores is probably not in proportion to what people actually want.

You can download the raw survey data here; some of the highlights from people who said they preferred slideouts:

"I preferred using an actual keyboard because I can actually feel the keys. After my hands get used to the keyboard, I could type very fast. Using a virtual one is much harder because you don't actually feel the keys you are typing."

"I can put my fingers on the actual keys just like a typewriter and know they won't slip off and hit the wrong key. I was heartbroken when then got rid of almost all qwerty keyboards in the new phones. They are now almost impossible to find."

"The slide-out keyboard offers more accuracy and feedback than a virtual keyboard. I can easily tell if I'm pressing the wrong letter key on a physical keyboard than a virtual one. I also prefer my keyboard to be off of the screen so I can easily see what I'm typing."

"I think its easier to type on a slide out keyboard. With the virtual ones I'm always spending half the time correcting the mistakes."

"I preferred slide-out keyboards because you could actually feel the crevices that separate each letter on the keyboard, and this allowed you to type much more efficiently. There's just something more beautiful and human about physically touching something rather than using the heat in your fingers to make unreal letters type on a screen."

On the other side of the aisle, the most common reasons that people gave for preferring virtual keyboards were that slideouts were too flimsy or bulky:

"Virtual keyboards are sturdier than slide out keyboards."

"The decreased overall weight of the device due to the lack of physical keyboard is the biggest benefit to me. Plus the added benefit is that virtual keyboard technology has come a long way in the last few years and offers unique features such as swiping words whereas a physical keyboard still limits you to typing and switching between buttons and the screen in order to select or correct words."

"A virtual keyboard is faster and less cumbersome than a slide out keyboard."

"I liked the tactile feeling of the slide out keyboard. I found the keyboard slide to be more bulky however. I like the virtual keyboard because it allows me to use a larger amount of screen space on my phone when I am not typing. You can also do cool keyboard gestures with the virtual keyboard, such as sliding the finger to type. The virtual keyboard also has an auto correct feature built in which is handy. My old slide out keyboard phone was cool at the time but lacks the features modern virtual keyboard have. Also, real keyboards make clicky noises, which can prevent you from sending texts out under your desk during meetings, haha."

(That last guy's right -- I've been out of the workforce long enough that I forgot you can't get away with texting in a meeting on a slideout, unless other people in the room are covering your noise by "taking notes" typing on their laptops.)

So - not everyone wants slideout keyboards, but a lot of people really, really want them, and the stores refuse to stock them. What gives?

The AT&T store manager simply said that they were more expensive to make, and people return them more often because they break more easily. Well of course it makes sense that the extra component costs more, but it seemed counterintuitive that the slideout keyboards are usually only found on the cheapest phones in the store (which don't qualify as true smartphones). It's odd for an expensive extra component to be found only in the cheapest models of a product line, as if Ford had announced that their self-parking technology would only come bundled with the Fiesta.

More importantly, it seems strange that a more expensive or even a more fragile component, cannot be made available at any price when so many people want it. If it costs more, surely they could just charge more. I'd pay at least an extra $100-$200 for a phone with a slideout keyboard (which is more than the entire retail cost of a dumbphone with a slideout keyboard, so the price increase on a real phone should be less than that). If it makes the phone more fragile and more likely to be returned, surely that could just be reflected in a higher monthly "insurance" fee to cover the cost of exchanging damaged phones (which is only about $5 per month anyway). Is this another example of market failure, even in a competitive industry? It's easy for Facebook to force changes down our throats, since we have nowhere else to go, but how did Verizon, AT&T, T-Mobile and Sprint all end up abandoning such a sizable portion of their customers, even while locked in a cutthroat battle with each other?

Maybe this can be the next big thing that T-Mobile does to differentiate themselves from everybody else (like when they broke ranks and decided to sell all phones at retail price with no long-term contracts) -- everybody knows their network is spottier, but it's usable, and if they're doing one thing right that you really care about, and everyone else is doing it wrong, that's reason enough to switch. Their pink-shirted CEO certainly likes making waves with his colorful metaphors about the other carriers screwing you over. If T-Mobile sold me a real phone with a slideout keyboard, I'm sure I'd stay with them for years, even though yesterday the rain (a fairly common phenomenon here in Bellevue, where T-Mobile U.S. is headquartered) caused the reception on the phone to go from 4G to 2G and then down to "G," which I didn't even know was a thing.

Bennett Haselton writes My LG Optimus F3Q was the lowest-end phone in the T-Mobile store, but a cheap phone is supposed to suck in specific ways that make you want to upgrade to a better model. This one is plagued with software bugs that have nothing to do with the cheap hardware, and thus lower one's confidence in the whole product line. Similar to the suckiness of the Stratosphere and Stratosphere 2 that I was subjected to before this one, the phone's shortcomings actually raise more interesting questions — about why the free-market system rewards companies for pulling off miracles at the hardware level, but not for fixing software bugs that should be easy to catch. Read below to see what Bennett has to say.

How long would it have taken you to find these bugs, as a beta tester?

• The phone's auto-correct changes single-quotes to double-quotes in contractions -- for example, when you type you're, the phone auto-corrects it to you"re .

• When you backspace over part of a word that you've typed and then type the rest of the word, auto-correct corrects based on the letters that you type after you've finished backspacing, rather than the letters in the entire word that you've just completed. For example, if you type couchsurfing and the phone auto-corrects it to concurring, then backspace over all of the letters except the initial co, and then type "uch" followed by a space to form the word "couch", the Optimus changes "uch" to "such" to form "cosuch", because it thinks it's auto-correcting just the "uch" fragment and doesn't see the entire word "couch".

• Taking a screen capture still doesn't work, just like it didn't work on the Stratosphere 2. There are official directions on how to do it, but you can follow the steps and nothing happens.

• The first time I launched the voice mail application, the app prompted me to freely choose a new PIN code, and then sternly warned me, Mao-like, that my supposedly freely chosen PIN code was "incorrect". (I never got it working, and just called in to the voice mail number manually whenever I wanted to check my messages.)

• When I bought a movie on Google Play and wanted to "pin" it to the phone -- i.e. download a static, non-streamed copy so that I could watch it offline, e.g. on a plane ride -- the phone didn't have enough internal storage left to save a copy of the movie (1.27 GB, most of it taken up in 1-2 MB increments by crapware already loaded on to the phone, so that only about 200 MB was left). So I tried saving the movie to a 32 GB SD card that I had plugged into the phone, but ran into the problem that Google Play wouldn't let me save the movie to the SD card, a problem described in Joe Levi's 2013 article "Why does Google hate your SD card?" and still not fixed almost a year later. (The comments posted on his article indicate that lots of people are pissed.)

  Unlike the other bugs, this may be an example of stupidity not at the testing level but at the design specification level -- perhaps this was done in a misguided effort to prevent illegal copying. But, as Levi says of this theory, "If the DRM being used on Android is sufficient enough for content providers to accept it when media is saved internally, they should also accept it when media is saved to an SD card. Otherwise, the DRM isn't really that trustworthy, is it?" It's pointless from a copy-protection point of view, since anyone who wants to pirate a movie can just download it from various BitTorrent sites anyway; all this "feature" does is alienate people who are trying to pay for a movie legally.

• In the Messaging (i.e. texting) app, you cannot search for messages by the name of the sender. Your conversations are listed in reverse chronological order by the date of the most recent message in each conversation, but to find a conversation with a particular person, you have to scroll down the entire list of conversations and keep your eyes peeled for the person's name.

• On certain mobile website forms (the Fandango site, for instance, and some others that I don't remember -- it's not clear why this happens on some website forms but not others), the phone won't let me type "special characters", the ones that appear in the upper-right corner of the keyboard keys (so that you can type the "@" symbol by first hitting the "Fn" key to access special characters, and then pressing the "2" key). This means that since I can't type the "@" symbol, I can't log in to any form that requires an email address as a username. (The workaround is to open the Gmail app, find an email address in an email message, copy the "@" symbol from the email address to the clipboard, and then paste it back in the browser form -- yes, I have to do every time I log in to a mobile site that has this problem.)

In my previous phone-suck article about the Samsung Stratosphere, I listed as many problems as I could think of at the time, and I completely forgot the fact that the phone recorded videos without any sound. (I know it wasn't a hardware problem with the microphone, since the phone app picked up my voice fine.) As part of my research into how to ruin Burning Man forever by telling "tourists" how to get there easily, I wanted to post a video of the quintessential Burning Man spectacle that makes all the dust and thirst and heat worthwhile -- and I had to post it with no sound recording, because Samsung's product testing is done by the same drunken bonobos that worked on the LG Optimus.

And both products raise the same question, not rhetorically, but seriously: How did this happen? More specifically, in a theoretical free market, any product improvement that costs only a small amount compared to the benefit it brings to consumers, should be implemented (and consumers will reward the company by paying additional dollars for the improvement, in proportion to the benefit it brings them). While it doesn't always work out that way in practice, it's hard to believe LG couldn't spring for a few English-language testers to point out that the phone shouldn't be correcting you're to you"re.

I think the answer in both cases is that the free market optimizes mainly for things that are easily quantifiable, like camera resolution and network speed, because those can be listed on the packaging and compared against other products. But the amount of stupid s*#t you run into while actually using the phone, is hard to define on an objective scale, so that's the first thing that companies will cut corners on, even if it's something that consumers would be willing to pay money for.

So my solution is still essentially the same as what I proposed after trashing the Stratosphere: Some Consumer-Reports-type outlet should rate phones on a Stupid S*#t Index (along with speed, reception, etc.), based on how much stupid s*#t they run into in a week of typical usage. Ideally the Stupid S*#t Index should be reduced to a number so that you can do a quick comparison between different models. If a cheap phone has a lot of stupid s*#t problems, but you don't mind because you want to save money, that's a valid choice, and if you want to pay more for a phone with less stupid s*#t, that's fine too. But people should know what they're buying.

More generally, I think people vastly overestimate the ability of the free market to meet consumer demand, in cases where the demand is for something that can't be easily quantified. I've spent a fair amount of time in "entrepreneurial" circles (while bouncing back and forth myself between entrepreneurship and regular jobs) and have heard the faithful reciting a lot of platitudes like "The market rewards the best product," or "Focus on building the best product you can make, and the customers will come." But most of them evidently didn't even believe it themselves -- they spent most of their efforts on search engine optimization, running content farms, networking with important business contacts, and other activities that didn't directly relate to the quality of their products. And who could blame them? Since their products weren't competing on qualities that were precisely quantifiable, there was no reason for any of them to try to create the "best" product, or even a particularly good one. And that strategy worked quite well for several of them.

On the other hand, when you're competing on a quantifiable metric like price, the best product or service can shoot straight to the top without wasting any time on zero-sum games like SEO or networking ass-kissery. If you're selling external hard drives on Amazon for $0.01, you'll make a lot of sales. You'll go broke, but in the meantime, the free market will connect you quite effectively with your customers.

So, make the mobile phone Stupid S*@t Index into something quantifiable, and maybe we'll have less stupid s#*t. One review body could publish the average rating from several different reviewers, or several different review bodies could publish their ratings and consumers could weight the averages themselves.

Not that it's a panacea -- I bought the LG Optimus not because it was the cheapest or because I didn't expect it to have bugs, but because it was the only offering with a slide-out keyboard, and I've become addicted to the precision of physical keys. (It is so much easier to let your fingertip feel its way to the right key first, and then actually press the key in a separate motion, rather than having to hope your fingertip lands on the right spot in the first place.) So I never returned the phone, they kept my money, and I suppose that makes me part of the problem.

Bennett Haselton writes My LG Optimus F3Q was the lowest-end phone in the T-Mobile store, but a cheap phone is supposed to suck in specific ways that make you want to upgrade to a better model. This one is plagued with software bugs that have nothing to do with the cheap hardware, and thus lower one's confidence in the whole product line. Similar to the suckiness of the Stratosphere and Stratosphere 2 that I was subjected to before this one, the phone's shortcomings actually raise more interesting questions — about why the free-market system rewards companies for pulling off miracles at the hardware level, but not for fixing software bugs that should be easy to catch. Read below to see what Bennett has to say.

How long would it have taken you to find these bugs, as a beta tester?

• The phone's auto-correct changes single-quotes to double-quotes in contractions -- for example, when you type you're, the phone auto-corrects it to you"re .

• When you backspace over part of a word that you've typed and then type the rest of the word, auto-correct corrects based on the letters that you type after you've finished backspacing, rather than the letters in the entire word that you've just completed. For example, if you type couchsurfing and the phone auto-corrects it to concurring, then backspace over all of the letters except the initial co, and then type "uch" followed by a space to form the word "couch", the Optimus changes "uch" to "such" to form "cosuch", because it thinks it's auto-correcting just the "uch" fragment and doesn't see the entire word "couch".

• Taking a screen capture still doesn't work, just like it didn't work on the Stratosphere 2. There are official directions on how to do it, but you can follow the steps and nothing happens.

• The first time I launched the voice mail application, the app prompted me to freely choose a new PIN code, and then sternly warned me, Mao-like, that my supposedly freely chosen PIN code was "incorrect". (I never got it working, and just called in to the voice mail number manually whenever I wanted to check my messages.)

• When I bought a movie on Google Play and wanted to "pin" it to the phone -- i.e. download a static, non-streamed copy so that I could watch it offline, e.g. on a plane ride -- the phone didn't have enough internal storage left to save a copy of the movie (1.27 GB, most of it taken up in 1-2 MB increments by crapware already loaded on to the phone, so that only about 200 MB was left). So I tried saving the movie to a 32 GB SD card that I had plugged into the phone, but ran into the problem that Google Play wouldn't let me save the movie to the SD card, a problem described in Joe Levi's 2013 article "Why does Google hate your SD card?" and still not fixed almost a year later. (The comments posted on his article indicate that lots of people are pissed.)

  Unlike the other bugs, this may be an example of stupidity not at the testing level but at the design specification level -- perhaps this was done in a misguided effort to prevent illegal copying. But, as Levi says of this theory, "If the DRM being used on Android is sufficient enough for content providers to accept it when media is saved internally, they should also accept it when media is saved to an SD card. Otherwise, the DRM isn't really that trustworthy, is it?" It's pointless from a copy-protection point of view, since anyone who wants to pirate a movie can just download it from various BitTorrent sites anyway; all this "feature" does is alienate people who are trying to pay for a movie legally.

• In the Messaging (i.e. texting) app, you cannot search for messages by the name of the sender. Your conversations are listed in reverse chronological order by the date of the most recent message in each conversation, but to find a conversation with a particular person, you have to scroll down the entire list of conversations and keep your eyes peeled for the person's name.

• On certain mobile website forms (the Fandango site, for instance, and some others that I don't remember -- it's not clear why this happens on some website forms but not others), the phone won't let me type "special characters", the ones that appear in the upper-right corner of the keyboard keys (so that you can type the "@" symbol by first hitting the "Fn" key to access special characters, and then pressing the "2" key). This means that since I can't type the "@" symbol, I can't log in to any form that requires an email address as a username. (The workaround is to open the Gmail app, find an email address in an email message, copy the "@" symbol from the email address to the clipboard, and then paste it back in the browser form -- yes, I have to do every time I log in to a mobile site that has this problem.)

In my previous phone-suck article about the Samsung Stratosphere, I listed as many problems as I could think of at the time, and I completely forgot the fact that the phone recorded videos without any sound. (I know it wasn't a hardware problem with the microphone, since the phone app picked up my voice fine.) As part of my research into how to ruin Burning Man forever by telling "tourists" how to get there easily, I wanted to post a video of the quintessential Burning Man spectacle that makes all the dust and thirst and heat worthwhile -- and I had to post it with no sound recording, because Samsung's product testing is done by the same drunken bonobos that worked on the LG Optimus.

And both products raise the same question, not rhetorically, but seriously: How did this happen? More specifically, in a theoretical free market, any product improvement that costs only a small amount compared to the benefit it brings to consumers, should be implemented (and consumers will reward the company by paying additional dollars for the improvement, in proportion to the benefit it brings them). While it doesn't always work out that way in practice, it's hard to believe LG couldn't spring for a few English-language testers to point out that the phone shouldn't be correcting you're to you"re.

I think the answer in both cases is that the free market optimizes mainly for things that are easily quantifiable, like camera resolution and network speed, because those can be listed on the packaging and compared against other products. But the amount of stupid s*#t you run into while actually using the phone, is hard to define on an objective scale, so that's the first thing that companies will cut corners on, even if it's something that consumers would be willing to pay money for.

So my solution is still essentially the same as what I proposed after trashing the Stratosphere: Some Consumer-Reports-type outlet should rate phones on a Stupid S*#t Index (along with speed, reception, etc.), based on how much stupid s*#t they run into in a week of typical usage. Ideally the Stupid S*#t Index should be reduced to a number so that you can do a quick comparison between different models. If a cheap phone has a lot of stupid s*#t problems, but you don't mind because you want to save money, that's a valid choice, and if you want to pay more for a phone with less stupid s*#t, that's fine too. But people should know what they're buying.

More generally, I think people vastly overestimate the ability of the free market to meet consumer demand, in cases where the demand is for something that can't be easily quantified. I've spent a fair amount of time in "entrepreneurial" circles (while bouncing back and forth myself between entrepreneurship and regular jobs) and have heard the faithful reciting a lot of platitudes like "The market rewards the best product," or "Focus on building the best product you can make, and the customers will come." But most of them evidently didn't even believe it themselves -- they spent most of their efforts on search engine optimization, running content farms, networking with important business contacts, and other activities that didn't directly relate to the quality of their products. And who could blame them? Since their products weren't competing on qualities that were precisely quantifiable, there was no reason for any of them to try to create the "best" product, or even a particularly good one. And that strategy worked quite well for several of them.

On the other hand, when you're competing on a quantifiable metric like price, the best product or service can shoot straight to the top without wasting any time on zero-sum games like SEO or networking ass-kissery. If you're selling external hard drives on Amazon for $0.01, you'll make a lot of sales. You'll go broke, but in the meantime, the free market will connect you quite effectively with your customers.

So, make the mobile phone Stupid S*@t Index into something quantifiable, and maybe we'll have less stupid s#*t. One review body could publish the average rating from several different reviewers, or several different review bodies could publish their ratings and consumers could weight the averages themselves.

Not that it's a panacea -- I bought the LG Optimus not because it was the cheapest or because I didn't expect it to have bugs, but because it was the only offering with a slide-out keyboard, and I've become addicted to the precision of physical keys. (It is so much easier to let your fingertip feel its way to the right key first, and then actually press the key in a separate motion, rather than having to hope your fingertip lands on the right spot in the first place.) So I never returned the phone, they kept my money, and I suppose that makes me part of the problem.

Bennett Haselton writes "If you read no further, use either the BestParking or ParkMe app to search all nearby parking garages for the cheapest spot, based on the time you're arriving and leaving. I'm interested in the question of why so few people know about these apps, how is it that they've been partially crowded out by other 'parking apps' that are much less useful, and why our marketplace for ideas and intellectual properly is still so inefficient." Read below to see what Bennett has to say.

I casually asked a couple of my friends in Seattle -- where street parking is often unavailable, and parking garages vary widely in price -- if they'd ever heard of an app that would let them find the cheapest available parking garage, based on the time they wanted to enter and the time they planned on leaving. (Street parking is usually cheaper if you can find it, but the app would be useful for times that you can't find any.) Most of my friends said that they'd never heard of such an app, but they'd definitely use one if it existed. I also looked up parking apps on Google but the small subset that I randomly tried out, didn't do what I needed. So I thought about writing a "Somebody-with-more-time-than-me-should-go-and-do-this-thing" article, similar to the ride-swapping piece, when one of my friends casually mentioned the BestParking app.

Well, I tried it and it worked. (Lest I be accused of undue favoritism, ParkMe does the same thing just as well, although I didn't find it until later.) In both apps, you bring up a map centered on your current location, or scroll the map to where you plan on looking for parking later. You enter the time that you'll be entering and leaving, and the app shows a map with each parking garage represented by an icon showing the dollar amount that it will cost to park for that time. Without these apps, comparing rates is an annoyingly complex process to do by hand, in a crowded city like Seattle with many garages with different rates (and different times when their "evening rates" kick in -- usually 5 PM, but ranging from 4 to 7 PM), but the apps factor all of that in to give you the cheapest garage for the given time range. You can tap the individual garage icons for more information (if you plan on returning by 11 PM but you're not sure, you'd probably prefer a 24-hour garage instead of one that locks up at midnight). Also, if you're sitting at your computer and you already know the neighborhood where you'll be parking later, you can do the same search on each of their websites. (Although if you are on your phone, please don't do this from a moving car, duh. In Seattle there are plenty of 3-minute spots where you can pull over and do a search.)

So, I've been quite happy with both apps -- but I thought it was interesting that almost none of my friends had ever heard of them. I threw a quick survey up on Amazon's Mechanical Turk website, which I've used before for crowdsourced surveys and other experiments. I polled 50 people, offering them 25 cents apiece to answer these questions:

Would you use these apps? Section A: Parking garage app

Suppose a website and/or smartphone app existed where you could specify a neighborhood of a city, and enter a start and end time for when you wanted to park, and the app would automatically find the cheapest parking garage for that time range (assuming its too hard to find street parking).

1. Are you aware of any such apps/websites that already exist? If yes, whats the name of the app? (No need to do a web search -- only answer "Yes" if you already know of such an app or website.)

2. Would you use such an app/website if it existed? (Or, if youre aware of such an app that already exists, do you use it?)

Yes/No Section B: Spare room rental app

Suppose a website and/or smartphone app existed where you could list a room in your house as a temporary rental, and visitors to your city could rent it out for a single night, or more.

3. Are you aware of any such apps/websites that already exist? If yes, whats the name of the app? (No need to do a web search -- only answer "Yes" if you already know of such an app or website.)

4. Would you use such an app/website if it existed? (Or, if youre aware of such an app that already exists, do you use it?)

Yes/No

The second section, about a spare room rental app, was thrown in as a control in the experiment -- I knew the answer to that question (AirBnB), and I thought a large portion of the survey-takers would too, so I wanted to make sure they weren't just filling out the survey with blow-off answers to get the 25 cents as fast as possible.

Of the 50 people who filled out the survey, 14 of them said they had heard of using AirBnB, Couchsurfing, or Craigslist for the purpose of renting out a room or finding one to rent (almost all of them mentioned AirBnB specifically). But of the same 50 respondents, only two of them mentioned any parking apps that they had heard of, and only one of them mentioned one of the two that I'd found which actually worked. (The other person mentioned an app called ParkWhiz, which, when I tested it out, only displayed one $17 parking garage in a neighborhood where I know of several $5 garages, which BestParking and ParkMe did list correctly.)

This seems to confirm the anecdotal evidence from my survey of my Seattle friends -- there is a great deficiency in awareness of these apps, relative to how useful people would find them if they knew about them.

So how is it that people are finding -- or not finding -- these apps? In a Google search for "parking app", the first result was an ad for ParkWhiz. BestParking and ParkMe did show up in the results, but so did another one called Parker, as well as a Mashable article by Kate Freeman listing "7 City Parking Apps to Save You Time, Money and Gas". Of the apps listed in the article, the only city-specific one that worked in Seattle (PrimoSpot) has been discontinued, and of the non-city-specific ones, only Parker is still around. (The article doesn't even mention BestParking or ParkMe, although I don't know if they existed when it was written.) Finally, a friend in my survey told me about an app called Parkopedia, which has over 100,000 downloads on Google Play (the same as BestParking, and more than ParkMe).

So even if it did occur to you to look for a parking-garage-finding app, the problem is that if you randomly picked one of the five most popular parking apps (BestParking, Parker, ParkMe, Parkopedia, and ParkWhiz), you might accidentally pick one of the three out of five that is a fail:

• ParkWhiz, as noted above, only showed one $17 garage in a neighborhood full of other, cheaper garages.

• Both ParkMe and Parkopedia display their results as a map with an icon marking each parking garage -- but with no price information. Simply having a map of parking garage locations isn't too useful, since you could get that by searching Google Maps for "parking" anyway. In both apps, you can click on parking garage icons to bring up a window showing their rates, but in Parker most of the listed garages just said "Contact facility for current rates". Parkopedia did usually display the rates for different garages -- but it's a pain to click on each of a dozen parking garage icons looking for the cheapest one. A typical area of downtown Seattle will have one garage where you can park for $5 for the evening, surrounded by garages where parking costs $10 or more, but Parkopedia doesn't make it easy to find it. And neither app lets you specify a start and end time for your parking so that you can find the cheapest garage for that time range.

So it seems odd that according to the Google Play store, Parkopedia has more downloads than ParkMe (100,000+ vs 50,000+), even though ParkMe seems a lot more useful. Meanwhile ParkWhiz, the one that found only one overpriced parking garage in a neighborhood full of cheaper ones, has fewer downloads but a slightly higher star rating in the app store than ParkMe. Of course in my parking-app survey of friends and Mechanical Turk users, the far-and-a-way winner was simply not knowing that any of these apps existed at all.

And here's why it matters to you even if you ride a granola-powered bike to work: I think this is a confirming instance of what I've been arguing for years, that the marketplace for ideas, inventions, and intellectual property is far less efficient than most people think it is. Every day a huge amount of human capital is squandered by people trying to jostle their competitors out of Google search results, or even just trying to raise the capital to advertise their products to people who would find them extremely useful, but will never find out about it if the venture capitalists don't come through with the money to advertise it. All of that is time and effort that could have instead gone towards making the products better.

I've suggested an algorithm based on "random-sample voting" as an antidote to some of these market inefficiencies, such as stopping people from buying votes on Digg, promoting the best ideas on Obama's "We The People" petition website, or even deciding whether J.K. Rowling is the world's greatest author or just lucky. Basically, in each scenario, the competing entities -- whether apps, or songs, or ideas for improving U.S. government policy -- would be rated by a sufficiently large random sample of qualified raters. ("Qualified raters" might mean economists in the case of the White House policy-petition website, or it might mean music consumers in the case of an algorithm to find the best new songs.) Each entity would receive an average rating from those raters, and then the entities with the highest average rating would be the ones promoted to the widest audience (at the top of Google search results, for example). It sounds deceptively simple, but it's far less amenable to "gaming the system", because you can't rope in your friends to vote for your app, or pay voters to rate you highly on Digg. The only way to win in this system is to make your song, idea, or app, the best that it can be -- which means your human capital is being channeled productively, instead of being wasted hiring an SEO company to try and knock your competition out of the top spot on Google.

If competition between parking apps worked this way, then all the current users of Parker, ParkWhiz and Parkopedia, would switch to BestParking and ParkMe, saving themselves a lot of hassle in the process, and those second-rate apps would have never even gotten on the ground unless they got their act together and implemented the same features. More broadly, if competition in the marketplace of ideas worked this way, then there wouldn't be so many users who really wish they could have an app like this, without realizing that the apps exist!

One striking thing about looking at a map of downtown parking garages, is how wildly the rates vary from each other, with $15 garages situated right next to the $5 ones. In theory, in a competitive marketplace, such rates should stabilize around a single price, for goods that are roughly comparable. But the $10 lots do still manage to get some customers who don't know any better, because it's just not practical to criss-cross a grid of several dozen city blocks looking for the cheapest garage. BestParking and ParkMe help people deal with this inefficient marketplace. So it's ironic that they're being held back by a marketplace for ideas that operates just as inefficiently in its own way.

Bennett Haselton writes "If you read no further, use either the BestParking or ParkMe app to search all nearby parking garages for the cheapest spot, based on the time you're arriving and leaving. I'm interested in the question of why so few people know about these apps, how is it that they've been partially crowded out by other 'parking apps' that are much less useful, and why our marketplace for ideas and intellectual properly is still so inefficient." Read below to see what Bennett has to say.

I casually asked a couple of my friends in Seattle -- where street parking is often unavailable, and parking garages vary widely in price -- if they'd ever heard of an app that would let them find the cheapest available parking garage, based on the time they wanted to enter and the time they planned on leaving. (Street parking is usually cheaper if you can find it, but the app would be useful for times that you can't find any.) Most of my friends said that they'd never heard of such an app, but they'd definitely use one if it existed. I also looked up parking apps on Google but the small subset that I randomly tried out, didn't do what I needed. So I thought about writing a "Somebody-with-more-time-than-me-should-go-and-do-this-thing" article, similar to the ride-swapping piece, when one of my friends casually mentioned the BestParking app.

Well, I tried it and it worked. (Lest I be accused of undue favoritism, ParkMe does the same thing just as well, although I didn't find it until later.) In both apps, you bring up a map centered on your current location, or scroll the map to where you plan on looking for parking later. You enter the time that you'll be entering and leaving, and the app shows a map with each parking garage represented by an icon showing the dollar amount that it will cost to park for that time. Without these apps, comparing rates is an annoyingly complex process to do by hand, in a crowded city like Seattle with many garages with different rates (and different times when their "evening rates" kick in -- usually 5 PM, but ranging from 4 to 7 PM), but the apps factor all of that in to give you the cheapest garage for the given time range. You can tap the individual garage icons for more information (if you plan on returning by 11 PM but you're not sure, you'd probably prefer a 24-hour garage instead of one that locks up at midnight). Also, if you're sitting at your computer and you already know the neighborhood where you'll be parking later, you can do the same search on each of their websites. (Although if you are on your phone, please don't do this from a moving car, duh. In Seattle there are plenty of 3-minute spots where you can pull over and do a search.)

So, I've been quite happy with both apps -- but I thought it was interesting that almost none of my friends had ever heard of them. I threw a quick survey up on Amazon's Mechanical Turk website, which I've used before for crowdsourced surveys and other experiments. I polled 50 people, offering them 25 cents apiece to answer these questions:

Would you use these apps? Section A: Parking garage app

Suppose a website and/or smartphone app existed where you could specify a neighborhood of a city, and enter a start and end time for when you wanted to park, and the app would automatically find the cheapest parking garage for that time range (assuming its too hard to find street parking).

1. Are you aware of any such apps/websites that already exist? If yes, whats the name of the app? (No need to do a web search -- only answer "Yes" if you already know of such an app or website.)

2. Would you use such an app/website if it existed? (Or, if youre aware of such an app that already exists, do you use it?)

Yes/No Section B: Spare room rental app

Suppose a website and/or smartphone app existed where you could list a room in your house as a temporary rental, and visitors to your city could rent it out for a single night, or more.

3. Are you aware of any such apps/websites that already exist? If yes, whats the name of the app? (No need to do a web search -- only answer "Yes" if you already know of such an app or website.)

4. Would you use such an app/website if it existed? (Or, if youre aware of such an app that already exists, do you use it?)

Yes/No

The second section, about a spare room rental app, was thrown in as a control in the experiment -- I knew the answer to that question (AirBnB), and I thought a large portion of the survey-takers would too, so I wanted to make sure they weren't just filling out the survey with blow-off answers to get the 25 cents as fast as possible.

Of the 50 people who filled out the survey, 14 of them said they had heard of using AirBnB, Couchsurfing, or Craigslist for the purpose of renting out a room or finding one to rent (almost all of them mentioned AirBnB specifically). But of the same 50 respondents, only two of them mentioned any parking apps that they had heard of, and only one of them mentioned one of the two that I'd found which actually worked. (The other person mentioned an app called ParkWhiz, which, when I tested it out, only displayed one $17 parking garage in a neighborhood where I know of several $5 garages, which BestParking and ParkMe did list correctly.)

This seems to confirm the anecdotal evidence from my survey of my Seattle friends -- there is a great deficiency in awareness of these apps, relative to how useful people would find them if they knew about them.

So how is it that people are finding -- or not finding -- these apps? In a Google search for "parking app", the first result was an ad for ParkWhiz. BestParking and ParkMe did show up in the results, but so did another one called Parker, as well as a Mashable article by Kate Freeman listing "7 City Parking Apps to Save You Time, Money and Gas". Of the apps listed in the article, the only city-specific one that worked in Seattle (PrimoSpot) has been discontinued, and of the non-city-specific ones, only Parker is still around. (The article doesn't even mention BestParking or ParkMe, although I don't know if they existed when it was written.) Finally, a friend in my survey told me about an app called Parkopedia, which has over 100,000 downloads on Google Play (the same as BestParking, and more than ParkMe).

So even if it did occur to you to look for a parking-garage-finding app, the problem is that if you randomly picked one of the five most popular parking apps (BestParking, Parker, ParkMe, Parkopedia, and ParkWhiz), you might accidentally pick one of the three out of five that is a fail:

• ParkWhiz, as noted above, only showed one $17 garage in a neighborhood full of other, cheaper garages.

• Both ParkMe and Parkopedia display their results as a map with an icon marking each parking garage -- but with no price information. Simply having a map of parking garage locations isn't too useful, since you could get that by searching Google Maps for "parking" anyway. In both apps, you can click on parking garage icons to bring up a window showing their rates, but in Parker most of the listed garages just said "Contact facility for current rates". Parkopedia did usually display the rates for different garages -- but it's a pain to click on each of a dozen parking garage icons looking for the cheapest one. A typical area of downtown Seattle will have one garage where you can park for $5 for the evening, surrounded by garages where parking costs $10 or more, but Parkopedia doesn't make it easy to find it. And neither app lets you specify a start and end time for your parking so that you can find the cheapest garage for that time range.

So it seems odd that according to the Google Play store, Parkopedia has more downloads than ParkMe (100,000+ vs 50,000+), even though ParkMe seems a lot more useful. Meanwhile ParkWhiz, the one that found only one overpriced parking garage in a neighborhood full of cheaper ones, has fewer downloads but a slightly higher star rating in the app store than ParkMe. Of course in my parking-app survey of friends and Mechanical Turk users, the far-and-a-way winner was simply not knowing that any of these apps existed at all.

And here's why it matters to you even if you ride a granola-powered bike to work: I think this is a confirming instance of what I've been arguing for years, that the marketplace for ideas, inventions, and intellectual property is far less efficient than most people think it is. Every day a huge amount of human capital is squandered by people trying to jostle their competitors out of Google search results, or even just trying to raise the capital to advertise their products to people who would find them extremely useful, but will never find out about it if the venture capitalists don't come through with the money to advertise it. All of that is time and effort that could have instead gone towards making the products better.

I've suggested an algorithm based on "random-sample voting" as an antidote to some of these market inefficiencies, such as stopping people from buying votes on Digg, promoting the best ideas on Obama's "We The People" petition website, or even deciding whether J.K. Rowling is the world's greatest author or just lucky. Basically, in each scenario, the competing entities -- whether apps, or songs, or ideas for improving U.S. government policy -- would be rated by a sufficiently large random sample of qualified raters. ("Qualified raters" might mean economists in the case of the White House policy-petition website, or it might mean music consumers in the case of an algorithm to find the best new songs.) Each entity would receive an average rating from those raters, and then the entities with the highest average rating would be the ones promoted to the widest audience (at the top of Google search results, for example). It sounds deceptively simple, but it's far less amenable to "gaming the system", because you can't rope in your friends to vote for your app, or pay voters to rate you highly on Digg. The only way to win in this system is to make your song, idea, or app, the best that it can be -- which means your human capital is being channeled productively, instead of being wasted hiring an SEO company to try and knock your competition out of the top spot on Google.

If competition between parking apps worked this way, then all the current users of Parker, ParkWhiz and Parkopedia, would switch to BestParking and ParkMe, saving themselves a lot of hassle in the process, and those second-rate apps would have never even gotten on the ground unless they got their act together and implemented the same features. More broadly, if competition in the marketplace of ideas worked this way, then there wouldn't be so many users who really wish they could have an app like this, without realizing that the apps exist!

One striking thing about looking at a map of downtown parking garages, is how wildly the rates vary from each other, with $15 garages situated right next to the $5 ones. In theory, in a competitive marketplace, such rates should stabilize around a single price, for goods that are roughly comparable. But the $10 lots do still manage to get some customers who don't know any better, because it's just not practical to criss-cross a grid of several dozen city blocks looking for the cheapest garage. BestParking and ParkMe help people deal with this inefficient marketplace. So it's ironic that they're being held back by a marketplace for ideas that operates just as inefficiently in its own way.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Slashdot contributor Bennett Haselton writes: "Internet users in Saudi Arabia, along with most users in the United Arab Emirates, are blocked by their respective government censors from accessing the websites of the Trinity Davison Lutheran Church, Deliverance Tabernacle Ministries in Pittsburgh, the Amitayu Buddhist Society of Taiwan, and GayFaith.org. An attempt to access any of those websites yields an error page like this one. However, the sites are not blocked because they conflict with the religions beliefs of those countries' governments. Rather, they are blocked because Smartfilter -- the American-made blocking program sold by McAfee, and used for state-mandated Internet censorship in those countries -- classifies those sites as "pornography". You can see the screen shots here, here, here and here." Read on for the rest of Bennett's thoughts.

I found these blocked sites by starting with a combination of URL lists and ad hoc spidering, and running as many sites as possible through the Saudi filters to catch the ones that were blocked. Some of the sites were blocked for reasons that were easy to guess -- for example, http://www.bighornbasinsfw.org/, the home page of the Big Horn Basin, Wyoming chapter of Sportsmen for Fish & Wildlife, was almost certainly blocked because of the slang term "nsfw" in their URL. http://www.AgainstPornography.org and http://www.SearchingForMySpermDonorFather.org were presumably blocked because of the presence of the words "porn" and "sperm".

On the other hand, there appears to be no rational reason why the Filipino American Women's Network, the Tuscon Jazz Institute, or the Sacramento Police Activities League would have been blocked by Smartfilter, even by accident. A partial list of the blocked sites that I found is in the blog post I wrote for Citizen Lab, an Internet censorship research center at the University of Toronto.

Articles about sites that are erroneously blocked by Internet censorship software, have a storied history. The first widely read piece was the article "Keys to the Kingdom" written by Brock Meeks and Declan McCullagh in 1996, calling out Cyber Patrol for blocking EnviroLink.org and the University of Newcastle Computer Science Department, and CYBERsitter for blocking the National Organization for Women. I made a minor name for myself and the Peacefire.org site in the late 1990's by writing more pages about sites blocked by other products, including some (like X-Stop and SurfWatch) which no longer exist, and others that are still around, including Smartfilter. I was also one of six people comprising the Censorware Project, a loosely organized group of volunteers that published a few more reports.

By the early 2000's, however, it became clear that anyone whose mind was likely to be changed by information about what kinds of sites were blocked by blocking software, would have changed their mind already (or would, if they came across the research that had already been done up to that point). So the further reports on Internet blocking software errors, by me and other people, slowed to a trickle. I wrote a report in January 2002 on the latest list of sites blocked by Cyber Patrol, a product that most people today have forgotten. In 2006 I worked with the ACLU of Washington to publish a report on sites erroneously blocked by FortiGuard, a program used on computers in some libraries in central Washington, as part of the ACLU's suit to challenge the constitutionality of the program's use on public library terminals. (The Washington State Supreme Court rejected the lawsuit on the grounds that, regardless of what sites were blocked on the computers, it didn't matter because an adult library patron could request for the filter to be turned off.) In 2007 I wrote an article for Slashdot titled "From Bess to Worse" listing some sites that were blocked by an Internet filtering program called Bess (which was later bought out by Smartfilter and discontinued).

Most people's awareness of this debate, if they had heard about it at all, was limited to the perception that "breast cancer sites" and sites about "chicken breast recipes" were sometimes filtered by Internet blocking programs. Or they heard that "Beaver College" actually had to change its name to avoid being censored by web filters. As I tried to explain in a FAQ (written, according to the Wayback Machine, in 1999, but which still broadly holds true today), these examples are true, but they miss the point. These examples make it sound as if blocking software companies are doing the best job they can under the circumstances, and that the errors are unavoidable due to limitations on machine intelligence. In reality, any software algorithm that blocks the American Board of Vocational Experts, the Hopewell United Methodist Church, and the Patriot Guard Riders of Mississippi, as "pornography" (as Smartfilter currently does), is probably not the best algorithm the company could have come up with -- but there's no incentive for them to try harder, because few people will ever look that deep.

And yet, people continue to remember the "breast cancer site" examples. This sounds to me like an example of the narrative fallacy -- people remember that breast cancer sites were blocked, because there's a tidy explanation. There is no tidy explanation for most other examples of blocked sites, so the meme never spreads very far. Conveniently for the blocking companies, the blocked-site errors which make the company look most sloppy (the Kennels at Simpson Creek Farms, the St. Francis Institute of Milwaukee, etc.) are precisely the ones that, due to the narrative fallacy, most people won't remember or hear about.

One company, CYBERsitter, did manage to make a few blocking decisions in the 1990s that were egregious enough that their antics did make the news, and did finally raise some people's awareness that the controversy over private Internet filtering extended beyond "breast cancer sites". After TIME Magazine's website published an article (no longer online) that criticized CYBERsitter's blocking policies, CYBERsitter responded by blocking TIME Magazine's pathfinder.com domain. A few months earlier, CYBERsitter had blacklisted the monthly e-Zine "The Ethical Spectacle, after the Spectacle's founder, Jonathan Wallace, published an article criticizing CYBERsitter for blocking my own Peacefire.org website. And Peacefire.org had been blocked, in turn, because of a page I wrote (now very much out of date) listing some of the sites that CYBERsitter blocked, including the International Gay and Lesbian Human Rights Commission and Mother Jones. (Nowadays, of course, nobody would be surprised that filtering companies block Peacefire.org, since the site publishes ample instructions on how to get around Internet blockers. But at the time, the site's first and only article was the list of sites blocked by CYBERsitter, which is why CYBERsitter received so much criticism for blocking the domain in retaliation.) CYBERsitter also threatened to have Meeks and McCullagh criminally prosecuted for writing "Keys to the Kingdom" and threatened to sue me over the page that I had made.)

The moral, it seems, is that if you want an example of a censored web site to stick in people's minds, it either has to be a forgivable error, or an insane vindictive dick move -- because in either of those cases, people will understand why it happened. The vast swaths of censored websites on the spectrum in between, the ones for which there is no rational explanation for the blocking, go ignored.

These days, though, American and Canadian "censorware" makers have also come under fire for selling censoring software to foreign governments which use them for country-wide censorship. Most of the criticism focuses, naturally, not on the kinds of sites that are accidentally blocked by the blocking software, but on the immorality of these companies enabling statewide foreign censorship in the first place. Netsweeper, Blue Coat, and McAfee have all made the claim that "Once we sell their product to them, we have no control over what they do with it" -- which, as I wrote previously in Slashdot, is nonsense, because for the product to be effective, it has to rely on updates to the blocked-site list, which are provided at regular intervals by the manufacturer. Cut off the updates, and the product will not work, at least not as well.

So the fact that McAfee has classified the Boy Scout Troop 87 of North Andover, the Pan-Iranist Party of Iran, and Reptile Conservation International as "Pornography" is (rightly) overshadowed by the fact that McAfee is selling to government censors in Saudi Arabia and the UAE in the first place. However, as long as the filters are installed, these blocked sites are at least part of the problem for users in those countries, just as much as they are for students or cubicle workers in the U.S. whose network administrators happen to use Smartfilter. And, of course, I sampled only a miniscule fraction of the Web to find these examples of blocked sites, so the true number of stupid blocks affecting Saudi and UAE users is likely to be much larger. For each individual example, you might reasonably ask, "Is it really a big deal if Saudis are blocked from accessing Boy Scout Troop 87 of North Andover?" But it adds up.

Bennett Haselton writes "Google created controversy by announcing that Google+ users will now be able to send email to Gmail users even without having those Gmail users' email addresses. I think this debate misses the point, because it's unlikely to create a deluge of unsolicited email to Gmail users, as long as Google can throttle outgoing messages from Google+ users and terminate abusive accounts. The real controversy should be over the fact that Google+ users can search a public database of the names of all Gmail users in the first place. And limiting the ability of Google+ users to write to those Gmail accounts, won't do anything to address that." Read below to see what Bennett has to say.

To begin with, remember that on Facebook (which I no longer use, but which I keep up with) does allow you to search for other members' names and send them messages even if they have not yet accepted your friend request. Facebook users are generally not shy when it comes to complaining about problems with the site, but I've never heard Facebook users complaining about junk messages from strangers. (It's true that if you get a message from a user outside of your friends list, it gets routed to the "Other" folder of your Facebook inbox. But similarly, Google says that messages from strangers on Google+ will get routed to a Gmail user's "Social" tab of the inbox.)

So I expect the amount of actual unsolicited emails from Google+ users to Gmail users to be almost a complete non-issue, for the same reason that it's not an issue on Facebook. I assume the reason that Facebook users get so few junk messages, is that Facebook can limit the number of outgoing messages sent per day by any one account (although I don't know what that limit is), and can shut down accounts that are reported for abuse. Yes, a spammer could continually create new accounts to send more messages, but if you create too many Facebook accounts from the same IP address, and each account created from that IP address gets flagged for abuse, Facebook might start disallowing new accounts created from that IP. You could switch your IP address continually, but at a certain point, spammers must have decided that creating disposable Facebook accounts for spamming purposes wasn't worth the trouble, because the simple fact is that they don't do it. So Gmail users are not in danger of buried in spam from Google+ accounts. (By contrast, conventional email spam grew to unmanageable proportions because anybody with an email server could send out millions of messages per day, unless their provider cut them off.)

On the other hand, I think we should be more concerned about the fact that anyone who creates a Gmail address automatically has a Google+ account created for them. This doesn't just mean that any of Google's claims about the "number of Google+ users" are inflated, if they're including everyone who signs up for a Gmail account. (That's a valid complaint, but it's between Google and their shareholders, since the rest of us don't need to care how many users Google+ actually has.) More importantly, it means that all of those users become part of a public database that is searchable by name.

As a test, I went to Gmail.com and created a new user account, entering the first and last name "Zanzibar Higglesbrain" which I figured was probably unique. (Fan fiction authors: knock yourselves out.) Then I logged back in under my own Google+ account, went to the people search page, searched for "Zanzibar Higglesbrain", and found 1 match. (I didn't even need the exact name -- entering "Zanzibar Hi" into the people search box, listed Mr. Higglesbrain among the results.)

Now, when I created the Higglesbrain account, how much up-front notice was I given that I would be adding myself to a public database? I went through the normal signup process, viewed through the eyes of a novice -- after typing in Gmail.com, I was redirected to a page on accounts.google.com with the innocuous title "Create your Google Account", and entered my personal information. On the next page is the somewhat confusingly worded message (I've also posted a screen shot here):

How you'll appear

Choose how you appear across Google by creating a public Google+ profile.
Include a photo - you can update it at any time.
[Link:] Add a photo
[Button:] Next step

This message is misleadingly worded because the phrase "by creating a public Google+ profile" implies that's something you can do, optionally, if you want to. It doesn't really disclose the fact that the profile is being created for you as a side effect of signing up for Gmail. The wording might be interpreted, rather, to mean that your profile will only be created if you upload a photo (which is not the case; your profile gets created regardless). And besides -- what if the user is a novice who went to Gmail.com because they saw all their friends using Gmail.com addresses, and have never even heard of "Google+"? If they haven't consented to their name being added to a publicly searchable database, it shouldn't be their responsibility to know what "Google+" is, so that they can object to their name being listed there.

After you click the "Next step" button, the final page in the account creation process says:

Welcome, [firstname]

Your new email address is [address]

Thanks for creating a Google Account. Use it to subscribe to channels on YouTube, video chat for free, save favorite places on Maps, and lots more.

Note what's conspicuously missing from this message: It doesn't mention Google+ at all, much less the fact that you have unwittingly "joined" it, where other users can find you.

I can think of a couple of scenarios where a user might object to their name being listed in a searchable user database, apart from just "on general principles". If you have a stalker in your past, and they find your name on Google+, it confirms for them that you're probably still alive, that you're probably active on the Internet, and that you're still going by the name that they knew you under. Or, if you have a very unique first name, anyone who knows it could search on Google+ to find your last name, even if you didn't want them to. Similarly, if you have a very unique last name, someone could use the search feature to find the names of your children and other relatives with the same last name, at least those of them that are using Gmail.

And this lack of user consent is a more serious problem on Gmail/Google+ than on Facebook, because most Facebook users create a profile with the general expectation that other Facebook users can find them. Some Facebook users had chosen not to make their accounts searchable -- and Facebook justifiably received a firestorm of criticism for removing that feature and forcing those users' profiles to become publicly searchable after all -- but the overwhelming majority of Facebook users had joined with the understanding that their profiles could be found by others. That's not a valid assumption about Gmail users -- if someone creates a Gmail.com email address, there's no reason to think that they believed they were joining a publicly searchable name database.

Google has tried to mollify people's concerns about emails from strangers on Google+, by specifying that anyone not already in your Google+ circles will only be able to send one message to your Gmail inbox, and will not be able to send more messages until you reply. But this misunderstands the privacy implications in, for example, the stalker scenario. If a stalker ex "Bob" really did find your name on Google+, they might try to tease out a reply by creating a Google+ account under the name of a friend "Alice" you and your ex had in common, and sending you a generic "How have you been doing lately?" message. Since that message probably won't raise any alarm bells (the message isn't asking for anything like a current address or phone number), you might not realize that just by replying, you've already done the damage (the stalker now knows your email address, plus the fact that it's still an actively used account).

Similarly, although you can modify your Gmail settings to prevent strangers on Google+ from messaging you, the ability to change a setting to fix a problem only helps a user if the user realizes when the problem is happening. For example, if the problem resulting from this new feature switch were a deluge of spam from strangers on Google+, then more and more users would get frustrated and look for information about how to stop the flood of spam, and most of them would find out about this setting and switch it off. But for combatting the stalker problem, this setting is useless, because by definition if a stalker finds you on Google+ (and tricks you into replying to a message and revealing your email address), you wouldn't know about that problem until the damage has already been done, at which point it's too late to solve it by changing a setting.

The only way to avoid this risk to people's privacy, would be for Google to ask Gmail users at the time they create a Gmail account: "Do you also want to create a Google+ account, yes or no? This means you will have a publicly searchable profile, and people who know your name will be able to find you." Some people would like to be found, some people would rather not be, and this would allow them to sort themselves properly.

But instead, we have an untold number of zombie Google+ accounts created whenever someone signs up for Gmail, which serve no purpose except to make it possible to find people who never confirmed that they wanted to be found -- all most likely for the reason given by Chris Taylor at Mashable, so that "Larry Page gets to claim increased Google+ user numbers on the next quarterly earnings call."

Slashdot contributor Bennett Haselton writes "A recent webinar for newsletter publishers suggested that if you want your emails not to be blocked as 'spam,' you paradoxically have to engage in some practices that contribute to the erosion of users' privacy, including some tactics similar to what many spammers are doing. The consequences aren't disastrous, but besides being a loss for privacy, it's another piece of evidence that free-market forces do not necessarily lead to spam filters that are optimal for end users." Read on for the rest of Bennett's thoughts.

Lest you think that spam filters only rarely make mistakes any more, recall the instance in which after I mailed out a group of 10 proxy websites to my own mailing list, the British "anti-spam" outfit Spamhaus blacklisted two of the domains, which caused the registrar (Afilias) to disable all 10 of the domains en masse, so that the sites simply disappeared from the Web. (This happened even though our mailing list is 100% closed-loop confirmed-opt-in; users have to reply to a confirmation message in order to join the list, so the actual emails were not "spam.") It took several days to find out what happened and restore the domains, during which Spamhaus and Afilias refused to answer any of my inquiries, and have to this day not reached out or explained what they're doing to avoid similar screw-ups in the future. And this was just the latest in a long line of headaches caused by spam filters including filters at Hotmail, AOL, Yahoo, and Gmail, which had regularly categorized our emails as "spam" and caused users to miss them.

So when the email deliverability company WhatCounts announced their October 16th webinar on how to avoid having your mails blocked as spam, I watched in real time with some interest. The webinar (which you can view here), was presented by Brad Gurley, the "Director of Deliverability" for WhatCounts, who has worked in the email "deliverability" industry for 10 years. While email deliverability services is one of the products that WhatCounts charges for, the presentation didn't contain any blatant plugs for their own services, so I'm taking the contents at face value. Even if any statements in the webinar happened to be incorrect, it's still safe to assume that the presentation represents mainstream thinking in the email deliverability industry, which will determine what recommendations are made to email senders.

I hasten to add that WhatCounts should not be blamed for any of the recommendations that they made that I'm counting as "eroding privacy"; their job was to answer the question, "What is the best way to make sure my emails don't get blocked as spam?", and they answered it. The fault, if any, should lie with the spam filters which encourage these practices. Furthermore, I'm only saying that the practices encouraged in the webinar are eroding user privacy, not violating it. (If you ask every new subscriber for their name and geographic location, I would call that an "erosion" of privacy if it normalizes the practice of collecting more user data than you need, but it's not a privacy violation as long as the user willingly gives it to you.)

The webinar begins with some recommendations that are actually good netiquette, such as cleaning subscriber lists regularly (removing bouncing addresses), and displaying a prominent "unsubscribe" link for users who want to leave. If you run a newsletter, and good netiquette isn't a compelling enough reason to put an "unsubscribe" link near the top, here is a direct quote from the webinar:

"The Unsubscribe link should be prominently placed within the message body. Unsubscribe links that are hidden or hard-to-find will generate spam complaints from unhappy users who want to unsubscribe. Placing the link in the preheader has been shown to reduce spam complaints in many cases."

That's one reason that every message that I send to my own newsletter, contains this text at the top:
[You are receiving this because you subscribed to the Circumventor distribution list. To unsubscribe from this list, click here: http://www.peacefire.org/circumventor/cv-unsub.html or reply with the word "unsubscribe" in the subject.] (I give people the option of replying with the word "unsubscribe", even though that creates some hassle for me to process those requests manually, because many of our users are on censored networks and cannot access the unsubscribe link on the peacefire.org website.)

But, on to the less-stellar news: the presentation also says that the key to getting users to keep opening your emails -- and hence to signal to the email providers like Hotmail and Yahoo that your mails are not "spam" — is "engagement." Gurley suggests that senders "tailor mailings to segments of subscribers based on demographic data," including segmenting users based on city or zip code. Nothing sounds wrong with that, except that to "tailor" the mailings based on demographic data, you have to have that demographic data -- i.e. ask users for their age, sex, location, income bracket, or other information at the time that they join the list.

As I said, I don't consider this a violation of privacy if the user gives their information voluntarily, it's just an erosion of privacy, because it normalizes the process of asking users for extra data when there's no clear reason why it's necessary. In the late 1990s, you could join most companies' email lists without providing any more information than an email address; if you were asked for more information, it was for an obvious reason (such as filling out a profile on match.com, or ordering a product to be shipped). The less information about users was stored all in one place, the less opportunity there would be for the company to abuse it, or to be bought out by some other company that would abuse it, or for someone to hack into their servers and steal the information outright.

Our mailing list in particular serves a segment of the population who are particularly privacy-conscious -- they're using our proxy sites to circumvent Internet blocking software, so in almost all cases, just the simple act of being our mailing list could get them in some amount of trouble with somebody (although the severity would vary). So by design, we collect the minimum amount of information -- the email address -- necessary to send new proxy sites to the users. The more information that we asked for, the less likely the user might be to sign up in the first place.

Again, companies are within their right to ask for this information, but I don't think the rest of us newsletter publishers should be penalized for not asking for it.

The presentation goes on to say that email providers such as Hotmail and Yahoo judge whether an email is "spam" based on what proportion of the time users open an email from that sender. As Gurley says, "Give people a reason to open your email and keep opening it." The trouble is that this penalizes email notifications where you can fit all of the relevant content into the subject line -- many of my emails say something like "new Circumventor: badbadger.info", and for most users, that's all they need to see. Some subscribers have specifically said that they always want to see the new proxy site name in the subject line, because they're on a network where they are blocked from accessing their full email inbox, but they can use other webpages to see the subject lines of recently received emails. (For example, Yahoo Mail users might be on network where Yahoo Mail is blocked, but if you're signed in to yahoo.com you can see the subject lines of your last few emails on the www.yahoo.com front page.) If I'm being penalized by spam filters because user's don't open my emails, then obviously that's incentivizing me to do the users a disservice, by putting the proxy site name only in the message body.

(This might be an issue that is highly specific to my particular mailing list, because most people don't run email newsletters where they can fit all of the relevant content into the subject lines. However it's easy to think of other web applications that have a need for subject-only notifications -- Google Calendar sends me an email whenever one of my calendar events is coming up -- and those shouldn't be penalized just because the user never opens them.)

Finally, the presentation suggests that senders unsubscribe any user who hasn't opened the last 50 emails you sent them. This might set off mild alarm bells with tech-savvy readers, who know that the only way to tell if a reader has opened your message, is to embed images into the messages -- and if your newsletter content doesn't lend itself to images, you have to plant a surreptitious "web bug" image into the email, a tiny image that serves no purpose except that if you open the message and the image loads, it tells the sender that the message has been read. (For this reason, if you open an email message that does contain images, most email clients will not display them unless you click "Show images" or something similar -- because otherwise, if images always loaded automatically, spammers could use web bugs to tell who was opening their emails. So in fact, if a user opens your message and doesn't click "Show images", you generally can't tell that they opened your email.)

Again, I would consider web bugs to be an erosion of privacy more than a violation of it, on the order of asking for the user's zip code at the time they join their newsletter -- in both cases, the reason being that you are collecting more information than is strictly necessary for the operation of your mailing list. (In the case of web bugs, the "information" you're collecting is whether the user opened your message or not.)

Some people feel more strongly about it. A recent message posted on MIT's "liberationtech" mailing list had this to say about "web bugs", to a person who was asking about why his newsletter was being blocked:

You do not appear to use web bugs in your mailing list messages. A wise choice: web bugs are malware, they're invasive and abusive, and they actively degrade the security of recipients...which is a pretty crappy way to treat one's audience.

I think this is over the top -- all that a web bug does, is tell the sender whether you opened their message -- but, whether this opinion is valid or not, some people out there feel that way, and using web bugs in your email might piss them off.

Although before you cut loose the users who haven't opened your last 50 emails, Gurley's presentation also suggests trying to win them back with one last message with a "teaser" subject line like "We're saying goodbye...", or "Are we not going to talk to you any more?", or "Are we breaking up?". I hate subject lines like that, whether from spammers or from people I've signed up to get mail from. (Although now that I think about it, I doubt I'm really that mad about the 1 second of my time that they wasted; I think I just resent the fact that even just for that 1 second, they actually had me fooled, and I thought it really was a message from a friend.)

But again, we can't kill the messenger: Brad Gurley's job was to do a presentation on how to get your emails past the spam filters at the major email providers, and if using "come-on" subject lines works, because it gets more users to open your messages, then that's part of the answer. (Remember, this presentation was aimed at opt-in email senders, not spammers.)

So, I don't know that I can do anything differently with my list as a result of the presentation. I think it would be too off-putting to users to ask for their age and zip code, and in any case it wouldn't do any good for all the users who have already signed up. I probably couldn't use web bugs even if I wanted to, because the web bugs would have to load the image from a website, and if the user opened the email from a network where Web access was censored, the network's filter might block the website that the web bug loaded the image from. And for a list with many members who are still in high school, and whose parents might read their email over their shoulder, I don't feel like trying to get their attention by sending them an email with the subject "Are we breaking up?"

The more important takeaway here, though, is that there's no reason to expect the free market to deliver spam filters that are optimal from the user's point of view. In a world where users had perfect information, if Hotmail told their users, "We're going to start flagging the newsletters in your inbox as 'junk mail' unless the sender asks for your zip code when you sign up, and uses teasing subject lines to get you to open the message, and uses web bugs to verify whether you've opened it," their users would likely say, "Screw you, I'm going to Gmail!" (Which many of their users have apparently said anyway.) If this doesn't happen, it's because the vast majority of users don't have enough information for the market in spam filters to function effectively. And thus there's nothing to stop Hotmail and Yahoo from imposing arbitrary conditions on senders through their spam filters, which will lead to more legitimate senders resorting to "come-on" subject lines and web bugs -- ironically, looking more like the spammers they're trying to differentiate themselves from.

Bennett Haselton writes "How did a $400-billion company ship millions of units of a phone with a calendar app that displays the wrong date, a texting app that can't reply to group texts, a screen capture function that doesn't work, and a phone app that won't let me use the keypad unless the speakerphone is on? The answer, perhaps, suggests deeper questions about why market forces fix certain problems but not others, and what to do about it." Read on for the rest of Bennett's thoughts.

I've been using either a Samsung Stratosphere or a Samsung Stratosphere 2 from September 2012 to the present. Where to begin?

• If you open the calendar application on the Stratosphere 2, it usually highlights tomorrow's date as "Today," and lists tomorrow's calendar entries as your list of things to do "Today." Here is a picture of my phone's screen taken on June 2, with the calendar app displaying "Today, Mon, Jun 3 2013" — despite the phone knowing the correct time is 9:22 PM on June 2.

  Strangely, in the morning the calendar app would display the correct day as "Today," but would switch to the wrong day some time in the afternoon, and eventually I decided that the calendar app was probably using Coordinated Universal Time to decide what "Today" was, which is 9 hours ahead of Pacific Standard Time.

• You'll notice that these images are not screen captures, but photos taken with another phone. This is because some time between the Stratosphere 1 and 2, the screen capture function broke — every support site says you're supposed to be able to take a screen cap on a Stratosphere by pressing the Home and Power buttons at the same time, and that works on the 1, but not on the 2.

• If someone else sends a text to multiple recipients including you, the Stratosphere gives you no indication that it's a group text, and there's no way for you to see the other recipients or reply to the whole group. (I had a lot of awkward "What, you were asking everybody, not just me?" moments before I realized what was going on.) Other users have been complaining about this for months, and it apparently affects more Android phones than just the Stratosphere.

• The built-in camera refuses to take a picture if the battery is low — it just says "Warning: low battery" and exits. Yes, I know they think they're doing it for my own good since the camera is a battery hog, but a few times I've wanted to take a picture where it was well worth using up a half a percent of my remaining battery life or however much it would have taken, but the phone wouldn't let me. That should be the user's decision, dammit.

• When I was in Canada last week, if I tried sending a text message longer than 160 characters, the phone would tell me that the message sent, but it would actually fail silently and never get delivered. I'm not sure whether to blame Verizon, Android, or Samsung for this one (or just, you know), but in the end someone has to take responsibility for the product, and the phone telling you that a message was sent when it actually got lost, is a complete fail. If it doesn't work, fine, give me an error message, but never tell the user a message got sent successfully if it didn't.

• During a phone call, the on-screen keypad doesn't work unless the phone is on speakerphone. If the speakerphone is off, the screen goes dark after about 1 second of inactivity, making it impossible to enter an account number or anything else. I can avoid this bug by turning on a speakerphone (which is how I know it's a software bug, not a problem with the touchscreen), but this is a pain if I'm in a public place and don't want to annoy everyone around me who would have to listen to all the voice prompts. (The phone's software seems to be following a rule like: "If the speakerphone is not on, then when the phone moves away from the user's face, assume the user is not actively using the phone and let the screen go dark" — where the bug is that it doesn't make an exception and keep the screen on if the user is actively pressing keys on the keypad.)

At first, these and many, many other bugs produce a state of mind that transcends annoyance to reach a kind of genuine curiosity, where you're asking "How did this happen?" not rhetorically, but because you actually want to know. But eventually the surprise wears off, and you're just left with bugs that are disproportionately aggravating because they obviously would have been caught during even the most basic UI testing. They're aggravating to me not because of how much they get in the way (you eventually get used to them), but because the existence of those bugs conveys a certain lazy attitude towards finding and fixing bugs at all.

I realize this is not a logical reaction. The aggravation you feel towards a bug should depend on how much the bug actually interferes with the user experience, not on how easily the manufacturer should have found it. Rationally speaking, the biggest problem with the phone right now (and the reason I'm having to mail to back to the manufacturer for a replacement) is that the charging port spontaneously broke, so that unless the micro USB charger is plugged in exactly right, the phone can't charge (even if you get it right and form a connection successfully, the connection breaks if you move the phone half an inch). Needless to say, that's exasperating — but it's hard for me to get mad at Samsung over that, because it's not an easy defect to catch at the manufacturing stage. On the other hand, if the calendar app displays the wrong day, I would say that someone should be fired over that except that probably nobody was assigned to do that testing in the first place.

I also posted questions about each of these problems on AndroidForums.com and AndroidCentral.com (those links show all questions recently posted from my username on each site), which have so far received hundreds of "views" but no replies. I mention this because some people think that if you do run into problems like these, all you have to do is post a question and The Community will help you out with a workaround. Nope.

Also, lest you think you can do away with these bugs by downloading third-party replacements for all of these apps, I spent part of an afternoon downloading different texting apps to see if any of them would fix even part of the problems I had with the built-in one. None of them worked much better, although several of them displayed pop-up ads over every third incoming text message, and most of them did not play nicely with each other, giving me no way to disable them so that their notifications would double and triple up on top of each other for every received text. So I gave up. Even if I thought I might eventually find a better app for texting, I didn't have time to test multiple replacements for every built-in default app that didn't work.

Farhad Manjoo has a column up at Slate arguing that the reason many Android phones suck is that they're laden down with adware attempting to extract more personal information and money from the user. I'm sure that's part of the problem, but I can't see how the manufacturer is making any money off of the bugs I ran into; they were just being lazy.

The problem, I think, is that phone manufacturers know that phone reviewers (and users, when they're choosing between models in the store) will focus on easily quanitifiable attributes, such as size, weight, battery life, and the number of megapixels in the camera. The number of aggravating bugs in the user interface is not something that is easy to compare across phones (and in any case would not be printed on the box). Thus market forces simply don't favor the development of a hassle-free interface, because in most cases the phone manufacturers wouldn't be rewarded for it.

And — I don't consider this too much of a stretch — this is where it connects with larger issues for me, because I've been arguing for years that the free market will usually fail to fix certain types of problems, often in the context of threats to free speech and civil liberties, especially if the user lacks information they need to compare multiple options. A major argument in favor of Net Neutrality is that the typical user wouldn't realize it if their ISP were throttling access to certain sites; they would just think that the remote site was responding slowly. Since that information would be hidden from the user, "the marketplace" won't solve the problem on its own. Similarly, every time I say that my Circumventor mailing list keeps getting blocked as "spam" by Yahoo, Hotmail, Gmail, or AOL (despite being 100% verified-opt-in, natch), someone tells me that if the free market is blocking my emails as unwanted, it must be because the users don't want them. That the free market might make a mistake (in this case, because users don't have full information about what is getting blocked as spam), doesn't occur to them. I think the belief in the infallibility of the free market, is one of the most widespread fallacies of our era — people who would never make the mistake of confusing correlation with causation, have no problem thinking that if a product or service gets blocked by a third-party intermediary, it must be because the end user didn't want it.

And so when I'm staring at my Stratosphere's calendar telling me that tomorrow is actually today, it brings out my aggravation not just towards Samsung, Google, and Verizon, but towards all the people I've heard over the years claiming that the marketplace will automatically reward good products and punish bad ones. If there weren't so many people who believed that, maybe we could have collectively put more effort into rating phones according to their usability, knowing that the "invisible hand" of the marketplace was not likely to solve those problems on its own, and maybe these bugs would have gotten fixed. Instead, the "marketplace" focuses disproportionately on attributes like dimensions, weight, and processor speed that are easily quantifiable.

So perhaps the solution — seriously — would be for some third-party review company to rate each new phone on the Stupid S#!% Index. They test the phone under normal usage, and each time they run into an idiotic bug like the calendar application not knowing what day it is, they file it under Stupid S#!%, and after some fixed period of phone usage they count up all the problems and rate the phone under the Stupid S#!% Index. For greater precision, you could compile multiple scores from different users for each phone and take the average. Now you have a quantifiable rating that can be used to compare one phone to another, and could incentivize manufacturers to do more testing on their phones in order to get a better Stupid S#!% Index score.

The message that Apple keeps pushing about the iPhone, after all, is essentially that it would get a good Stupid S#!% Index rating. In his keynote address at the 2011 Apple Worldwide Developers Conference, Steve Jobs repeated the words "It just works" like a mantra — unlike, presumably, everyone else's stuff. iPhones don't score well on price, openness, or compatibility with other companies' products (I always have to tell people that my car charger is not an iPhone charger, it's a literally-every-other-smartphone-in-the-entire-world charger) — but all of that scarcely matters to some people as long as It Just Works.

Well, I couldn't tell you. I can't test an iPhone under normal usage because I'm too addicted to the Stratosphere's slide-out keyboard, which enables me to type much faster than a touchscreen but which only comes on a few Android and Windows phones, and not on any version of the iPhone. Maybe I'll try one more time to make the switch to a touchscreen while my Stratosphere is in the shop.

Yes, these most First-World of First World Problems — especially the bugs specific to the Stratosphere — only apply to a small fraction of the population. But it should be a lesson for anyone who thinks the "free market" would prevent this sort of thing from happening.

Meanwhile, every time I hear an ad talking about how "thin" some new phone is going to be, I just want to say to the phone the same thing that I want to tell all the anorexic girls in nightclubs: You're already thin enough. So stop worrying about being thin, and just try to work on not being so f@#$ing stupid.

Bennett Haselton writes "How did a $400-billion company ship millions of units of a phone with a calendar app that displays the wrong date, a texting app that can't reply to group texts, a screen capture function that doesn't work, and a phone app that won't let me use the keypad unless the speakerphone is on? The answer, perhaps, suggests deeper questions about why market forces fix certain problems but not others, and what to do about it." Read on for the rest of Bennett's thoughts.

I've been using either a Samsung Stratosphere or a Samsung Stratosphere 2 from September 2012 to the present. Where to begin?

• If you open the calendar application on the Stratosphere 2, it usually highlights tomorrow's date as "Today," and lists tomorrow's calendar entries as your list of things to do "Today." Here is a picture of my phone's screen taken on June 2, with the calendar app displaying "Today, Mon, Jun 3 2013" — despite the phone knowing the correct time is 9:22 PM on June 2.

  Strangely, in the morning the calendar app would display the correct day as "Today," but would switch to the wrong day some time in the afternoon, and eventually I decided that the calendar app was probably using Coordinated Universal Time to decide what "Today" was, which is 9 hours ahead of Pacific Standard Time.

• You'll notice that these images are not screen captures, but photos taken with another phone. This is because some time between the Stratosphere 1 and 2, the screen capture function broke — every support site says you're supposed to be able to take a screen cap on a Stratosphere by pressing the Home and Power buttons at the same time, and that works on the 1, but not on the 2.

• If someone else sends a text to multiple recipients including you, the Stratosphere gives you no indication that it's a group text, and there's no way for you to see the other recipients or reply to the whole group. (I had a lot of awkward "What, you were asking everybody, not just me?" moments before I realized what was going on.) Other users have been complaining about this for months, and it apparently affects more Android phones than just the Stratosphere.

• The built-in camera refuses to take a picture if the battery is low — it just says "Warning: low battery" and exits. Yes, I know they think they're doing it for my own good since the camera is a battery hog, but a few times I've wanted to take a picture where it was well worth using up a half a percent of my remaining battery life or however much it would have taken, but the phone wouldn't let me. That should be the user's decision, dammit.

• When I was in Canada last week, if I tried sending a text message longer than 160 characters, the phone would tell me that the message sent, but it would actually fail silently and never get delivered. I'm not sure whether to blame Verizon, Android, or Samsung for this one (or just, you know), but in the end someone has to take responsibility for the product, and the phone telling you that a message was sent when it actually got lost, is a complete fail. If it doesn't work, fine, give me an error message, but never tell the user a message got sent successfully if it didn't.

• During a phone call, the on-screen keypad doesn't work unless the phone is on speakerphone. If the speakerphone is off, the screen goes dark after about 1 second of inactivity, making it impossible to enter an account number or anything else. I can avoid this bug by turning on a speakerphone (which is how I know it's a software bug, not a problem with the touchscreen), but this is a pain if I'm in a public place and don't want to annoy everyone around me who would have to listen to all the voice prompts. (The phone's software seems to be following a rule like: "If the speakerphone is not on, then when the phone moves away from the user's face, assume the user is not actively using the phone and let the screen go dark" — where the bug is that it doesn't make an exception and keep the screen on if the user is actively pressing keys on the keypad.)

At first, these and many, many other bugs produce a state of mind that transcends annoyance to reach a kind of genuine curiosity, where you're asking "How did this happen?" not rhetorically, but because you actually want to know. But eventually the surprise wears off, and you're just left with bugs that are disproportionately aggravating because they obviously would have been caught during even the most basic UI testing. They're aggravating to me not because of how much they get in the way (you eventually get used to them), but because the existence of those bugs conveys a certain lazy attitude towards finding and fixing bugs at all.

I realize this is not a logical reaction. The aggravation you feel towards a bug should depend on how much the bug actually interferes with the user experience, not on how easily the manufacturer should have found it. Rationally speaking, the biggest problem with the phone right now (and the reason I'm having to mail to back to the manufacturer for a replacement) is that the charging port spontaneously broke, so that unless the micro USB charger is plugged in exactly right, the phone can't charge (even if you get it right and form a connection successfully, the connection breaks if you move the phone half an inch). Needless to say, that's exasperating — but it's hard for me to get mad at Samsung over that, because it's not an easy defect to catch at the manufacturing stage. On the other hand, if the calendar app displays the wrong day, I would say that someone should be fired over that except that probably nobody was assigned to do that testing in the first place.

I also posted questions about each of these problems on AndroidForums.com and AndroidCentral.com (those links show all questions recently posted from my username on each site), which have so far received hundreds of "views" but no replies. I mention this because some people think that if you do run into problems like these, all you have to do is post a question and The Community will help you out with a workaround. Nope.

Also, lest you think you can do away with these bugs by downloading third-party replacements for all of these apps, I spent part of an afternoon downloading different texting apps to see if any of them would fix even part of the problems I had with the built-in one. None of them worked much better, although several of them displayed pop-up ads over every third incoming text message, and most of them did not play nicely with each other, giving me no way to disable them so that their notifications would double and triple up on top of each other for every received text. So I gave up. Even if I thought I might eventually find a better app for texting, I didn't have time to test multiple replacements for every built-in default app that didn't work.

Farhad Manjoo has a column up at Slate arguing that the reason many Android phones suck is that they're laden down with adware attempting to extract more personal information and money from the user. I'm sure that's part of the problem, but I can't see how the manufacturer is making any money off of the bugs I ran into; they were just being lazy.

The problem, I think, is that phone manufacturers know that phone reviewers (and users, when they're choosing between models in the store) will focus on easily quanitifiable attributes, such as size, weight, battery life, and the number of megapixels in the camera. The number of aggravating bugs in the user interface is not something that is easy to compare across phones (and in any case would not be printed on the box). Thus market forces simply don't favor the development of a hassle-free interface, because in most cases the phone manufacturers wouldn't be rewarded for it.

And — I don't consider this too much of a stretch — this is where it connects with larger issues for me, because I've been arguing for years that the free market will usually fail to fix certain types of problems, often in the context of threats to free speech and civil liberties, especially if the user lacks information they need to compare multiple options. A major argument in favor of Net Neutrality is that the typical user wouldn't realize it if their ISP were throttling access to certain sites; they would just think that the remote site was responding slowly. Since that information would be hidden from the user, "the marketplace" won't solve the problem on its own. Similarly, every time I say that my Circumventor mailing list keeps getting blocked as "spam" by Yahoo, Hotmail, Gmail, or AOL (despite being 100% verified-opt-in, natch), someone tells me that if the free market is blocking my emails as unwanted, it must be because the users don't want them. That the free market might make a mistake (in this case, because users don't have full information about what is getting blocked as spam), doesn't occur to them. I think the belief in the infallibility of the free market, is one of the most widespread fallacies of our era — people who would never make the mistake of confusing correlation with causation, have no problem thinking that if a product or service gets blocked by a third-party intermediary, it must be because the end user didn't want it.

And so when I'm staring at my Stratosphere's calendar telling me that tomorrow is actually today, it brings out my aggravation not just towards Samsung, Google, and Verizon, but towards all the people I've heard over the years claiming that the marketplace will automatically reward good products and punish bad ones. If there weren't so many people who believed that, maybe we could have collectively put more effort into rating phones according to their usability, knowing that the "invisible hand" of the marketplace was not likely to solve those problems on its own, and maybe these bugs would have gotten fixed. Instead, the "marketplace" focuses disproportionately on attributes like dimensions, weight, and processor speed that are easily quantifiable.

So perhaps the solution — seriously — would be for some third-party review company to rate each new phone on the Stupid S#!% Index. They test the phone under normal usage, and each time they run into an idiotic bug like the calendar application not knowing what day it is, they file it under Stupid S#!%, and after some fixed period of phone usage they count up all the problems and rate the phone under the Stupid S#!% Index. For greater precision, you could compile multiple scores from different users for each phone and take the average. Now you have a quantifiable rating that can be used to compare one phone to another, and could incentivize manufacturers to do more testing on their phones in order to get a better Stupid S#!% Index score.

The message that Apple keeps pushing about the iPhone, after all, is essentially that it would get a good Stupid S#!% Index rating. In his keynote address at the 2011 Apple Worldwide Developers Conference, Steve Jobs repeated the words "It just works" like a mantra — unlike, presumably, everyone else's stuff. iPhones don't score well on price, openness, or compatibility with other companies' products (I always have to tell people that my car charger is not an iPhone charger, it's a literally-every-other-smartphone-in-the-entire-world charger) — but all of that scarcely matters to some people as long as It Just Works.

Well, I couldn't tell you. I can't test an iPhone under normal usage because I'm too addicted to the Stratosphere's slide-out keyboard, which enables me to type much faster than a touchscreen but which only comes on a few Android and Windows phones, and not on any version of the iPhone. Maybe I'll try one more time to make the switch to a touchscreen while my Stratosphere is in the shop.

Yes, these most First-World of First World Problems — especially the bugs specific to the Stratosphere — only apply to a small fraction of the population. But it should be a lesson for anyone who thinks the "free market" would prevent this sort of thing from happening.

Meanwhile, every time I hear an ad talking about how "thin" some new phone is going to be, I just want to say to the phone the same thing that I want to tell all the anorexic girls in nightclubs: You're already thin enough. So stop worrying about being thin, and just try to work on not being so f@#$ing stupid.

Bennett Haselton writes "How did a $400-billion company ship millions of units of a phone with a calendar app that displays the wrong date, a texting app that can't reply to group texts, a screen capture function that doesn't work, and a phone app that won't let me use the keypad unless the speakerphone is on? The answer, perhaps, suggests deeper questions about why market forces fix certain problems but not others, and what to do about it." Read on for the rest of Bennett's thoughts.

I've been using either a Samsung Stratosphere or a Samsung Stratosphere 2 from September 2012 to the present. Where to begin?

• If you open the calendar application on the Stratosphere 2, it usually highlights tomorrow's date as "Today," and lists tomorrow's calendar entries as your list of things to do "Today." Here is a picture of my phone's screen taken on June 2, with the calendar app displaying "Today, Mon, Jun 3 2013" — despite the phone knowing the correct time is 9:22 PM on June 2.

  Strangely, in the morning the calendar app would display the correct day as "Today," but would switch to the wrong day some time in the afternoon, and eventually I decided that the calendar app was probably using Coordinated Universal Time to decide what "Today" was, which is 9 hours ahead of Pacific Standard Time.

• You'll notice that these images are not screen captures, but photos taken with another phone. This is because some time between the Stratosphere 1 and 2, the screen capture function broke — every support site says you're supposed to be able to take a screen cap on a Stratosphere by pressing the Home and Power buttons at the same time, and that works on the 1, but not on the 2.

• If someone else sends a text to multiple recipients including you, the Stratosphere gives you no indication that it's a group text, and there's no way for you to see the other recipients or reply to the whole group. (I had a lot of awkward "What, you were asking everybody, not just me?" moments before I realized what was going on.) Other users have been complaining about this for months, and it apparently affects more Android phones than just the Stratosphere.

• The built-in camera refuses to take a picture if the battery is low — it just says "Warning: low battery" and exits. Yes, I know they think they're doing it for my own good since the camera is a battery hog, but a few times I've wanted to take a picture where it was well worth using up a half a percent of my remaining battery life or however much it would have taken, but the phone wouldn't let me. That should be the user's decision, dammit.

• When I was in Canada last week, if I tried sending a text message longer than 160 characters, the phone would tell me that the message sent, but it would actually fail silently and never get delivered. I'm not sure whether to blame Verizon, Android, or Samsung for this one (or just, you know), but in the end someone has to take responsibility for the product, and the phone telling you that a message was sent when it actually got lost, is a complete fail. If it doesn't work, fine, give me an error message, but never tell the user a message got sent successfully if it didn't.

• During a phone call, the on-screen keypad doesn't work unless the phone is on speakerphone. If the speakerphone is off, the screen goes dark after about 1 second of inactivity, making it impossible to enter an account number or anything else. I can avoid this bug by turning on a speakerphone (which is how I know it's a software bug, not a problem with the touchscreen), but this is a pain if I'm in a public place and don't want to annoy everyone around me who would have to listen to all the voice prompts. (The phone's software seems to be following a rule like: "If the speakerphone is not on, then when the phone moves away from the user's face, assume the user is not actively using the phone and let the screen go dark" — where the bug is that it doesn't make an exception and keep the screen on if the user is actively pressing keys on the keypad.)

At first, these and many, many other bugs produce a state of mind that transcends annoyance to reach a kind of genuine curiosity, where you're asking "How did this happen?" not rhetorically, but because you actually want to know. But eventually the surprise wears off, and you're just left with bugs that are disproportionately aggravating because they obviously would have been caught during even the most basic UI testing. They're aggravating to me not because of how much they get in the way (you eventually get used to them), but because the existence of those bugs conveys a certain lazy attitude towards finding and fixing bugs at all.

I realize this is not a logical reaction. The aggravation you feel towards a bug should depend on how much the bug actually interferes with the user experience, not on how easily the manufacturer should have found it. Rationally speaking, the biggest problem with the phone right now (and the reason I'm having to mail to back to the manufacturer for a replacement) is that the charging port spontaneously broke, so that unless the micro USB charger is plugged in exactly right, the phone can't charge (even if you get it right and form a connection successfully, the connection breaks if you move the phone half an inch). Needless to say, that's exasperating — but it's hard for me to get mad at Samsung over that, because it's not an easy defect to catch at the manufacturing stage. On the other hand, if the calendar app displays the wrong day, I would say that someone should be fired over that except that probably nobody was assigned to do that testing in the first place.

I also posted questions about each of these problems on AndroidForums.com and AndroidCentral.com (those links show all questions recently posted from my username on each site), which have so far received hundreds of "views" but no replies. I mention this because some people think that if you do run into problems like these, all you have to do is post a question and The Community will help you out with a workaround. Nope.

Also, lest you think you can do away with these bugs by downloading third-party replacements for all of these apps, I spent part of an afternoon downloading different texting apps to see if any of them would fix even part of the problems I had with the built-in one. None of them worked much better, although several of them displayed pop-up ads over every third incoming text message, and most of them did not play nicely with each other, giving me no way to disable them so that their notifications would double and triple up on top of each other for every received text. So I gave up. Even if I thought I might eventually find a better app for texting, I didn't have time to test multiple replacements for every built-in default app that didn't work.

Farhad Manjoo has a column up at Slate arguing that the reason many Android phones suck is that they're laden down with adware attempting to extract more personal information and money from the user. I'm sure that's part of the problem, but I can't see how the manufacturer is making any money off of the bugs I ran into; they were just being lazy.

The problem, I think, is that phone manufacturers know that phone reviewers (and users, when they're choosing between models in the store) will focus on easily quanitifiable attributes, such as size, weight, battery life, and the number of megapixels in the camera. The number of aggravating bugs in the user interface is not something that is easy to compare across phones (and in any case would not be printed on the box). Thus market forces simply don't favor the development of a hassle-free interface, because in most cases the phone manufacturers wouldn't be rewarded for it.

And — I don't consider this too much of a stretch — this is where it connects with larger issues for me, because I've been arguing for years that the free market will usually fail to fix certain types of problems, often in the context of threats to free speech and civil liberties, especially if the user lacks information they need to compare multiple options. A major argument in favor of Net Neutrality is that the typical user wouldn't realize it if their ISP were throttling access to certain sites; they would just think that the remote site was responding slowly. Since that information would be hidden from the user, "the marketplace" won't solve the problem on its own. Similarly, every time I say that my Circumventor mailing list keeps getting blocked as "spam" by Yahoo, Hotmail, Gmail, or AOL (despite being 100% verified-opt-in, natch), someone tells me that if the free market is blocking my emails as unwanted, it must be because the users don't want them. That the free market might make a mistake (in this case, because users don't have full information about what is getting blocked as spam), doesn't occur to them. I think the belief in the infallibility of the free market, is one of the most widespread fallacies of our era — people who would never make the mistake of confusing correlation with causation, have no problem thinking that if a product or service gets blocked by a third-party intermediary, it must be because the end user didn't want it.

And so when I'm staring at my Stratosphere's calendar telling me that tomorrow is actually today, it brings out my aggravation not just towards Samsung, Google, and Verizon, but towards all the people I've heard over the years claiming that the marketplace will automatically reward good products and punish bad ones. If there weren't so many people who believed that, maybe we could have collectively put more effort into rating phones according to their usability, knowing that the "invisible hand" of the marketplace was not likely to solve those problems on its own, and maybe these bugs would have gotten fixed. Instead, the "marketplace" focuses disproportionately on attributes like dimensions, weight, and processor speed that are easily quantifiable.

So perhaps the solution — seriously — would be for some third-party review company to rate each new phone on the Stupid S#!% Index. They test the phone under normal usage, and each time they run into an idiotic bug like the calendar application not knowing what day it is, they file it under Stupid S#!%, and after some fixed period of phone usage they count up all the problems and rate the phone under the Stupid S#!% Index. For greater precision, you could compile multiple scores from different users for each phone and take the average. Now you have a quantifiable rating that can be used to compare one phone to another, and could incentivize manufacturers to do more testing on their phones in order to get a better Stupid S#!% Index score.

The message that Apple keeps pushing about the iPhone, after all, is essentially that it would get a good Stupid S#!% Index rating. In his keynote address at the 2011 Apple Worldwide Developers Conference, Steve Jobs repeated the words "It just works" like a mantra — unlike, presumably, everyone else's stuff. iPhones don't score well on price, openness, or compatibility with other companies' products (I always have to tell people that my car charger is not an iPhone charger, it's a literally-every-other-smartphone-in-the-entire-world charger) — but all of that scarcely matters to some people as long as It Just Works.

Well, I couldn't tell you. I can't test an iPhone under normal usage because I'm too addicted to the Stratosphere's slide-out keyboard, which enables me to type much faster than a touchscreen but which only comes on a few Android and Windows phones, and not on any version of the iPhone. Maybe I'll try one more time to make the switch to a touchscreen while my Stratosphere is in the shop.

Yes, these most First-World of First World Problems — especially the bugs specific to the Stratosphere — only apply to a small fraction of the population. But it should be a lesson for anyone who thinks the "free market" would prevent this sort of thing from happening.

Meanwhile, every time I hear an ad talking about how "thin" some new phone is going to be, I just want to say to the phone the same thing that I want to tell all the anorexic girls in nightclubs: You're already thin enough. So stop worrying about being thin, and just try to work on not being so f@#$ing stupid.

Frequent contributor Bennett Haselton writes: "Hotmail and Yahoo Mail are apparently sharing a secret blacklist of domain names such that any mention of these domains will cause a message to be bounced back to the sender as spam. I found out about this because — surprise! — some of my new proxy site domains ended up on the blacklist. Hotmail and Yahoo are stonewalling, but here's what I've dug up so far — and why you should care." Read on for much more on how Bennett figured out what's going on, and why it's a hard problem to solve.

On December 7th I sent out a normal batch of emails to the Circumventor mailing list, where I send out new proxy sites for getting around Internet filters. I registered seven new domains and sent each domain to one seventh of the list; the list contains about 420,000 addresses, so each one went to about 60,000 people. (Each new site is only sent to a random subset of the list, so that a blocking company can't just subscribe one address to the list and block all new sites as soon as they're mailed out.)

The list is also comprised of 100%-verified-opt-in addresses, meaning that a new subscriber has to reply to a confirmation message in order to be added to the list. That's considered the gold standard for responsible mailing, but major email providers keep finding new ways to block the emails as "spam," which sometimes provide interesting insights into how the filters work behind the scenes.

After the last mailing, for example, all of my newly registered domains got disabled by the registrar because two of the domains had been incorrectly blacklisted by the Spamhaus Domain Block List. It took two days to discover the problem and then several hours to trace the problem to Spamhaus, although once I found Spamhaus's automated form I was able to get the domains un-blacklisted immediately. So the registrar re-enabled the domains a few hours later, although the traffic to the domains never returned to its previous levels. Spamhaus, meanwhile, continues to claim the DBL is a "zero false-positive" list, and has yet to acknowledge the error or contact me to help get to the bottom of how it happened. Well, they know how to reach me.

At least this time around, my domains didn't get disabled. Instead, the messages rolled out for a few hours with no problem (replies from users indicated that at least some hotmail.com and yahoo.com users were receiving them), until bounces abruptly started coming in from hotmail.com and yahoo.com addresses saying:

----- Transcript of session follows -----
... while talking to mta5.am0.yahoodns.net.:
>>> DATA
<<< 550 Message Contains SPAM Content
554 5.0.0 Service unavailable

After pummeling my address with bounce messages (to the point where my own Gmail account started bouncing because it was getting hammered with so many bounce messages from Hotmail and Yahoo), when the dust finally settled, I tried reproducing the error by sending test messages from my server's IP address to a test Hotmail account. It turns out that out of the seven different URLs that I had been mailing to our users, four of the domains in those URLs would generate a "550 Message Contains SPAM Content" error when sent from my IP to a Hotmail address, and the other three did not. The message didn't have to contain the banned domain in the From: address; the message would get blocked if it even mentioned the domain anywhere in the message body. (This only happened when sending from my own IP address at peacefire.org. It didn't happen if I tried sending a message from my Gmail account to a Hotmail address, even if the message contained one of the four banned domain names, so the issue probably won't reproduce if you try sending a test message yourself.)

But interestingly, Yahoo Mail started bouncing my messages at about the same time — out of the seven domain names, the same four domain names were being bounced by Yahoo Mail as by Hotmail, also with the error "550 Message Contains SPAM Content." That's far too unlikely to be a coincidence, so it looks as if Hotmail and Yahoo Mail are using a common secret blacklist of domain names that cause a message to be blocked as spam. (As it happens, the other three domains were also being bounced by Yahoo Mail with the error "Message Contains SUSPECT Content" — as opposed to "SPAM Content" — while those three domains were not blocked by Hotmail at all. That of course is aggravating, but the real clue lies in the fact that both Yahoo Mail and Hotmail were giving "SPAM Content" errors to the exact same subset of domains.)

I don't want to publish the list of all seven domain names here, so as not to make it too easy for censorware companies to block them all, but one of the four blacklisted domains was 'golflanding.com.' (All of the new domains I register are nonsensical two-word combinations, since those are the only .com domains that are likely to be (1) still available and (2) easy to remember.) As soon as it seemed like Hotmail and Yahoo Mail were working off of a common blacklist, I checked to see if Spamhaus had screwed up again and listed our domains, but none of the seven domains were on Spamhaus's lists.

I looked up golflanding.com on the blacklistalert.org service, which checks against all major spam blacklists, but no hits were listed there either (except for on some defunct services which haven't been updated in years).

So if Hotmail and Yahoo Mail are both using the domain blacklist, perhaps it's a list compiled by one company and then licensed to the other, or perhaps it's a third-party list not widely known to the public. (Hotmail uses their own SmartScreen filter, but I've found nothing online about Yahoo using it as well.) It's conceivable that one or more of the domains might have gotten blacklisted as a result of Hotmail or Yahoo users clicking their "This is spam" button. However, Hotmail allows newsletter publishers to view data about what percent of their messages to Hotmail users are being flagged by users as "spam," and when I looked up the stats for our IP, they showed a "complaint rate" of less than 0.1% (usually the rest of people hitting 'Junk Mail' to unsubscribe from the list). Assuming that the complaint rates are similar for Yahoo Mail, it's unlikely that the domains got blacklisted as a result of user complaints, unless the blacklist trigger has a ridiculously low complaint threshold.

Neither the Hotmail postmaster site nor the Yahoo postmaster site mention anything about a list of domain names that could cause a message to be blocked for mentioning the domains in the message body. Yahoo Mail does provide a support form for newsletter publishers to send inquiries about why their mail is being blocked; I submitted that on Saturday and started a thread with email "support," although so far their response has just been to copy and paste articles from the Postmaster site, with tips like "Send email only to those that want it." Each time, I reply saying, No, this is not the problem, the problem is that the domains in the messages are getting incorrectly blacklisted, and each time, support cheerfully sends me another article. If I'm not literally talking to a bot, I might as well be.

I opened a similar ticket with Hotmail, and they sent me a form letter saying that the emails were being blocked because of SmartScreen, and that as a matter of policy, they would refuse to fix any errors being made by the SmartScreen filter. Waiting to see if I get a reply from a human next.

So why should you care? Well, for one thing, if you care about users in China and Iran being able to receive proxies to get around their Internet blockers, right now Hotmail and Yahoo are thwarting these proxies more effectively than those countries' own censors are. Yes, these are real people who really do write back to me after a mailing goes out, telling me about how they were able to use the proxies to receive banned political information, and sometimes how long the proxy lasted before the censors blocked it. This week, they had to do without.

But more importantly, this is an example of a general problem: That there are certain types of issues, like blocking of legitimate mail by spam filters, where the "free market" does not deliver the best experience to consumers, and the costs get passed on to everybody. Sometimes the problems could be solved with some effort, but the effort does not get made, because people believe that the free market will solve the problem, or that it already has.

In theory, if consumers have enough information about different companies and their services, the companies can compete to provide the best product to users. The problem is that if one type of information is systematically hidden from users — in this case, the fact that their mail provider is blocking mails from reaching them — then the "theory" falls apart. Since spam getting into your inbox is a visible problem, but missed email messages are an invisible problem, Hotmail's incentive is not to give the user the best experience, but rather to err on the side of blocking legitimate messages — even if the user might prefer to get slightly more spam, than to miss one important email that they were waiting for.

This means we're not just talking about a few messages getting caught in filters, which could happen even in an efficient marketplace. We're talking about a permanent equilibrium where the user gets a sub-par experience by default — a trade-off that causes them to miss more messages than they want to — and senders have to pay the cost of overcoming the marketplace inefficiencies. (Which means if the sender is a business you buy from or a charity you support, the costs get passed on to you.)

Pretty much the entire financial cost of sending email, is attributable to the failure of the "free market" to motivate email providers to deliver non-spam emails into their user's inboxes. If a company or organization uses an email list hosting company like AWeber or Constant Contact to email their users, they pay a fee of about $1 per month for every 100 users on their list (which would run me about $4,000 per month). That fee doesn't go towards bandwidth — even a 1-million-subscriber list, emailed once a month, would use less than 3 GB per month of bandwidth, which is what GeoCities was was giving away for free 10 years ago. What you're paying for is the fact that AWeber and Constant Contact have friends in the right places at Hotmail, Yahoo, and Gmail, so if your mails are getting blocked, they know the people to call to fix the problem. If you run your own list instead of paying a hosting fee to AWeber or Constant Contact, you'll end up paying other costs indirectly, through loss of income when your messages don't reach recipients, or in time and money spent trying to fix the issue. (I have to take this option anyway, since I send different URLs to different random subsets of my list, which is not supported by AWeber or Constant Contact.)

On the other hand, if the market actually "worked" — if email providers did reliably deliver non-spam messages to their users — a company or charity could run their own list for virtually zero cost, and would be able to keep all of that money. (I incur no up-front fees for running my own list; all of the costs are the time spent trying to get Yahoo, Gmail, and Hotmail to stop blocking it.) So every time you donate to a charity or buy from an online retailer, a little bit of that money goes towards the cost of that organization having to fight past marketplace failures in order to get their email to you.

I don't think there's an easy algorithmic solution, like crowdsourcing Facebook complaints or using random-sample voting on Digg. Generally, I just think we need more awareness of the fact that, under certain conditions (including those surrounding email deliverability), the "free market" is virtually guaranteed to arrive at a non-optimal solution. One manifestation of that awareness would be if Hotmail, Yahoo Mail, and Gmail created public points of contact where legitimate email publishers could find out why their emails were blocked, and had real humans responding to the messages and fixing the problems. By default, the imperfect information in the marketplace leads toward an equilibrium that errs on the side of blocking too much legitimate email, so anything that pushes the equilibrium back towards more legitimate messages getting delivered will improve the experience for users and lower costs for senders.

Besides, there's a more basic ethical issue here. If you're Hotmail and you tell your users that you're providing them with "email accounts," then those users expect those accounts to work — including having the ability to receive mails from mailing lists that they've signed up for. Helping legitimate emails get through to users is not just a matter of addressing a marketplace inefficiency, it's a matter of honesty.

Larry Lessig's book "Code is Law" describes how default choices built into the architecture of the Internet and other environments — the "code" — can steer our behavior in ways that we might not choose otherwise. I'm making essentially the same point in saying that some problems are not fixed by market forces, because people are not aware of the problem at all. I think the evidence and the reasoning are straightforward in this case, but it's hard to convince people who have adopted it as an axiom that whatever the free market arrives at, must be the solution. My favorite single sentence in Lessig's book was, "Put your Ayn Rand away." I could imagine the years of pushing against dogmatic fanaticism that led him to write that sentence, and I knew how he felt.

Frequent contributor Bennett Haselton writes: "Hotmail and Yahoo Mail are apparently sharing a secret blacklist of domain names such that any mention of these domains will cause a message to be bounced back to the sender as spam. I found out about this because — surprise! — some of my new proxy site domains ended up on the blacklist. Hotmail and Yahoo are stonewalling, but here's what I've dug up so far — and why you should care." Read on for much more on how Bennett figured out what's going on, and why it's a hard problem to solve.

On December 7th I sent out a normal batch of emails to the Circumventor mailing list, where I send out new proxy sites for getting around Internet filters. I registered seven new domains and sent each domain to one seventh of the list; the list contains about 420,000 addresses, so each one went to about 60,000 people. (Each new site is only sent to a random subset of the list, so that a blocking company can't just subscribe one address to the list and block all new sites as soon as they're mailed out.)

The list is also comprised of 100%-verified-opt-in addresses, meaning that a new subscriber has to reply to a confirmation message in order to be added to the list. That's considered the gold standard for responsible mailing, but major email providers keep finding new ways to block the emails as "spam," which sometimes provide interesting insights into how the filters work behind the scenes.

After the last mailing, for example, all of my newly registered domains got disabled by the registrar because two of the domains had been incorrectly blacklisted by the Spamhaus Domain Block List. It took two days to discover the problem and then several hours to trace the problem to Spamhaus, although once I found Spamhaus's automated form I was able to get the domains un-blacklisted immediately. So the registrar re-enabled the domains a few hours later, although the traffic to the domains never returned to its previous levels. Spamhaus, meanwhile, continues to claim the DBL is a "zero false-positive" list, and has yet to acknowledge the error or contact me to help get to the bottom of how it happened. Well, they know how to reach me.

At least this time around, my domains didn't get disabled. Instead, the messages rolled out for a few hours with no problem (replies from users indicated that at least some hotmail.com and yahoo.com users were receiving them), until bounces abruptly started coming in from hotmail.com and yahoo.com addresses saying:

----- Transcript of session follows -----
... while talking to mta5.am0.yahoodns.net.:
>>> DATA
<<< 550 Message Contains SPAM Content
554 5.0.0 Service unavailable

After pummeling my address with bounce messages (to the point where my own Gmail account started bouncing because it was getting hammered with so many bounce messages from Hotmail and Yahoo), when the dust finally settled, I tried reproducing the error by sending test messages from my server's IP address to a test Hotmail account. It turns out that out of the seven different URLs that I had been mailing to our users, four of the domains in those URLs would generate a "550 Message Contains SPAM Content" error when sent from my IP to a Hotmail address, and the other three did not. The message didn't have to contain the banned domain in the From: address; the message would get blocked if it even mentioned the domain anywhere in the message body. (This only happened when sending from my own IP address at peacefire.org. It didn't happen if I tried sending a message from my Gmail account to a Hotmail address, even if the message contained one of the four banned domain names, so the issue probably won't reproduce if you try sending a test message yourself.)

But interestingly, Yahoo Mail started bouncing my messages at about the same time — out of the seven domain names, the same four domain names were being bounced by Yahoo Mail as by Hotmail, also with the error "550 Message Contains SPAM Content." That's far too unlikely to be a coincidence, so it looks as if Hotmail and Yahoo Mail are using a common secret blacklist of domain names that cause a message to be blocked as spam. (As it happens, the other three domains were also being bounced by Yahoo Mail with the error "Message Contains SUSPECT Content" — as opposed to "SPAM Content" — while those three domains were not blocked by Hotmail at all. That of course is aggravating, but the real clue lies in the fact that both Yahoo Mail and Hotmail were giving "SPAM Content" errors to the exact same subset of domains.)

I don't want to publish the list of all seven domain names here, so as not to make it too easy for censorware companies to block them all, but one of the four blacklisted domains was 'golflanding.com.' (All of the new domains I register are nonsensical two-word combinations, since those are the only .com domains that are likely to be (1) still available and (2) easy to remember.) As soon as it seemed like Hotmail and Yahoo Mail were working off of a common blacklist, I checked to see if Spamhaus had screwed up again and listed our domains, but none of the seven domains were on Spamhaus's lists.

I looked up golflanding.com on the blacklistalert.org service, which checks against all major spam blacklists, but no hits were listed there either (except for on some defunct services which haven't been updated in years).

So if Hotmail and Yahoo Mail are both using the domain blacklist, perhaps it's a list compiled by one company and then licensed to the other, or perhaps it's a third-party list not widely known to the public. (Hotmail uses their own SmartScreen filter, but I've found nothing online about Yahoo using it as well.) It's conceivable that one or more of the domains might have gotten blacklisted as a result of Hotmail or Yahoo users clicking their "This is spam" button. However, Hotmail allows newsletter publishers to view data about what percent of their messages to Hotmail users are being flagged by users as "spam," and when I looked up the stats for our IP, they showed a "complaint rate" of less than 0.1% (usually the rest of people hitting 'Junk Mail' to unsubscribe from the list). Assuming that the complaint rates are similar for Yahoo Mail, it's unlikely that the domains got blacklisted as a result of user complaints, unless the blacklist trigger has a ridiculously low complaint threshold.

Neither the Hotmail postmaster site nor the Yahoo postmaster site mention anything about a list of domain names that could cause a message to be blocked for mentioning the domains in the message body. Yahoo Mail does provide a support form for newsletter publishers to send inquiries about why their mail is being blocked; I submitted that on Saturday and started a thread with email "support," although so far their response has just been to copy and paste articles from the Postmaster site, with tips like "Send email only to those that want it." Each time, I reply saying, No, this is not the problem, the problem is that the domains in the messages are getting incorrectly blacklisted, and each time, support cheerfully sends me another article. If I'm not literally talking to a bot, I might as well be.

I opened a similar ticket with Hotmail, and they sent me a form letter saying that the emails were being blocked because of SmartScreen, and that as a matter of policy, they would refuse to fix any errors being made by the SmartScreen filter. Waiting to see if I get a reply from a human next.

So why should you care? Well, for one thing, if you care about users in China and Iran being able to receive proxies to get around their Internet blockers, right now Hotmail and Yahoo are thwarting these proxies more effectively than those countries' own censors are. Yes, these are real people who really do write back to me after a mailing goes out, telling me about how they were able to use the proxies to receive banned political information, and sometimes how long the proxy lasted before the censors blocked it. This week, they had to do without.

But more importantly, this is an example of a general problem: That there are certain types of issues, like blocking of legitimate mail by spam filters, where the "free market" does not deliver the best experience to consumers, and the costs get passed on to everybody. Sometimes the problems could be solved with some effort, but the effort does not get made, because people believe that the free market will solve the problem, or that it already has.

In theory, if consumers have enough information about different companies and their services, the companies can compete to provide the best product to users. The problem is that if one type of information is systematically hidden from users — in this case, the fact that their mail provider is blocking mails from reaching them — then the "theory" falls apart. Since spam getting into your inbox is a visible problem, but missed email messages are an invisible problem, Hotmail's incentive is not to give the user the best experience, but rather to err on the side of blocking legitimate messages — even if the user might prefer to get slightly more spam, than to miss one important email that they were waiting for.

This means we're not just talking about a few messages getting caught in filters, which could happen even in an efficient marketplace. We're talking about a permanent equilibrium where the user gets a sub-par experience by default — a trade-off that causes them to miss more messages than they want to — and senders have to pay the cost of overcoming the marketplace inefficiencies. (Which means if the sender is a business you buy from or a charity you support, the costs get passed on to you.)

Pretty much the entire financial cost of sending email, is attributable to the failure of the "free market" to motivate email providers to deliver non-spam emails into their user's inboxes. If a company or organization uses an email list hosting company like AWeber or Constant Contact to email their users, they pay a fee of about $1 per month for every 100 users on their list (which would run me about $4,000 per month). That fee doesn't go towards bandwidth — even a 1-million-subscriber list, emailed once a month, would use less than 3 GB per month of bandwidth, which is what GeoCities was was giving away for free 10 years ago. What you're paying for is the fact that AWeber and Constant Contact have friends in the right places at Hotmail, Yahoo, and Gmail, so if your mails are getting blocked, they know the people to call to fix the problem. If you run your own list instead of paying a hosting fee to AWeber or Constant Contact, you'll end up paying other costs indirectly, through loss of income when your messages don't reach recipients, or in time and money spent trying to fix the issue. (I have to take this option anyway, since I send different URLs to different random subsets of my list, which is not supported by AWeber or Constant Contact.)

On the other hand, if the market actually "worked" — if email providers did reliably deliver non-spam messages to their users — a company or charity could run their own list for virtually zero cost, and would be able to keep all of that money. (I incur no up-front fees for running my own list; all of the costs are the time spent trying to get Yahoo, Gmail, and Hotmail to stop blocking it.) So every time you donate to a charity or buy from an online retailer, a little bit of that money goes towards the cost of that organization having to fight past marketplace failures in order to get their email to you.

I don't think there's an easy algorithmic solution, like crowdsourcing Facebook complaints or using random-sample voting on Digg. Generally, I just think we need more awareness of the fact that, under certain conditions (including those surrounding email deliverability), the "free market" is virtually guaranteed to arrive at a non-optimal solution. One manifestation of that awareness would be if Hotmail, Yahoo Mail, and Gmail created public points of contact where legitimate email publishers could find out why their emails were blocked, and had real humans responding to the messages and fixing the problems. By default, the imperfect information in the marketplace leads toward an equilibrium that errs on the side of blocking too much legitimate email, so anything that pushes the equilibrium back towards more legitimate messages getting delivered will improve the experience for users and lower costs for senders.

Besides, there's a more basic ethical issue here. If you're Hotmail and you tell your users that you're providing them with "email accounts," then those users expect those accounts to work — including having the ability to receive mails from mailing lists that they've signed up for. Helping legitimate emails get through to users is not just a matter of addressing a marketplace inefficiency, it's a matter of honesty.

Larry Lessig's book "Code is Law" describes how default choices built into the architecture of the Internet and other environments — the "code" — can steer our behavior in ways that we might not choose otherwise. I'm making essentially the same point in saying that some problems are not fixed by market forces, because people are not aware of the problem at all. I think the evidence and the reasoning are straightforward in this case, but it's hard to convince people who have adopted it as an axiom that whatever the free market arrives at, must be the solution. My favorite single sentence in Lessig's book was, "Put your Ayn Rand away." I could imagine the years of pushing against dogmatic fanaticism that led him to write that sentence, and I knew how he felt.

Frequent contributor Bennett Haselton writes: "Hotmail and Yahoo Mail are apparently sharing a secret blacklist of domain names such that any mention of these domains will cause a message to be bounced back to the sender as spam. I found out about this because — surprise! — some of my new proxy site domains ended up on the blacklist. Hotmail and Yahoo are stonewalling, but here's what I've dug up so far — and why you should care." Read on for much more on how Bennett figured out what's going on, and why it's a hard problem to solve.

On December 7th I sent out a normal batch of emails to the Circumventor mailing list, where I send out new proxy sites for getting around Internet filters. I registered seven new domains and sent each domain to one seventh of the list; the list contains about 420,000 addresses, so each one went to about 60,000 people. (Each new site is only sent to a random subset of the list, so that a blocking company can't just subscribe one address to the list and block all new sites as soon as they're mailed out.)

The list is also comprised of 100%-verified-opt-in addresses, meaning that a new subscriber has to reply to a confirmation message in order to be added to the list. That's considered the gold standard for responsible mailing, but major email providers keep finding new ways to block the emails as "spam," which sometimes provide interesting insights into how the filters work behind the scenes.

After the last mailing, for example, all of my newly registered domains got disabled by the registrar because two of the domains had been incorrectly blacklisted by the Spamhaus Domain Block List. It took two days to discover the problem and then several hours to trace the problem to Spamhaus, although once I found Spamhaus's automated form I was able to get the domains un-blacklisted immediately. So the registrar re-enabled the domains a few hours later, although the traffic to the domains never returned to its previous levels. Spamhaus, meanwhile, continues to claim the DBL is a "zero false-positive" list, and has yet to acknowledge the error or contact me to help get to the bottom of how it happened. Well, they know how to reach me.

At least this time around, my domains didn't get disabled. Instead, the messages rolled out for a few hours with no problem (replies from users indicated that at least some hotmail.com and yahoo.com users were receiving them), until bounces abruptly started coming in from hotmail.com and yahoo.com addresses saying:

----- Transcript of session follows -----
... while talking to mta5.am0.yahoodns.net.:
>>> DATA
<<< 550 Message Contains SPAM Content
554 5.0.0 Service unavailable

After pummeling my address with bounce messages (to the point where my own Gmail account started bouncing because it was getting hammered with so many bounce messages from Hotmail and Yahoo), when the dust finally settled, I tried reproducing the error by sending test messages from my server's IP address to a test Hotmail account. It turns out that out of the seven different URLs that I had been mailing to our users, four of the domains in those URLs would generate a "550 Message Contains SPAM Content" error when sent from my IP to a Hotmail address, and the other three did not. The message didn't have to contain the banned domain in the From: address; the message would get blocked if it even mentioned the domain anywhere in the message body. (This only happened when sending from my own IP address at peacefire.org. It didn't happen if I tried sending a message from my Gmail account to a Hotmail address, even if the message contained one of the four banned domain names, so the issue probably won't reproduce if you try sending a test message yourself.)

But interestingly, Yahoo Mail started bouncing my messages at about the same time — out of the seven domain names, the same four domain names were being bounced by Yahoo Mail as by Hotmail, also with the error "550 Message Contains SPAM Content." That's far too unlikely to be a coincidence, so it looks as if Hotmail and Yahoo Mail are using a common secret blacklist of domain names that cause a message to be blocked as spam. (As it happens, the other three domains were also being bounced by Yahoo Mail with the error "Message Contains SUSPECT Content" — as opposed to "SPAM Content" — while those three domains were not blocked by Hotmail at all. That of course is aggravating, but the real clue lies in the fact that both Yahoo Mail and Hotmail were giving "SPAM Content" errors to the exact same subset of domains.)

I don't want to publish the list of all seven domain names here, so as not to make it too easy for censorware companies to block them all, but one of the four blacklisted domains was 'golflanding.com.' (All of the new domains I register are nonsensical two-word combinations, since those are the only .com domains that are likely to be (1) still available and (2) easy to remember.) As soon as it seemed like Hotmail and Yahoo Mail were working off of a common blacklist, I checked to see if Spamhaus had screwed up again and listed our domains, but none of the seven domains were on Spamhaus's lists.

I looked up golflanding.com on the blacklistalert.org service, which checks against all major spam blacklists, but no hits were listed there either (except for on some defunct services which haven't been updated in years).

So if Hotmail and Yahoo Mail are both using the domain blacklist, perhaps it's a list compiled by one company and then licensed to the other, or perhaps it's a third-party list not widely known to the public. (Hotmail uses their own SmartScreen filter, but I've found nothing online about Yahoo using it as well.) It's conceivable that one or more of the domains might have gotten blacklisted as a result of Hotmail or Yahoo users clicking their "This is spam" button. However, Hotmail allows newsletter publishers to view data about what percent of their messages to Hotmail users are being flagged by users as "spam," and when I looked up the stats for our IP, they showed a "complaint rate" of less than 0.1% (usually the rest of people hitting 'Junk Mail' to unsubscribe from the list). Assuming that the complaint rates are similar for Yahoo Mail, it's unlikely that the domains got blacklisted as a result of user complaints, unless the blacklist trigger has a ridiculously low complaint threshold.

Neither the Hotmail postmaster site nor the Yahoo postmaster site mention anything about a list of domain names that could cause a message to be blocked for mentioning the domains in the message body. Yahoo Mail does provide a support form for newsletter publishers to send inquiries about why their mail is being blocked; I submitted that on Saturday and started a thread with email "support," although so far their response has just been to copy and paste articles from the Postmaster site, with tips like "Send email only to those that want it." Each time, I reply saying, No, this is not the problem, the problem is that the domains in the messages are getting incorrectly blacklisted, and each time, support cheerfully sends me another article. If I'm not literally talking to a bot, I might as well be.

I opened a similar ticket with Hotmail, and they sent me a form letter saying that the emails were being blocked because of SmartScreen, and that as a matter of policy, they would refuse to fix any errors being made by the SmartScreen filter. Waiting to see if I get a reply from a human next.

So why should you care? Well, for one thing, if you care about users in China and Iran being able to receive proxies to get around their Internet blockers, right now Hotmail and Yahoo are thwarting these proxies more effectively than those countries' own censors are. Yes, these are real people who really do write back to me after a mailing goes out, telling me about how they were able to use the proxies to receive banned political information, and sometimes how long the proxy lasted before the censors blocked it. This week, they had to do without.

But more importantly, this is an example of a general problem: That there are certain types of issues, like blocking of legitimate mail by spam filters, where the "free market" does not deliver the best experience to consumers, and the costs get passed on to everybody. Sometimes the problems could be solved with some effort, but the effort does not get made, because people believe that the free market will solve the problem, or that it already has.

In theory, if consumers have enough information about different companies and their services, the companies can compete to provide the best product to users. The problem is that if one type of information is systematically hidden from users — in this case, the fact that their mail provider is blocking mails from reaching them — then the "theory" falls apart. Since spam getting into your inbox is a visible problem, but missed email messages are an invisible problem, Hotmail's incentive is not to give the user the best experience, but rather to err on the side of blocking legitimate messages — even if the user might prefer to get slightly more spam, than to miss one important email that they were waiting for.

This means we're not just talking about a few messages getting caught in filters, which could happen even in an efficient marketplace. We're talking about a permanent equilibrium where the user gets a sub-par experience by default — a trade-off that causes them to miss more messages than they want to — and senders have to pay the cost of overcoming the marketplace inefficiencies. (Which means if the sender is a business you buy from or a charity you support, the costs get passed on to you.)

Pretty much the entire financial cost of sending email, is attributable to the failure of the "free market" to motivate email providers to deliver non-spam emails into their user's inboxes. If a company or organization uses an email list hosting company like AWeber or Constant Contact to email their users, they pay a fee of about $1 per month for every 100 users on their list (which would run me about $4,000 per month). That fee doesn't go towards bandwidth — even a 1-million-subscriber list, emailed once a month, would use less than 3 GB per month of bandwidth, which is what GeoCities was was giving away for free 10 years ago. What you're paying for is the fact that AWeber and Constant Contact have friends in the right places at Hotmail, Yahoo, and Gmail, so if your mails are getting blocked, they know the people to call to fix the problem. If you run your own list instead of paying a hosting fee to AWeber or Constant Contact, you'll end up paying other costs indirectly, through loss of income when your messages don't reach recipients, or in time and money spent trying to fix the issue. (I have to take this option anyway, since I send different URLs to different random subsets of my list, which is not supported by AWeber or Constant Contact.)

On the other hand, if the market actually "worked" — if email providers did reliably deliver non-spam messages to their users — a company or charity could run their own list for virtually zero cost, and would be able to keep all of that money. (I incur no up-front fees for running my own list; all of the costs are the time spent trying to get Yahoo, Gmail, and Hotmail to stop blocking it.) So every time you donate to a charity or buy from an online retailer, a little bit of that money goes towards the cost of that organization having to fight past marketplace failures in order to get their email to you.

I don't think there's an easy algorithmic solution, like crowdsourcing Facebook complaints or using random-sample voting on Digg. Generally, I just think we need more awareness of the fact that, under certain conditions (including those surrounding email deliverability), the "free market" is virtually guaranteed to arrive at a non-optimal solution. One manifestation of that awareness would be if Hotmail, Yahoo Mail, and Gmail created public points of contact where legitimate email publishers could find out why their emails were blocked, and had real humans responding to the messages and fixing the problems. By default, the imperfect information in the marketplace leads toward an equilibrium that errs on the side of blocking too much legitimate email, so anything that pushes the equilibrium back towards more legitimate messages getting delivered will improve the experience for users and lower costs for senders.

Besides, there's a more basic ethical issue here. If you're Hotmail and you tell your users that you're providing them with "email accounts," then those users expect those accounts to work — including having the ability to receive mails from mailing lists that they've signed up for. Helping legitimate emails get through to users is not just a matter of addressing a marketplace inefficiency, it's a matter of honesty.

Larry Lessig's book "Code is Law" describes how default choices built into the architecture of the Internet and other environments — the "code" — can steer our behavior in ways that we might not choose otherwise. I'm making essentially the same point in saying that some problems are not fixed by market forces, because people are not aware of the problem at all. I think the evidence and the reasoning are straightforward in this case, but it's hard to convince people who have adopted it as an axiom that whatever the free market arrives at, must be the solution. My favorite single sentence in Lessig's book was, "Put your Ayn Rand away." I could imagine the years of pushing against dogmatic fanaticism that led him to write that sentence, and I knew how he felt.

Frequent contributor Bennett Haselton writes this week with his favorite novelty science gift items for 2012. Levitation engines, puzzles, optical illusions brought to life, and all of the tips and tricks he's found for getting the products to work correctly. Decorative, whimsical, and not too expensive — except for the items that have earned it by being pretty amazing. Read on for the details, and be sure to mention other good possibilities (Just 14 shopping days left until Christmas) in the comments below.

You already know how to find all the latest iPad or iPhone accessories, or how to find all the licensed merchandise if your BFF is a fan of some specific franchise. The items in this list are things that most people wouldn't even think to look for, but that I thought seemed interesting once I found out that they existed.

I'm more of a science geek than a gadget geek, so this list is built around optical illusions, whimsy, conversation pieces that demonstrate some scientific principle, and a reasonable budget. (The "Swinging Sticks Kinetic Energy Sculpture" from ThinkGeek is a work of art, but at $225, the price is apparently set to extract as much as possible from all the people who have to have one after seeing it in Iron Man 2.)

Also, unless otherwise noted, I've actually tried everything listed here and verified that it actually works; there were some items that I really wanted to make work, but couldn't. The Double Sand Sculpture, for example, looks great (especially in colors other than that ugly orange), but in all three models that American Science & Surplus sent me — the original plus the two free replacements — air bubbles formed in the hourglasses after a few days, which blocked the sand grains from flowing through the apertures. I could also never get Educational Innovations' Color Changing Nail Polish to change color, even under a UV light. And I loved the look of the Tornado Fountain from Fascinations.com, but no matter how I calibrated it, the drain at the bottom made a squirting and scraping sound like the last dregs of water draining from a bathtub, which pretty much killed its potential as a "tranquil" conversation piece. (As far as I can tell, any tabletop water fountain that costs less than $100 is either too noisy or doesn't work, but I haven't given up looking.) Of course, if you can get any of those things to work, more power to you.

For most of these items I've included the tips and tricks that I've accumulated for getting the full effect out of the product, tips that in some cases would have saved me a lot of hassle if I'd known them when the product first arrived. So you get the full benefit of my impulsive early-September Christmas shopping.

Neither I nor Slashdot make any profit from these links (except some items are from ThinkGeek, which is a corporate cousin of Slashdot for a few more weeks — but I didn't know that when I was making this list, and besides, it's not like you can put together a geek gift guide without including some stuff from ThinkGeek anyway).

Here are some of the things I've found that look as cool in person as they do in their catalog photos, and actually work:

- - - - - - - - - - - - - - - - - - - -

Levitron Revolution
Made by Fascinations.com, $100 from Innovatoys.com.

I bought my first "Levitron"-branded product out of a Sky Mall catalog 15 years ago, assuming the picture of the levitating spinning top had to be a doctored photo, and half-set on proving that the product was a sham. I had spent enough time trying to levitate repelling magnets as a kid to conclude that it "couldn't be done," but I held out the faintest glimmer of hope that this might be the holy grail that I'd given up chasing about 10 years earlier. When the box arrived, I spent all evening and a sleepness night trying to get it working (the original product had to be calibrated and balanced very carefully, and you could waste a lot of time trying to make it work if the weights or alignments were slightly off), until just as the sun was coming up, I got the spinning top to levitate above the magnetic base for about four seconds before falling, and felt as if it had all been worth it. And the Levitron product line has come a long way since then, so you probably won't have to journey to the edge of your sanity to get this latest one working.

The Levitron Revolution is a levitation device which uses a base containing four computer-controlled magnets, and a magnetic disc that levitates about 1/2-inch above the base and can support a weight of up to 1 pound placed on top of it while continuing to levitate. It still takes a bit of practice to learn how to position the disc above the base to start the levitation, but the payoff is worth the effort. You can even rotate the base sideways and upside down, and the levitating disc will stay in the same position relative to the base while you turn it.

I used mine to levitate a crystal specimen that I got from a specialty gem store, which set me back about another $30, but I liked the way it glittered in the lights from the magnetic base. The rock was labeled "quartz / pyrite / sphalerite" at the store, and if you're looking for a similar rock to go with the Levitron Revolution, it looks like you can find one on Google Shopping for less than I paid for mine.

You can also use the Levitron Revolution for homemade illusions like levitating a cupcake in mid-air. (A Hostess dessert cup has a circular cavity on top to hold strawberries and whipped cream; turn it upside down and it fits perfectly over the Levitron disc. The book underneath the cupcake in the video was hollowed out to contain the magnetic base.)

Innovatoys sells several other Levitron products made by Fascinations, which all fall into two categories: those based on the classic Levitron design (which include any product showing the yellow-necked Levitron spinning top), and those based on the newer Levitron Revolution technology (everything else). I also have a Levitron CherryWood which is part of the "classic" lineup. The pros and cons of the two series are:

• The classic Levitron levitates the spinning top a full two inches above the base, which is much more visually impressive than the 1/2-inch that the magnetic disc floats above the base of the Levitron Revolution.
• The classic Levitron has to be hand-spun, however, and takes even more practice to operate than the Levitron Revolution.
• The classic Levitron has to be perfectly level for the top to float (the base comes with three adjustable legs to help you level it perfectly); the Levitron Revolution can be tilted and rotated, and the magnetic disc will continue to float in position relative to the base.
• The classic Levitron levitates in a very delicate equilibrium, with just the slightest touch being enough to push the floating top out out of balance and make it fall, so it can't be used to support other objects (and the top is spinning so fast that you wouldn't be able to see anything attached to it anyway). The Levitron Revolution floating disc can be touched and objects can be placed on top of it without pushing it out of equilibrium.
• The classic Levitron requires no power to operate, but because the top has to keep spinning at a high rate for the gyroscopic force to keep it from flipping over, after about two minutes the air friction will slow down the top enough that it falls. The Levitron Revolution will levitate forever as long as the DC power supply is connected.

The Levitron invention itself has something of a contentious history (recounted here and here). Evidently, the physicist Ray Harrigan had patented a similar device a few years earlier and showed it to Bill Hones, who later got his own patent for a similar device and called it the "Levitron," but Hones was advised by his own lawyer that his own invention was sufficiently different from Harrigan's that he could market it without infringing Harrigan's patent or giving him credit or royalties. Apparently Harrigan was so disgusted and distrustful of his own lawyer that he never took the issue to court, so we'll never know what a judge would have thought. (The only issue which was ever litigated in court was over a former re-seller's use of the trademark "Levitron" — but that seems more straightforward, since the company that made up the word and trademarked it, owns it, completely separate from the merits of the invention that bears the name.) Some physicists have mixed feelings about the Levitron because of this, but it was apparently Harrigan's choice not to pursue the issue. (Besides, the new Levitron Revolution design uses nothing of Harrigan's idea, so some might feel that it's less "tainted".)

For cheaper levitation that takes no skill to operate, you can get the Diamagnetic Levitation Kit from Educational Innovations or search for pyrolitic graphite levitation on eBay — much less visually impressive though, with the graphite sheet levitating only 1 millimeter above the magnets.

Or for a more expensive conversation piece, the Levitron Lamp ($450 from InnovaToys or $400 from WorldToHome) levitates an entire lampshade above the base. I haven't tried that one out though.

- - - - - - - - - - - - - - - - - - - -

Levitating Picture Frames
Heart-shaped frame $25 from ZOpid; rectangular frame $70 from Hammacher Schlemmer.

Computer-controlled levitation operating on a similar principle to the Levitron Revolution products. The $25 ZOpid picture frame is currently hanging out in Amazon limbo with a solitary 1-star review from a customer whose model broke after 4 months. But I think they look fine, and I'm giving two of them as gifts and crossing my fingers that I'm not that unlucky. With both the ZOpid and the Hammacher Schlemmer frames, unfortunately, there's apparently no way to switch off the LED lights (short of turning off the whole model).

Protip: You can prepare these as gifts by using photos downloaded from a friend's Facebook profile, but Facebook reduces the quality of uploaded photos, so that if you print them out, the pixellation will be noticeable up close. If you want the photos to look the best, you need to print them from high-res originals.

- - - - - - - - - - - - - - - - - - - -

Hanayama Japanese Pocket Puzzles
$13 from ThinkGeek and other vendors; some puzzles available for slightly less on eBay.

Some disassembly puzzles are complete fails, either because there are so many separately moving pieces that you can't manipulate the puzzles in your hands at all (e.g. Yin and Yang"), or the moving parts are hidden from view so you can only "solve" them by pure guesswork (e.g. the "Bolted Closed" puzzle). The Hanayama pocket puzzles actually get it right — you can see all the pieces and move them comfortably in your hands, so solving them is just a matter of figuring out the right sequence of moves.

These are basically grown-up versions of the twisted nail puzzles you might have grown up with (and which you could also get, of course, as much cheaper stocking stuffers). But the Hanayama ones look good as shelf knick-knacks as well.

Hanayama pocket puzzles come with no solution included, but you can download a solution by going to this page and submitting your email address to request a download link.

- - - - - - - - - - - - - - - - - - - -

LED Jellyfish Mood Lamp
$35 from ThinkGeek and other vendors; no cheaper alternatives on eBay

Works more or less as shown in the video, with one caveat: In both the first model that I tried, and the free replacement ThinkGeek sent me when I reported the problem, the transitions between the different colors were much more abrupt and jarring than the smooth "color fade" shown in the video. (For some reason, some color LEDs would switch from completely on to completely off at the same time that other LEDs would switch on.) Unfortunately this small problem completely breaks the "reverie" effect of staring at the jellyfish floating around in the water, so I just set mine to a single color without using the transition effect.

Protip: You have to use real distilled water like the instructions tell you. I tried to make it work with regular tap water, and bubbles kept forming around the jellyfish and causing them to float to the surface. Fill it with distilled water and the jellyfish should sink beneath the surface without too much trouble.

Note, Fascinations has come out with a similar product, again sold on Innovatoys.com; I haven't tried that one, so it might be better (might actually get the color transition right), or it might not. Discovery Kids also makes a similar product which I haven't seen and which has been pulling pretty bad reviews on Amazon.

- - - - - - - - - - - - - - - - - - - -

Vino Vault and Cryptex Puzzle Pod
$30 and $22 from 4Thought Products LLC

The Puzzle Pod is a gift container that can only be opened by arranging the 5 rings to spell out a 5-letter password. It arrives pre-configured with the keyword "GRAPE"; once opened, you can re-configure the Pod with a new 5-letter secret word, seal a gift inside, and gift it to a recipient who has to find the secret word to open the puzzle and retrieve the gift. (It's re-usable, and you can set a different 5-letter "password" every time.) The Vino Vault is a larger version of the Puzzle Pod that can hold a bottle of wine.

I've only sampled the Puzzle Pod, so I can just vouch for the fact that it works exactly as described and doesn't get stuck or break easily. When you line up the letters of the secret word correctly, it actually slides smoothly open like it's supposed to.

- - - - - - - - - - - - - - - - - - - -

Ambiguous Vase
$33 from Grand Illusions Ltd (ships from the UK)

This is a real-life version of the Rubin vase optical illusion. For years, Grand Illusions sold only a ceramic version for about $400 (plus another $200 to ship to the U.S.), but in November 2012 they released the $33 plastic version. It can also be used as a real vase (as long as you don't mind the barrier running down the center that divides the two halves).

- - - - - - - - - - - - - - - - - - - -

Steam Powered Top
$14 from Grand Illusions (ships from the UK)

The world's simplest steam engine, made from a tube of copper pushed through a piece of cork, as shown in the demo video. Wikipedia explains the principle here — when the water in the copper tube is heated by the candle flame and boils, it expands and pushes out the ends of the tubes (driving the spinning motion). When the water contracts again, in sucks in water through the ends of the tubes — but the sucking motion pulls in water from all directions (while the expulsion of water pushes in only one direction), so the suction doesn't counteract the propulsion, and the top continues spinning.

Now, the original version is from Germany (and comes with detailed German instructions); the version that I got came with a sheet of English instructions that weren't as detailed. The instructions say to push the copper tube through the cork platform and "bend the tube at a 90-degree angle"; however if you just try bending the tube, it will probably crimp and create a hole, making it useless. To bend the tube so that it curves gradually, place your thumb on the cork next to where the tube protrudes, and use the fingers of your other hand to gently push the tube so that curves around your thumb. (This is spelled out in the original German instructions.)

Also, the instructions say to fill the copper tube by holding it under running tap water. This didn't work at all for me, since the tube is only about 2mm wide and the surface tension of water makes it hard to "push" it into a tube that small. Fortunately, a straw from a grocery-store juicebox fits perfectly over the other end of the copper tube, so if you submerge the other end in water, you can suck on the straw to fill the tube that way. (It's just copper after all, not lead.)

Finally, if you leave the cork floating in water too long, it eventually gets waterlogged and sinks, and as far as I can tell it's very hard to dry it out and bring it back to its original buoyancy. The workarounds for this are: (1) to increase the buoyancy, first put another tea light directly into your bowl of water so that it floats, and then lower the top into the water on top of that tea light, which will then help keep the top afloat; and (2) don't leave the top floating in water when not in use.

- - - - - - - - - - - - - - - - - - - -

"Flying F*CK" Remote-Control Helicopter
$20 from ThinkGeek

Again with the ThinkGeek swag; I swear I didn't know.

This is pretty self-explanatory, except I've tried two of them and the product doesn't seem to work too well as an actual remote-control helicopter; one of them couldn't hover in place (its two modes were "shooting up at the ceiling" or "falling"), and with the other, the R/C didn't seem to work through furniture. But that's probably OK since the whole point of this gift is in the giving and not the having.

In my case, I hid it behind a friend's chair at his birthday party, then at the appropriate time gave a speech ending with, "And so I thought, what do I give my friend to mark this occasion? What do I give? After much thought, I decided, this is what I give:..." There followed a dramatic pause where I pressed the "up" control on the remote, and nothing happened, whereupon I muttered, appropriately enough, "Fuck", then wandered over behind my friend's chair, repeated the setup line, pressed the remote button, at which point the copter shot up, banged into a chair and fell to the ground, whereupon for my third attempt I just picked it up and held it on the palm of my hand, pressed the remote, and the copter took flight and finally delivered the punch line, and all was good. If I'm there when he re-gifts it (since we both agreed that was the point of a gift like this), I hope it works better for him.

- - - - - - - - - - - - - - - - - - - -

Falling Sand Sculptures
$13 for the smaller 'Sandscape'; $80 for the larger 'Deep Sea Round'; both available from Educational Innovations

These both make good decorations and shelf widgets. The sand in the Sandscape always falls in more or less the same pattern, since it's pre-determined by the gaps in the shelves holding the sand; the Deep Sea Round is more interesting since the pattern is determined by the placement of air bubbles and varies every time.

Pro tip: water evaporates from both of these, so eventually the water level will drop and the volume of air will increase, getting in the way of the sand flow. The 'Deep Sea Round' comes with a syringe that you can use to draw out air and inject more water into the aperture on the side. The cheaper 'Sandscape' doesn't come with a syringe, but it has a hole in the side where you can use a syringe to inject more water, if you buy the syringe separately.

- - - - - - - - - - - - - - - - - - - -

Galileo Thermometer
$17 for a wood-mounted model from Office Playground; cheaper ones available without wood mounting

Just your basic elegant conversation piece demonstrating the principle that the density of a liquid changes with temperature. Pro tip: If you get the wood mounted one, before emailing the seller to complain that it's not working because all the spheres are bunched together at the wrong end, make sure it's not upside-down. (I realized, before I hit Send, that the felt-covered end goes on the bottom.)

- - - - - - - - - - - - - - - - - - - -

All of the remaining items on this list do exactly what they say they do, with no need for any special instructions not included by the manufacturer, so I'm just going to list them:

Glass Water Faucet — $50 from Uncommon Goods — a nice double optical illusion (faucet suspended in space, and glass-as-water).

Slicked Grandfather Clock — $30-$60 depending on who's selling it.

Tin Can Robot Kit — about $15 from various vendors — my stepdad and I assembled one using one of his beloved Hansen's soda cans.

Mini metal DIY sculptures — the Metal Works sculptures from Innovatoys ($7-$12) take some time to assemble but they come out looking pretty much like the pictures and make good shelf decorations. These Mikro sculptures ($10 and up, also available from Grand Illusions if you're filling your shopping cart there) are a bit easier to assemble since you just have to bend some shapes out from the metal sheet that they're carved from.

Ulexite "Television Stones" — $10 from Educational Innovations — a naturally occuring rock containing thousands of parallel fiber optic strands. Give it as a gift together with a square of patterned fabric so you can see the eerie effect when you place the rock against the fabric and the pattern "magically" appears on the opposite side of the rock.

And finally, if you need a last-minute gag gift for someone, browse through the gum and hand sanitizers from BlueQ.com — they're not geek-themed, but at $5.49 for the hand sanitizers and $1.39 for the gum, you can afford to stock up so you'll have a reserve of gag gifts suited for a variety of different people's tastes (except, of course, good taste).

And those are my favorites for gift-giving season 2012. You can send me suggestions for any items in this category that I've missed; I'll be back for Valentine's Day.

- - - - - - - - - - - - - - - -
Remember, if you have a feature idea, we'd love to hear it.

Slashdot regular contributor Bennett Haselton writes "My last article on prediction markets contained an erroneous assumption, one whose implications are far-reaching enough that they deserve their own article. (And if you read to the end, I'm offering $100 to be split between the readers who submit the best alternative solution or the best counter-argument to the points made here.)" Read below for the rest of Bennett's thoughts.

In my last article, I wrote:

There could be rules and safeguards to prevent abuses of the system (rules that could be imposed by U.S. law, even if they're not enforced by overseas betting markets), such as not allowing individuals to bet more than $500. (This is already enforced by the Iowa Electronic Markets.) That's small enough to stop individual bettors from trying to manipulate the market through enormous wagers (although they might find ways to do that anyway). It's also small enough that it wouldn't be worth it for any one individual to try and influence a political outcome just to win a bet. You could try to enlist your friends to help you place a collective $10,000 bet on a single outcome, but the more people you rope into your coalition, the greater the chances of someone (a) turning you in for violating the betting laws, or (b) taking the $500 you lent them, and then refusing to pay it back if they win their portion of the wager.

There's an error here, but one subtle enough that even all the commenters (with no shortage of the usual snark) missed it. To begin with, consider what happens if two different betting markets are taking bets at different odds for the same event.

Suppose CappedEx, a futures exchange that limits each user to betting $500, is publishing 4:1 odds of an Obama victory. If you bet $40 that Obama will win and he wins, you get paid $10 (from other users on the exchange), but if he loses, you pay out $40. Meanwhile FreedomEx, an exchange that has no betting limit for any user, is publishing 6:1 odds for Obama winning. Bet $60 on Obama, and you get $10 if he wins, but pay $60 if he loses. On both markets, of course you can bet in the other direction as well.

What do you conclude from this? That the un-capped FreedomEx probably has more accurate odds, and that as James Surowiecki (author of The Wisdom of Crowds) said, betting limits "make [the markets] less accurate" and "real money is what makes it work"? Or that CappedEx, with its safeguards against manipulation, is more reliable, and FreedomEx is being manipulated by someone trying to change the reported odds of their favored candidate winning? Or that there is simply some random fluctuation in the odds as reported by various markets, so they'll naturally diverge at times?

The correct answer is: you should stop wasting time "concluding" things, and get online as soon as possible and make bets in both markets, because if they're allowing bets to be placed at different odds, you can guarantee yourself a profit.

Make a $50 bet in CappedEx on Obama to win (4:1 odds), and a $10 bet in FreedomEx on Romney to win (1:6 odds). If Obama wins, you win $12.50 in the CappedEx market and lose your $10 in FreedomEx, for a $2.50 profit. If Romney wins, you lose $50 in the CappedEx market but win $60 in FreedomEx, for a $10 profit. With a little algebra, you can show that any time the two markets allow you to place bets at different odds ratios, you can make a guaranteed profit by picking a ratio somewhere in the middle (in this case, the two ratios were 1:4 and 1:6, so we picked 1:5) and making separate bets in the two markets in opposite directions, for amounts in that ratio. (A commenter on the Marginal Revolution blog describes exactly how he made an almost risk-free profit through this kind of "pure arbitrage play". He said it was "almost" risk free because of other factors like currency conversion fluctuations.)

Now, any time a good is trading for a lower price in market A than it is in market B, and the costs of shifting the good between the two markets is negligible, traders will start to buy the good in market A and re-sell it for a profit in market B (the traditional definition of "arbitrage"). This increases demand in market A (driving the price up) and increases supply in market B (driving the price down) until the price difference disappears. In the same way, any time two prediction markets have different "market odds" for the same event, as arbitrage players lock in guaranteed profits by placing opposite bets in the two markets, the market odds in the two markets will converge toward each other until the gap is negligible. This is true even if one of the markets has a cap on what people can invest or how much they can stake on any particular outcome.

For Intrade, there couldn't be a worse time for someone to be pointing this out, but it seems logically inescapable: As long as there is a prediction market anywhere in the world that allows unlimited wagering on a particular outcome, all other prediction markets (whether they are capped or not) can be manipulated indirectly, by playing a large wager in the non-capped market. I was wrong to say that you would have to "enlist your friends" to place bets in the capped market, building a large coalition of market-manipulators (and hoping that none of them would rat you out for using them to circumvent wager-limiting rules). By placing a large wager in the non-capped market, and shifting the market odds there so that they're different from the odds in the capped market, you can indirectly "enlist" all the users in the capped market, to place arbitrage bets and make a guaranteed profit. When this happens, the odds in both the capped market and the non-capped market will shift, as the gap between them narrows -- which means you have manipulated the market odds in the capped market, without ever going near it yourself.

In this case, why have caps on the amounts wagered in prediction markets at all? (The Iowa Electronic Markets have a maximum investment balance of $500, and a 2008 paper, "The Promise of Prediction Markets, authored by several prominent economists, advocated the creation of prediction markets with a maximum investment of $2,000.) Presumably the cap is not to prevent unlucky investors from losing their life's savings, since the law already allows multiple ways to do that, by betting on volatile stocks in the stock market. And it won't stop market manipulation, if the capped market can still be manipulated by using another non-capped market as a proxy. Robin Hanson, Professor of Economics at George Mason University and one of the co-authors of the 2008 paper, candidly told me that the cap was just a matter of selling the idea: "As a practical matter, many people's comfort with such markets increases when there is a cap, so they are more likely to accept the proposal with a cap. So it makes one seem more reasonable to propose a cap, if one can get most of the benefits one wanted from such a system that has a cap, relative to one without it."

So is there a solution to the manipulation problem? Actually, is it even a problem? Robin Hansen and Ryan Oprea wrote another paper arguing that manipulators can improve prediction markets, by subsidizing the existing players in the markets and rewarding them for paying attention. (If a "manipulative" bet causes a sudden shift in the reported odds, opportunistic investors can place bets essentially wagering that the odds will return back to their previous level.) Economist Alex Tabarrok makes the same point here. This opportunism also means that the market shift caused by a manipulative bet usually corrects itself within a few minutes.

Presumably, if more people start to take prediction markets seriously, the incentives to manipulate them would increase. As Tabarrok adds, "prediction markets have truly arrived when people think they are worth manipulating". At the same time though, as more people start to take prediction markets seriously, presumably they'll attract more actual users, and since the amount of money required to shift the market is proportional to the amount already invested by everyone else, this means it will require larger amounts of money to shift the market odds to the same degree.

So these economists all seem to think that prediction market manipulation is a good thing, and that the prediction markets themselves are an even better good thing even when they can be manipulated, but now I'm not so sure. If people do think that market odds are worth manipulating, presumably the point is to create a self-fulfilling prophecy: People think that Romney's chances have gone up, so they become more incentivized to support him and vote for him, and soon his chances actually have gone up (although possibly not to the full extent of the boost in the manipulated market odds, so the manipulator may still lose money). If you can boost Romney's market odds even for a few minutes just by spending a few tens of thousands of dollars, how much would it cost to sustain the higher odds for several hours -- and what if those hours were at a crucial time in the election or in the news reporting cycle?

What if, contrary to my last assumption, people start to take prediction markets seriously enough to be influenced by them, but the prediction markets don't see a proportionate influx of actual investors and money -- so the cost of manipulating them remains about the same? IF prediction markets gain more influence in people's actual voting decisions, BUT those markets don't see an influx of new users, AND an election is close enough that the market odds could make a difference depending on when they're reported, AND someone spends enough to sustain the manipulated odds during crucial periods during the election... Well, that's a lot of assumptions you have to grant, but individually they're quite plausible -- and if all of them hold true, you could change the outcome of a presidential election for just a few million dollars spent on the prediction markets.

And in fact, if you successfully swung the election, you'd actually win all the wagers you had just placed -- which means that now rich manipulators can throw their election to their preferred candidate, and make a bundle. It also means that all those opportunists who usually act to "correct" the market odds deviations, by taking your free money when you start placing manipulative bets, could realize that your bets might actually change the outcome, and would decline to take your money -- which in turn means it would be even cheaper for manipulators to change the outcome, creating a self-reinforcing cycle. If smart bettors see that once a behemoth starts the market moving, the behemoth will probably win, they'll just get out of its way and clear an easier path.

The same kind of trick wouldn't normally work on the stock market -- if you're wealthy enough that you can increase the share price of a stock by buying enough of it to shift the market, then when you try to reap your profits by unloading the stock, the price will drift back down as you're selling it off. (Or if your purchases do manage to create a self-fulfilling prophecy -- your infusion of cash into the company enables them to realize their plans and become a genuine success -- well, then you're just a successful angel investor, more power to you.) But a presidential election prediction market would be analogous to a stock where if you can keep the price artificially inflated for several crucial hours on November 6th, 2012, then the price becomes permanently locked in at that point and you can sell it off for a profit, regardless of the value of the underlying company.

So, according to my own reasoning, this idea that I was so gung-ho about a few days ago, could not only be used to create a type of financial instrument that rewards manipulation more perversely than anything we've ever seen, but could also let a Saudi prince pick the next leader of the free world on a bet.

I'm not sure if there's a solution. I'm not a libertarian so I was never in favor of prediction markets as a matter of "personal liberty"; I was in favor of them because they're useful insofar as they can harness the wisdom of crowds to convey important information. But if they can be manipulated to influence real-world events, is it worth it?

In keeping with the theory that money does motivate people to think harder about such things, I'm once again offering $100 to be split between the readers who email me the best-argued solutions to this problem -- or the best counter-argument to any point I've made here. Put "prediction markets" in the subject line. If your submission wins a portion of the award, you can either claim the money for yourself, or to be donated to a preferred charity in your name. (I reserve the right to pay out less than the allotted $100 if there aren't enough worthy submissions, but that didn't happen last time.) Any sufficiently valuable comments are eligible even if they're not strictly counter-arguments or suggested alternatives, and I'll post a follow-up article summarizing what people send in. You can't make as much off of me, as you could have made by taking some market manipulator's intentionally losing bet on Intrade that Romney was going to win the election, but at least it's legal.

Slashdot regular contributor Bennett Haselton writes "My last article on prediction markets contained an erroneous assumption, one whose implications are far-reaching enough that they deserve their own article. (And if you read to the end, I'm offering $100 to be split between the readers who submit the best alternative solution or the best counter-argument to the points made here.)" Read below for the rest of Bennett's thoughts.

In my last article, I wrote:

There could be rules and safeguards to prevent abuses of the system (rules that could be imposed by U.S. law, even if they're not enforced by overseas betting markets), such as not allowing individuals to bet more than $500. (This is already enforced by the Iowa Electronic Markets.) That's small enough to stop individual bettors from trying to manipulate the market through enormous wagers (although they might find ways to do that anyway). It's also small enough that it wouldn't be worth it for any one individual to try and influence a political outcome just to win a bet. You could try to enlist your friends to help you place a collective $10,000 bet on a single outcome, but the more people you rope into your coalition, the greater the chances of someone (a) turning you in for violating the betting laws, or (b) taking the $500 you lent them, and then refusing to pay it back if they win their portion of the wager.

There's an error here, but one subtle enough that even all the commenters (with no shortage of the usual snark) missed it. To begin with, consider what happens if two different betting markets are taking bets at different odds for the same event.

Suppose CappedEx, a futures exchange that limits each user to betting $500, is publishing 4:1 odds of an Obama victory. If you bet $40 that Obama will win and he wins, you get paid $10 (from other users on the exchange), but if he loses, you pay out $40. Meanwhile FreedomEx, an exchange that has no betting limit for any user, is publishing 6:1 odds for Obama winning. Bet $60 on Obama, and you get $10 if he wins, but pay $60 if he loses. On both markets, of course you can bet in the other direction as well.

What do you conclude from this? That the un-capped FreedomEx probably has more accurate odds, and that as James Surowiecki (author of The Wisdom of Crowds) said, betting limits "make [the markets] less accurate" and "real money is what makes it work"? Or that CappedEx, with its safeguards against manipulation, is more reliable, and FreedomEx is being manipulated by someone trying to change the reported odds of their favored candidate winning? Or that there is simply some random fluctuation in the odds as reported by various markets, so they'll naturally diverge at times?

The correct answer is: you should stop wasting time "concluding" things, and get online as soon as possible and make bets in both markets, because if they're allowing bets to be placed at different odds, you can guarantee yourself a profit.

Make a $50 bet in CappedEx on Obama to win (4:1 odds), and a $10 bet in FreedomEx on Romney to win (1:6 odds). If Obama wins, you win $12.50 in the CappedEx market and lose your $10 in FreedomEx, for a $2.50 profit. If Romney wins, you lose $50 in the CappedEx market but win $60 in FreedomEx, for a $10 profit. With a little algebra, you can show that any time the two markets allow you to place bets at different odds ratios, you can make a guaranteed profit by picking a ratio somewhere in the middle (in this case, the two ratios were 1:4 and 1:6, so we picked 1:5) and making separate bets in the two markets in opposite directions, for amounts in that ratio. (A commenter on the Marginal Revolution blog describes exactly how he made an almost risk-free profit through this kind of "pure arbitrage play". He said it was "almost" risk free because of other factors like currency conversion fluctuations.)

Now, any time a good is trading for a lower price in market A than it is in market B, and the costs of shifting the good between the two markets is negligible, traders will start to buy the good in market A and re-sell it for a profit in market B (the traditional definition of "arbitrage"). This increases demand in market A (driving the price up) and increases supply in market B (driving the price down) until the price difference disappears. In the same way, any time two prediction markets have different "market odds" for the same event, as arbitrage players lock in guaranteed profits by placing opposite bets in the two markets, the market odds in the two markets will converge toward each other until the gap is negligible. This is true even if one of the markets has a cap on what people can invest or how much they can stake on any particular outcome.

For Intrade, there couldn't be a worse time for someone to be pointing this out, but it seems logically inescapable: As long as there is a prediction market anywhere in the world that allows unlimited wagering on a particular outcome, all other prediction markets (whether they are capped or not) can be manipulated indirectly, by playing a large wager in the non-capped market. I was wrong to say that you would have to "enlist your friends" to place bets in the capped market, building a large coalition of market-manipulators (and hoping that none of them would rat you out for using them to circumvent wager-limiting rules). By placing a large wager in the non-capped market, and shifting the market odds there so that they're different from the odds in the capped market, you can indirectly "enlist" all the users in the capped market, to place arbitrage bets and make a guaranteed profit. When this happens, the odds in both the capped market and the non-capped market will shift, as the gap between them narrows -- which means you have manipulated the market odds in the capped market, without ever going near it yourself.

In this case, why have caps on the amounts wagered in prediction markets at all? (The Iowa Electronic Markets have a maximum investment balance of $500, and a 2008 paper, "The Promise of Prediction Markets, authored by several prominent economists, advocated the creation of prediction markets with a maximum investment of $2,000.) Presumably the cap is not to prevent unlucky investors from losing their life's savings, since the law already allows multiple ways to do that, by betting on volatile stocks in the stock market. And it won't stop market manipulation, if the capped market can still be manipulated by using another non-capped market as a proxy. Robin Hanson, Professor of Economics at George Mason University and one of the co-authors of the 2008 paper, candidly told me that the cap was just a matter of selling the idea: "As a practical matter, many people's comfort with such markets increases when there is a cap, so they are more likely to accept the proposal with a cap. So it makes one seem more reasonable to propose a cap, if one can get most of the benefits one wanted from such a system that has a cap, relative to one without it."

So is there a solution to the manipulation problem? Actually, is it even a problem? Robin Hansen and Ryan Oprea wrote another paper arguing that manipulators can improve prediction markets, by subsidizing the existing players in the markets and rewarding them for paying attention. (If a "manipulative" bet causes a sudden shift in the reported odds, opportunistic investors can place bets essentially wagering that the odds will return back to their previous level.) Economist Alex Tabarrok makes the same point here. This opportunism also means that the market shift caused by a manipulative bet usually corrects itself within a few minutes.

Presumably, if more people start to take prediction markets seriously, the incentives to manipulate them would increase. As Tabarrok adds, "prediction markets have truly arrived when people think they are worth manipulating". At the same time though, as more people start to take prediction markets seriously, presumably they'll attract more actual users, and since the amount of money required to shift the market is proportional to the amount already invested by everyone else, this means it will require larger amounts of money to shift the market odds to the same degree.

So these economists all seem to think that prediction market manipulation is a good thing, and that the prediction markets themselves are an even better good thing even when they can be manipulated, but now I'm not so sure. If people do think that market odds are worth manipulating, presumably the point is to create a self-fulfilling prophecy: People think that Romney's chances have gone up, so they become more incentivized to support him and vote for him, and soon his chances actually have gone up (although possibly not to the full extent of the boost in the manipulated market odds, so the manipulator may still lose money). If you can boost Romney's market odds even for a few minutes just by spending a few tens of thousands of dollars, how much would it cost to sustain the higher odds for several hours -- and what if those hours were at a crucial time in the election or in the news reporting cycle?

What if, contrary to my last assumption, people start to take prediction markets seriously enough to be influenced by them, but the prediction markets don't see a proportionate influx of actual investors and money -- so the cost of manipulating them remains about the same? IF prediction markets gain more influence in people's actual voting decisions, BUT those markets don't see an influx of new users, AND an election is close enough that the market odds could make a difference depending on when they're reported, AND someone spends enough to sustain the manipulated odds during crucial periods during the election... Well, that's a lot of assumptions you have to grant, but individually they're quite plausible -- and if all of them hold true, you could change the outcome of a presidential election for just a few million dollars spent on the prediction markets.

And in fact, if you successfully swung the election, you'd actually win all the wagers you had just placed -- which means that now rich manipulators can throw their election to their preferred candidate, and make a bundle. It also means that all those opportunists who usually act to "correct" the market odds deviations, by taking your free money when you start placing manipulative bets, could realize that your bets might actually change the outcome, and would decline to take your money -- which in turn means it would be even cheaper for manipulators to change the outcome, creating a self-reinforcing cycle. If smart bettors see that once a behemoth starts the market moving, the behemoth will probably win, they'll just get out of its way and clear an easier path.

The same kind of trick wouldn't normally work on the stock market -- if you're wealthy enough that you can increase the share price of a stock by buying enough of it to shift the market, then when you try to reap your profits by unloading the stock, the price will drift back down as you're selling it off. (Or if your purchases do manage to create a self-fulfilling prophecy -- your infusion of cash into the company enables them to realize their plans and become a genuine success -- well, then you're just a successful angel investor, more power to you.) But a presidential election prediction market would be analogous to a stock where if you can keep the price artificially inflated for several crucial hours on November 6th, 2012, then the price becomes permanently locked in at that point and you can sell it off for a profit, regardless of the value of the underlying company.

So, according to my own reasoning, this idea that I was so gung-ho about a few days ago, could not only be used to create a type of financial instrument that rewards manipulation more perversely than anything we've ever seen, but could also let a Saudi prince pick the next leader of the free world on a bet.

I'm not sure if there's a solution. I'm not a libertarian so I was never in favor of prediction markets as a matter of "personal liberty"; I was in favor of them because they're useful insofar as they can harness the wisdom of crowds to convey important information. But if they can be manipulated to influence real-world events, is it worth it?

In keeping with the theory that money does motivate people to think harder about such things, I'm once again offering $100 to be split between the readers who email me the best-argued solutions to this problem -- or the best counter-argument to any point I've made here. Put "prediction markets" in the subject line. If your submission wins a portion of the award, you can either claim the money for yourself, or to be donated to a preferred charity in your name. (I reserve the right to pay out less than the allotted $100 if there aren't enough worthy submissions, but that didn't happen last time.) Any sufficiently valuable comments are eligible even if they're not strictly counter-arguments or suggested alternatives, and I'll post a follow-up article summarizing what people send in. You can't make as much off of me, as you could have made by taking some market manipulator's intentionally losing bet on Intrade that Romney was going to win the election, but at least it's legal.

Frequent contributor Bennett Haselton writes: After I sent 10 new proxy sites to my (confirmed-opt-in) mailing list, two of them ended up on one of Spamhaus's blacklists, and as a result, all 10 domains were disabled by the domain registrar, so the sites disappeared from the Web. Did you even know this could happen?"

Since 2005 I've been running a proxy mailing list where users sign up to receive new proxy sites by email. (Proxy sites are sites for getting around Internet blocking software; most proxy sites that you can find through Google are already blocked by major blocking programs, which is why you would sign up to receive new ones by email, to use them until they get blocked as well.) In all that time, we've followed what are considered best practices for email newsletters: every new subscriber is sent a confirmation message by email, and they have to reply to that message, confirming that they really want to subscribe to the emails, before being added to the list. This practice, known as "verified-opt-in," is considered the gold standard for responsible emailing, since it ensures that everyone on your list actually wants to get your emails. (It also ensures that if you accuse an email publisher of spamming because you received their unwanted emails, they can't say, "Oh, one of your friends must have added you" — since if they're using verified-opt-in like they're supposed to, your friends can't add you.) I'm front-loading a lot of information here, although if you saw the words "Spamhaus errors" in the title, you may recognize the technique of literary foreshadowing being employed.

Despite conforming to verified-opt-in standards, the proxy emails have at times been blocked by spam filters used by Hotmail, Gmail, Yahoo Mail, AOL Mail, and various other systems. However, last month was the first time that an incorrect blacklisting caused the domains themselves to be disabled, so that the sites disappeared from the Internet entirely.

On September 17th I registered 10 new .info domains through NameCheap, set up new proxy sites at each of those domains, and mailed each site to 1/10th of our proxy mailing list. (Sending new sites only to a subset of the list makes it harder for blocking software companies to join the list and find all new sites as soon as they're released.) All seemed to be going well until October 2, when subscribers started telling me that they were getting "host not found" errors when trying to reach the sites. I tried the sites myself, found that they were indeed inaccessible, and spent about an hour testing for various problems with DNS servers and domain record settings, before logging in to NameCheap and seeing a message next to each of the new domains saying "domain locked due to illegal activity; please email legal@enom.com." (NameCheap being a reseller for the domain registrar eNom.)

So I sent eNom an email and followed up with a phone call to see if they could speed things up, since complaints kept pouring in from users that the sites were unreachable. eNom said that the domains had actually been suspended by Afilias, the company that handles all .info domain registrations no matter who you buy the domain from, and eNom was in the process of talking with Afilias. So I called Afilias myself to ask about getting the domains unlocked, but they refused to talk to me and said that they could only respond to inquiries from eNom. This, of course, is ridiculous — if someone notifies you that you or your company has made a error, you can investigate the issue no matter who brings it to your attention — and especially in cases where you're literally accusing someone of unspecified "illegal activity," you should bend over backwards to respond to any indication that you might have made a mistake. But they refused to do anything, so I waited for a response back from eNom.

A day and a half ticked by, with emails continuing to come in from our users wondering why the domains had disappeared, until finally eNom forwarded me a response from Afilias saying that two of my ten domains ("drybook.info" and "rootface.info") had been blacklisted by the UK-based organization Spamhaus on their Domain Block List. Spamhaus operates several different alleged "spam" blacklists, and claims that the DBL is a list of domains found in spam messages. The DBL FAQ says that it is "built predominantly using automated spamtraps and email flow monitoring" and "has many checks to prevent legitimate domains being listed," even going so far as to call it a "zero false-positive" list.

Even though only two of the ten domains that I had registered that day had been blacklisted by Spamhaus, Afilias had responded by disabling the entire group of ten domains that I had bought at the same time.

Now here's where I caught a bit of a break: It turns out I was able to get the domains instantly removed from the DBL by entering them in a form on the Spamhaus site and clicking a button, which took me to a page saying:

DBL removal successful
The domain was successfully removed from the DBL. Please allow 30 minutes for servers around the world to update their data. Please note that the domain will be re-listed if malicious activity is detected in the future.

Although, even this easy part of the process didn't inspire much confidence. Not that I wanted Spamhaus to make it harder for me to de-list by domain names, of course, but if you really think your blacklist is 100% accurate, why would you let anyone get any domain removed at any time just by submitting it in a form? In fact, this would seem to give an advantage to spammers over regular website owners — because a spammer, who knows about blacklists and would find it worthwhile to game the system in his favor, would be more likely to know about the Spamhaus DBL and the form for getting their domains de-listed. Whereas for a regular non-spamming website owner, it would take far more time to find out that their domains had been de-activated, that the de-activation had occurred because of an incorrect Spamhaus listing, etc.

Once the listing had been removed, I emailed eNom, who emailed Afilias, who eventually re-activated the domains after a few more hours. But the traffic never returned to the levels that it had been at before the domains were deleted, as most of our users had apparently concluded that the sites had been blocked or taken offline.

Spamhaus did not respond to requests for comment on this story. In fact, Spamhaus does not give you a way to contact them if you have been wrongly blacklisted — their "contacts" page redirects you to the "Blocklist Removal Center" if your domain is blocked, but that only leads you to the automated removal tools, not a way to contact the organization. I did email their "Press Office" email address, on the grounds that I was writing an article for Slashdot in addition to being a wrongly blacklisted domain owner, but didn't get an answer.

So I have no idea what will happen with the next group of domains that I send out to our proxy list. If Spamhaus signed up one of their "spamtrap" email addresses to our mailing list, then presumably any domain mentioned in a message sent to that email, will get automatically blacklisted (even though of course since they signed up the email address to our mailing list, that means it's not spam). If that happens, the entire next batch of domains might get disabled by Afilias as well.

Meanwhile, Spamhaus continues to claim that the DBL is a "zero false-positive" list. I don't know how many other false positives are on the list or how many domains have been abruptly disabled as a result, but if it's this easy to get incorrectly blacklisted, my money is not on "zero."

Bennett Haselton is back with a thought provoking essay about not just an incident of Internet censorship on an American university campus, but a proposed method of propagating news, so that relevant stories aren't buried as easily by chance or time. Bennett writes: "The real scandal in the story of Arizona State University blocking students' access to the Change.org website, is not just that it happened, but that the block persisted for two months without being mentioned in the media. As a card-carrying member of the 'outrage grapevine,' I surely think we need a way to respond faster." Read on for the rest.

This is a tale of censorship. From about December 7th until February 3rd, Arizona State University was blocking all users of its network from accessing the Change.org website, where users can create petitions and circulate them for other users to sign. (The lame excuse offered by the university was that a student had created a petition and was using the change.org site to "spam" other ASU accounts; of course, even if that had been the real reason, it would have easily been possible for ASU to block mail from the change.org servers, without blocking all students from accessing the website.) On February 3rd, after a furor of sudden media attention, the block was lifted.

But that's not the worst instance of censorship in this story. What's more disconcerting is that for the two months that the block was in place, the university's decision to block the website received no media coverage at all. This despite the fact that it was a political website being blocked, at a university with over 70,000 students — a publicly funded university, where a court would have almost certainly found that the blocking violated the First Amendment, had the case ever gone to trial.

I first heard about the original tumblr blog post describing the blocking situation, when someone posted the link on my Facebook wall. So as I went to my profile to read it, I was already predisposed to be pissed off, since almost every link that someone posts on my wall is either an outright scam, or a one-sided rant about an issue that is actually much more complicated than the author thinks it is. Well, it was a one-sided rant, all right, but it was about an issue where there was really only one side: ASU evidently got annoyed about a petition on change.org protesting tuition hikes, so they blocked the site. As I re-read the post, I kept thinking: How can this be true, if we haven't heard about it anywhere else? Perhaps an overzealous ASU network admin put the block in place, and it was reversed just a few hours later, but the tumblr post never got updated? I emailed the blog post's author, Eric Haywood, and the owners of change.org, asking how long the block had lasted before the site was un-blocked — I just assumed that the block couldn't possibly still be in place, two months later. But they confirmed that it was.

The link got blogged and re-blogged around tumblr a few times in December and January, and then, at about the same time as I was sending my emails, the issue suddenly "tipped" into public awareness as it was linked from a widely-read reddit post. Then the blocking received its first official "media" coverage in an article in the ASU student newspaper, the State Press. (Eric Haywood called the article "just ASU spreading it's own propaganda about this issue (they own, run and control the State Press)". I don't know about propaganda, but it did seem a little amateurish — the article says "The author of the original blog post is unknown", even though the guy's name, Eric Haywood, was listed in the post, along with his email address.) Then finally the story spilled over into the "real" media with an article in the Huffington Post, in which the author pointed out that the blocking likely violated the First Amendment. (A few hours after that article appeared, the university unblocked the site so that ASU students could access Change.org on their network again.)

None of the articles commented, however, on how the issue had remained buried for so long; the State Press article said only that the tumblr blog "began circulating the Internet Thursday." A reader could be forgiven for reading the articles and scratching their head and thinking: What is it that just happened? If the site has been blocked for two months, why is this only being written about now?

The answer, I think, is that most people don't realize how arbitrary the process is that determines what issues get news coverage and which ones don't. Before I got involved in a few issues that did receive media coverage (in my late teens, through Peacefire and in co-operative projects with others), I had just assumed that "the news" consisted of all stories that somebody in the media business considered to be "news-worthy." Some journalists just want to sell papers (or attract page-views), while other (better) journalists strive to tell the most important stories — but either way, surely their decision to cover something, or not, should depend on attributes of the story, right? Not on whatever else happened to be going on, or other random circumstances? But then, when I started to be involved in efforts to actually get media coverage for this or that issue, some issues ended up receiving far more coverage than even I thought they really deserved, and others received far less.

Sometimes reporters would frankly admit that they thought something was a good story, but they couldn't cover it because their plate was full that day, and even if they had time later, by that time the issue would be too "cold." Some years ago, I wrote in Slashdot about an experiment in which I sued some spammers in Small Claims court, and filed the court briefs with some of the pages stuck together with a sliver of paper. When the judges rejected the motions (as I expected, since Small Claims judges have been near-uniformly hostile to spam suits), I went to the courthouse to look at the files and found the pages still attached, indicating that the judges had rejected the motions without reading them. What I didn't mention in the original article, was that I had planned at first to give the exclusive story to a Seattle Times reporter, who came down to the courthouse to see the files and interviewed me afterwards. The paper must have thought there was a real story there, since they later sent a photographer to come down and take pictures of the files as well. But then something else landed on the reporter's desk and pushed the story back a few days, and days became weeks, and then the beat switched to a different reporter. When I eventually called to ask if they were still interested, they replied, essentially, that without a current "hook", they couldn't write the story, because now it would look like they weren't doing their jobs for the long intervening period when they didn't write about it, so it was better now to drop it entirely.

Traditional media seems hamstrung by two limitations here: (1) an inefficiency at finding the most important stories that most "deserve" to be written about; and (2) a convention that you can't cover something that's more than a few days old, because then the story looks "dated." The Internet doesn't seem to suffer from limitation #2, as demonstrated by the fact that the blocking of change.org at ASU on December 7th was still able to ignite a controversy on February 3rd. But it does still suffer from limitation #1, as illustrated by the Internet's near-total silence on the issue from December 7th through February 2nd.

Many other people have a pet issue that they think is being "suppressed" by the "liberal media" or the "corporate-owned media" (depending on which side they're on), but the evidence suggests that no conspiracy is necessary to keep an important story from being written about. Sometimes arbitrariness and chance is enough.

My naive earlier assumption — that stories received media coverage because of some combination of attributes of those stories — seems to be a specific instance of a cognitive fallacy, where if you observe that some group of things achieved some end result Z, and all of those things started out possessing some attribute X, then you think that attribute X caused the achievement of result Z. In this case, because we observe that most stories which receive news coverage are important and interesting (with obvious exceptions), we assume that most interesting and important news stories will receive news coverage. Thus, it's frustrating and counterintuitive when we find out about an issue that cries out to be written about, but was ignored by the media. The truth is more likely to be that for every important and interesting story that gets coverage, there are likely to be many other equally important and interesting stories that never make it into the news.

(By the way, I've been unable to find a precise name for the cognitive fallacy wherein if you observe that all things which achieve goal Z have attribute X, then you come to think that attribute X is a good predictor of achieving goal Z. It's not the same as the "post hoc fallacy" or the mistaken belief that "correlation equals causation," because both of those are about the illusion of causation. I'm talking about the correlation being an illusion in the first place — where people come to believe that attribute X is a good predictor of achieving result Z, ignoring the fact that there may be enormous numbers of cases where attribute X is true, but which never go on to achieve result Z. If you know the exact name of that fallacy, shoot me an email and submit a comment below.)

In an earlier article, I proposed a system that would eliminate the arbitrariness in determining which pieces of content are selected to be "the best" and broadcast to a larger audience. I suggested using the algorithm to determine which songs could be pushed out to listeners of a streaming music system, but it could be modified to select which news stories would be considered "important" enough to push out to readers of a news site. (The gist of the idea is that you have each piece of content rated by a random sample of users chosen from the system, and if their average rating is high enough, it gets pushed out to everyone else. If the random sample size is large enough, their average rating will be non-arbitrary, and will be determined by the attributes of the content itself.)

Maybe that algorithm is flawed or maybe someone could find a better one, but the more important thing to realize is that we don't live in that world now, where the attention given to an event is determined by attributes of that event. In the world we actually live in, it's safe to assume that many events take place every day that would have been covered by the news, if it hadn't been for a reporter's missed phone call or some other random happenstance. I have no doubt that the blocking of Change.org on ASU's network could have been a front-page story on CNN, under the right circumstances. I just think that in an ideal world, it should have ended up as a front-page story on CNN regardless of the "circumstances" — but real life, no favorable circumstances means no CNN story.

That might seem like a lot to read into a single case of media silence about a political website being censored at a state university. But while Change.org is no longer blocked at ASU, the inefficient and arbitrary means by which news "events" are discovered and distributed to a wide audience will be with us for a long time.

Slashdot regular contributor Bennett Haselton wrote in this week to say that "The proposed "Telex" anti-censorship system could technically work, but unless I'm missing something, it would more cost-effective to spend the same resources on fighting censorship using existing technologies." His essay on the subject follows.

Professor Alex Halderman published a paper in July describing a new anti-censorship system called Telex, whereby users in censored countries could request banned websites by sending an encrypted request to an SSL-enabled website (i.e., a Web address beginning with https://) outside of their country -- even if the owner of the SSL-enabled website is not participating in the scheme. Since encrypted communications usually contain some random variation, that random variation can be used to embed hidden messages, which can then be decoded by any third-party observer who intercepts the communication and knows how to decode the hidden message. The third-party observer still cannot decode the original encrypted communication between the end user and the SSL-enabled website -- SSL is designed to be unbreakable by all but the intended recipient -- but the observer can decode the "side message" that was designed to be intercepted in transit. So a Telex-enabled router, in the process of passing the communication along, would notice the hidden request for a banned website, and pass the requested content back to the original user.

By analogy, suppose Mrs. Smith wants to send a letter to a friend. Mrs. Smith knows the letter will be sealed, and supposedly unopenable by the postman. But Mrs. Smith also has many choices of colored envelopes to use, and she has agreed with the postman on a color-coded system -- red for "Meet me tonight at the Motel 6", blue for "Not tonight, he suspects something" -- that the postman can "decode" when he picks up the envelope for delivery. The choice of envelope color is the "random variation" inherent in the sending of the message, which the message sender can use to send a "side message" to anyone who passes it along and who knows the system. The postman -- who is analogous to the Telex-enabled router -- has no access to the original sealed message inside the envelope, but he understands the side message just fine. (A Telex user may have no control over what routers their messages pass through, though, so they simply have to hope that there are enough Telex-enabled routers on the Internet that one of them will pick up the message and decode it. Imagine many different amorous mail carriers in the Postal Service, and any one of them who finds the colored envelope will be happy to show up at the appointed time, if Mrs. Smith is not picky.)

The novel feature of Telex is that it would not require the cooperation of the owner of the SSL-enabled website in order to work. You could send an encrypted communication to any website -- https://www.paypal.com/ for example -- and any Telex-enabled routers along the pathway traveled by the connection, would be able to decode the embedded message hidden in the randomness of the encryption. By contrast, for a user to make use of a typical proxy website like Vtunnel, the owner of the Vtunnel website has to set up the site as a proxy; this means the supply of such sites is limited to those websites whose owners have installed proxy software, and the censors have a greater chance of finding and blocking them all. Telex, on the other hand, would continue to work as long as the user in the censored country was able to access any SSL-enabled website, as long as their request happened to pass through a Telex-enabled router.

So far, so good. But this would presumably require an investment of at least several million dollars by any major backbone provider who wanted to try it, by re-configuring their major routers to speak the Telex protocol, and then potentially hundreds of millions of dollars for a sustained long-term effort. (As Halderman says, "We like to envision this technology as a possible government-level response to government-level censorship.") So here's my question: If any backbone provider (or government entity) wanted to go to that trouble to support the cause of fighting Internet censorship, why wouldn't it be much more straightforward for them to just set up proxy websites themselves?

Professor Halderman didn't respond to my inquiry on that point. The Telex FAQ notes that censorious governments can easily block new proxy sites once they find out about them. But in many censored countries, most proxy sites are not blocked, either because the government isn't trying, or they can't keep up. In China, hardly any proxy sites are blocked at all, as the government seems to put more of their resources into suppressing local dissent directly. Meanwhile in Iran, the censors do put more resources into actually blocking proxy sites -- but because Iran is on the U.S. State Department's embargo list, Iranian censors can't buy Internet censoring software from U.S. companies, so they have to find and block the sites themselves. As a result, newly released proxy sites often stay unblocked longer in Iran than they do in other Middle Eastern countries that use U.S.-made blocking software. Meanwhile, Saudi Arabia, for whatever reason, doesn't seem to block proxy sites at all for the time being. (Saudi Arabia is a strange outlier, since most conservative Islamic countries that filter the Web, also block proxy sites as well. It's not clear why Saudi Arabia doesn't.) So if a government or a philanthropist wants to help the cause of fighting censorship, just set up some proxy sites and pay to keep them running -- and you'll be helping the residents of all of those countries right away, for starters. This is in fact what Voice of America (through their various proxy programs) and the founders of UltraSurf (a privately funded network of anti-censorship servers) have been doing all along.

Even in the case of countries like U.A.E. and Yemen that are reasonably quick at finding and blocking proxy sites (as a result of using Western-made blocking software), the most cost-effective way to help these users is probably to set up more proxy sites, hosted at different locations and with perhaps with legitimate-looking "decoy" content, so that U.S. censorware companies can't keep up. My experience has been that the more money you spend (using unique IP addresses, buying .com domains instead of cheap .info ones, and setting up lots of proxies so that each one is sent to only a subset of your target audience), the longer the proxy sites last. You can also use proxy-like services (such as Tor, Hotspot Shield and UltraSurf) to route traffic through dedicated servers, to circumvent censorship in a way that is more transparent and convenient to the end user.

In short, existing proxy sites (and proxy-like services) do the job pretty well for many censored countries, and a massive cash expenditure on setting up more proxies (equivalent to the cost of setting up the Telex system) would probably be enough to demolish all other national filtering schemes completely. The software and tools to run proxy sites have already been tried and tested; all it takes to run them is money. Telex, by contrast, would require backbone providers to alter the architecture of their systems -- which means large-scale testing, isolation of any problems that arise, and countless other potential headaches. And that's not even counting the fact that censorious countries might detect which backbone providers are using Telex, and block all traffic from their countries to any sites hosted on those networks.

So I think Telex is a brilliant technical achievement, and I'd be happy if it got deployed, but I'd be scratching my head as to why the backbone providers (or the government, or whoever sponsored the effort) decided to kill a gnat with a flamethrower. I deal in flyswatters for a living, and they get the job done.

Slashdot regular contributor Bennett Haselton wrote in this week to say that "The proposed "Telex" anti-censorship system could technically work, but unless I'm missing something, it would more cost-effective to spend the same resources on fighting censorship using existing technologies." His essay on the subject follows.

Professor Alex Halderman published a paper in July describing a new anti-censorship system called Telex, whereby users in censored countries could request banned websites by sending an encrypted request to an SSL-enabled website (i.e., a Web address beginning with https://) outside of their country -- even if the owner of the SSL-enabled website is not participating in the scheme. Since encrypted communications usually contain some random variation, that random variation can be used to embed hidden messages, which can then be decoded by any third-party observer who intercepts the communication and knows how to decode the hidden message. The third-party observer still cannot decode the original encrypted communication between the end user and the SSL-enabled website -- SSL is designed to be unbreakable by all but the intended recipient -- but the observer can decode the "side message" that was designed to be intercepted in transit. So a Telex-enabled router, in the process of passing the communication along, would notice the hidden request for a banned website, and pass the requested content back to the original user.

By analogy, suppose Mrs. Smith wants to send a letter to a friend. Mrs. Smith knows the letter will be sealed, and supposedly unopenable by the postman. But Mrs. Smith also has many choices of colored envelopes to use, and she has agreed with the postman on a color-coded system -- red for "Meet me tonight at the Motel 6", blue for "Not tonight, he suspects something" -- that the postman can "decode" when he picks up the envelope for delivery. The choice of envelope color is the "random variation" inherent in the sending of the message, which the message sender can use to send a "side message" to anyone who passes it along and who knows the system. The postman -- who is analogous to the Telex-enabled router -- has no access to the original sealed message inside the envelope, but he understands the side message just fine. (A Telex user may have no control over what routers their messages pass through, though, so they simply have to hope that there are enough Telex-enabled routers on the Internet that one of them will pick up the message and decode it. Imagine many different amorous mail carriers in the Postal Service, and any one of them who finds the colored envelope will be happy to show up at the appointed time, if Mrs. Smith is not picky.)

The novel feature of Telex is that it would not require the cooperation of the owner of the SSL-enabled website in order to work. You could send an encrypted communication to any website -- https://www.paypal.com/ for example -- and any Telex-enabled routers along the pathway traveled by the connection, would be able to decode the embedded message hidden in the randomness of the encryption. By contrast, for a user to make use of a typical proxy website like Vtunnel, the owner of the Vtunnel website has to set up the site as a proxy; this means the supply of such sites is limited to those websites whose owners have installed proxy software, and the censors have a greater chance of finding and blocking them all. Telex, on the other hand, would continue to work as long as the user in the censored country was able to access any SSL-enabled website, as long as their request happened to pass through a Telex-enabled router.

So far, so good. But this would presumably require an investment of at least several million dollars by any major backbone provider who wanted to try it, by re-configuring their major routers to speak the Telex protocol, and then potentially hundreds of millions of dollars for a sustained long-term effort. (As Halderman says, "We like to envision this technology as a possible government-level response to government-level censorship.") So here's my question: If any backbone provider (or government entity) wanted to go to that trouble to support the cause of fighting Internet censorship, why wouldn't it be much more straightforward for them to just set up proxy websites themselves?

Professor Halderman didn't respond to my inquiry on that point. The Telex FAQ notes that censorious governments can easily block new proxy sites once they find out about them. But in many censored countries, most proxy sites are not blocked, either because the government isn't trying, or they can't keep up. In China, hardly any proxy sites are blocked at all, as the government seems to put more of their resources into suppressing local dissent directly. Meanwhile in Iran, the censors do put more resources into actually blocking proxy sites -- but because Iran is on the U.S. State Department's embargo list, Iranian censors can't buy Internet censoring software from U.S. companies, so they have to find and block the sites themselves. As a result, newly released proxy sites often stay unblocked longer in Iran than they do in other Middle Eastern countries that use U.S.-made blocking software. Meanwhile, Saudi Arabia, for whatever reason, doesn't seem to block proxy sites at all for the time being. (Saudi Arabia is a strange outlier, since most conservative Islamic countries that filter the Web, also block proxy sites as well. It's not clear why Saudi Arabia doesn't.) So if a government or a philanthropist wants to help the cause of fighting censorship, just set up some proxy sites and pay to keep them running -- and you'll be helping the residents of all of those countries right away, for starters. This is in fact what Voice of America (through their various proxy programs) and the founders of UltraSurf (a privately funded network of anti-censorship servers) have been doing all along.

Even in the case of countries like U.A.E. and Yemen that are reasonably quick at finding and blocking proxy sites (as a result of using Western-made blocking software), the most cost-effective way to help these users is probably to set up more proxy sites, hosted at different locations and with perhaps with legitimate-looking "decoy" content, so that U.S. censorware companies can't keep up. My experience has been that the more money you spend (using unique IP addresses, buying .com domains instead of cheap .info ones, and setting up lots of proxies so that each one is sent to only a subset of your target audience), the longer the proxy sites last. You can also use proxy-like services (such as Tor, Hotspot Shield and UltraSurf) to route traffic through dedicated servers, to circumvent censorship in a way that is more transparent and convenient to the end user.

In short, existing proxy sites (and proxy-like services) do the job pretty well for many censored countries, and a massive cash expenditure on setting up more proxies (equivalent to the cost of setting up the Telex system) would probably be enough to demolish all other national filtering schemes completely. The software and tools to run proxy sites have already been tried and tested; all it takes to run them is money. Telex, by contrast, would require backbone providers to alter the architecture of their systems -- which means large-scale testing, isolation of any problems that arise, and countless other potential headaches. And that's not even counting the fact that censorious countries might detect which backbone providers are using Telex, and block all traffic from their countries to any sites hosted on those networks.

So I think Telex is a brilliant technical achievement, and I'd be happy if it got deployed, but I'd be scratching my head as to why the backbone providers (or the government, or whoever sponsored the effort) decided to kill a gnat with a flamethrower. I deal in flyswatters for a living, and they get the job done.

On Monday we discussed news of a Supreme Court ruling which held that violent video games deserved free speech protection under the First Amendment. Now, frequent Slashdot contributor Bennett Haselton writes with this followup that questions the Court's consistency in such matters. "I'm glad the Supreme Court struck down the California law against selling violent video games to minors, but reading over the decision, I had the odd feeling that the arguments by the dissenters made more sense than the majority — mainly because of the hypocrisy of continuing to ban sexuality while giving violence a pass." Read on for the rest of Bennett's thoughts.

John Landis said, "R is when you bare a woman's breast, PG is when you cut it off." That is apparently now also the law of the land regarding video games, according to the Supreme Court's June 27th decision (PDF) overturning a California law that banned sales of violent video games to minors. I'm glad the Supreme Court struck down the law, but reading over the decision, I had the odd feeling that even though I agreed with the majority's conclusion, the actual arguments made by the dissenters made more sense, primarily because of the hypocrisy of the majority in treating sex as more taboo than violence.

The majority opinion, written by Scalia, has already been widely quoted as a ringing defense of free speech:

"Reading Dante is unquestionably more cultured and intellectually edifying than playing Mortal Kombat. But these cultural and intellectual differences are not constitutional ones. Crudely violent video games, tawdry TV shows, and cheap novels and magazines are no less forms of speech than The Divine Comedy, and restrictions upon them must survive strict scrutiny..."

But Scalia continues to believe that the government does have the right to ban the sale of nudity and sexuality to minors (as decided in the Supreme Court's 1968 Ginsberg v. New York decision), just not violence. So he kept qualifying statements like the one above by adding "except for pornography", like a judicial version of the fortune cookie "in bed" game:

"[A]s a general matter, . . . government has no power to restrict expression because of its message, its ideas, its subject matter, or its content... There are of course exceptions. These limited areas, such as obscenity... represent well-defined and narrowly limited classes of speech, the prevention and punishment of which have never been thought to raise any Constitutional problem."
...
"Speech that is neither obscene as to youths nor subject to some other legitimate proscription cannot be suppressed solely to protect the young from ideas or images that a legislative body thinks unsuitable for them."

So he's continuing the Supreme Court's tradition of carving out of a First Amendment exception for sex, but won't make one for gratuitous violence. I would be against banning either type of content, but if I were forced to ban one of the two, I would definitely pick violence. Wouldn't you?

As Steven Breyer wrote in his dissent:

"But what sense does it make to forbid selling to a 13-year-old boy a magazine with an image of a nude woman, while protecting a sale to that 13-year-old of an interactive video game in which he actively, but virtually, binds and gags the woman, then tortures and kills her? What kind of First Amendment would permit the government to protect children by restricting sales of that extremely violent video game only when the woman -- bound, gagged, tortured, and killed -- is also topless?"

Well, he's right, isn't he? Except he misses the point that perhaps the remedy is not to ban violent video games, but to overturn the precedent that photos of topless women are harmful.

Alito seemed to agree with Breyer, when he wrote in a decision joined by Roberts:

"Victims by the dozens are killed with every imaginable implement, including machine guns, shotguns, clubs, hammers, axes, swords, and chainsaws. Victims are dismembered, decapitated, disemboweled, set on fire, and chopped into little pieces. They cry out in agony and beg for mercy... The objective of one game is to rape a mother and her daughters; in another, the goal is to rape Native American women."

(Alito was technically not dissenting, because he agreed that the current law was impermissibly vague, but filed a separate opinion because he was at pains to emphasize that he thought some future law against violent video games might be constitutional.) The implication seems clear: "If we can ban some things for minors — like pornography — then good God, can't we ban this stuff too?"

Scalia, in his majority opinion, responds to Alito's description of game violence: "Justice Alito recounts all these disgusting video games in order to disgust us — but disgust is not a valid basis for restricting expression." But this is just hypocritical — because Scalia, throughout his own decision, kept deferring to the Ginsberg Supreme Court ruling, which said that the government could ban porn sales to minors if it depicted sex acts in way that the "average person" would consider "patently offensive with respect to what is suitable for minors" (along with some other criteria). In other words, if it causes disgust.

Breyer and Alito also made similar arguments to each other on another reasonable-sounding point — that industry self-regulation might not last long, now that the law has been struck down. As Alito wrote:

"The Court does not mention the fact that the industry adopted this system in response to the threat of federal regulation, Brief for Activision Blizzard, Inc., as Amicus Curiae 7-10, a threat that the Court's opinion may now be seen as largely eliminating. Nor does the Court acknowledge that compliance with this system at the time of the enactment of the California law left much to be desired — or that future enforcement may decline if the video-game industry perceives that any threat of government regulation has vanished."

Breyer agreed:

"And the industry could easily revert back to the substantial noncompliance that existed in 2004, particularly after today's broad ruling reduces the industry's incentive to police itself."

This sounds more realistic than Scalia's recitation of the video game industry party line:

"The video-game industry has in place a voluntary rating system designed to inform consumers about the content of games... This system does much to ensure that minors cannot purchase seriously violent games on their own, and that parents who care about the matter can readily evaluate the games their children bring home."

What do you want to bet that Breyer and Alito are right, and enforcement of the rating system will decline now?

Compare this with another case, when Communications Decency Act of 1996 (essentially banning the "seven dirty words" on the Internet) was struck down in 1997 at least in part because a "less restrictive means" existed for censoring content in the home — parental blocking software. I didn't like blocking software much, but as a statement of fact, it existed, and was a less restrictive means than the law. The crucial difference there was that parents who used blocking software, weren't using it in response to a government threat of legislation, they were using it because they wanted to, and didn't stop using it after the law was struck down. There's no reason to think the same is true for industry self-applied video game ratings.

Finally, Breyer (but not Alito) rejected the argument that the California law should be struck down for vagueness, arguing that it was no more vague than laws against selling pornography minors, which the court had upheld:

"Comparing the language of California's statute (set forth supra, at 1-2) with the language of New York's statute (set forth immediately above), it is difficult to find any vagueness-related difference. Why are the words "kill," "maim," and "dismember" any more difficult to understand than the word "nudity?" ... California only departed from the Miller formulation [the Supreme Court case that defined obscenity] in two significant respects: It substituted the word "deviant" for the words "prurient" and "shameful," and it three times added the words "for minors." The word "deviant" differs from "prurient" and "shameful," but it would seem no less suited to defining and narrowing the reach of the statute."

Well, I think he's right. They're all just words, and they don't have crystal clear boundaries, but you pretty much know what they mean, and there's no reason why one group of words is more vague than the other. (In fact, in a 2008 article I argued that you could measure scientifically the vagueness of a law — just show the law to different test subjects, along with some made-up scenarios, and ask whether those scenarios violated the law or not. I'm quite confident that if you applied that test to these two different laws, you would measure about the same level of "vagueness".)

Again, I don't accept the justices' premise that the government has any business banning the sale of either sexual or violent content. But if you're going to grant the premise that they can and should, then Alito and/or Breyer seem to have made better arguments than the majority on at least those three points: That violence probably deserves less constitutional protection than sex, that the industry isn't likely to keep regulating itself if they no longer think they have to, and there's no reason that "kill" and "maim" are any more vague than "nudity".

(By the way, when I say the "dissenters sounded more reasonable", I am not including Clarence Thomas, whose entire solo dissent was devoted to research showing that the Founding Fathers did not believe people under 18 had First Amendment rights at all. If Clarence Thomas thought really hard, could he think of any other category of people who were denied full civil rights in the 1700s, and hence why we wouldn't want to apply that standard today?)

Fortunately, the majority did get the most important point right, which is that studies do not show a causal relationship between video game playing and real-life acts of violence. As Scalia wrote:

"The State's evidence is not compelling. California relies primarily on the research of Dr. Craig Anderson and a few other research psychologists whose studies purport to show a connection between exposure to violent video games and harmful effects on children. These studies have been rejected by every court to consider them, and with good reason: They do not prove that violent video games cause minors to act aggressively (which would at least be a beginning). Instead, "[n]early all of the research is based on correlation, not evidence of causation, and most of the studies suffer from significant, admitted flaws in methodology." Video Software Dealers Assn. 556 F. 3d, at 964. They show at best some correlation between exposure to violent entertainment and minuscule real-world effects, such as children's feeling more aggressive or making louder noises in the few minutes after playing a violent game than after playing a nonviolent game."

Unfortunately, Scalia lacked the nerve to say that this point should have been the only point that mattered, in a society where freedom is the default unless there's a good reason to the contrary. Because the logical consequence of that, would have been that since the "evidence" for the harmful effects of pornography is even weaker, then the government has no business banning that, either.

The problem constraining all nine justices is that they felt bound by the prior Ginsberg ruling making it permissible to ban sales of pornography to minors, so their options were limited to (a) striking down the video game law while ignoring the hypocrisy of continuing to ban pornography, or (b) pointing out that violent video games are probably at least as distasteful. This ignores the possibility that they could have just (c) overturned their prior ruling, as they have done many times before.

If I were a justice writing for the majority, my whole opinion would be:

Well, we can only make an exception to the First Amendment if there's solid evidence of real harm, and there is no scientifically valid evidence of harm here, so the law violates the First Amendment and is struck down. Oh, and that goes for Ginsberg too, next time it comes up. How much did you guys pay for law school again?

Unfortunately, Obama has said that he's looking for Supreme Court candidates that display "empathy", and what I said would probably hurt the other justices' feelings, so don't hold your breath for my being nominated.

Slashdot regular contributor Bennett Haselton writes: "An anti-porn organization's Facebook page is disabled by Facebook, and then resurrected. Was the page the victim of a 'complaint mob,' and could the previously-discussed 'voting algorithm' have saved the page from being shut down?"

Speaking of Facebook pages being unjustly shut down, on Monday the anti-porn Facebook page http://www.facebook.com/PornHarms/, run by the non-profit Morality in Media, was abruptly disabled by Facebook. The page had 35,000 "likes" at the time the plug was pulled. Morality in Media CEO Patrick Trueman, who also ran the Facebook page, says he never received any warning from Facebook before the page was removed.

Some time on Wednesday, the page was restored. I had emailed a contact at Facebook to ask why the page was shut down, and he replied later to say that it had been deleted in error and the page had been restored. (He didn't say whether the page was on track to being restored anyway, or whether it would have remained down indefinitely if I hadn't pinged him.)

Facebook did not respond to inquiries as to why the page was removed, but as Evgeny Morozov has pointed out regarding political pages (and as many other users have heard from people's anecdotal experiences having pages pulled without explanation), it's common for pages on Facebook and YouTube to get removed that were almost certainly not violating those sites' Terms of Service. If enough users decide to file "abuse complaints" simultaneously against a piece of content on Facebook or YouTube, this has a good chance of getting the content removed, whether the complaints were legitimate or were simply part of an organized campaign of filing false complaints.

Meanwhile, I correspond with dozens of people every week on Facebook (usually people who use my proxy sites to get on Facebook at school or work), and about once a week I get an automated message from Facebook that says, "You have been sending harassing messages to other users," and goes on to sternly list the types of messages that violate Facebook's TOS. (Only twice has this resulted in my account actually getting locked, and it was unlocked after I bugged my friend at Facebook about it.)

I figure that these are either the result of users clicking "Report this message" accidentally, or parents hacking into their kids' accounts, reading their messages, and then trying to get the account shut down of the person who was talking their kid about proxy sites. In either case, I assume it's not the result of an "organized campaign," but perhaps your account gets locked if you're unlucky enough that two or three people file complaints within the same short time frame.

So I have no reason to doubt Mr. Trueman's claim that the PornHarms Facebook page never contained any content that violated Facebook's TOS. He says the page mostly contained links to academic research supposedly demonstrating the harmful effects of pornography, and that while the target audience was adult academics, there was nothing in the content that most parents would consider inappropriate for underage viewers. There was certainly no actual pornography on the page, not even in censored form with the fun parts blurred out (although I didn't check every single academic paper linked from the site to see if any of them might have used pixellated/censored porn for illustrative purposes). Trueman also says that they prevented third-party users from posting on the PornHarms page directly, and regularly monitored the page's content to remove any "inappropriate" comments that users had written in response to the officially authorized posts. (Of course, even if the page admins hadn't done this, inappropriate comments should be the basis for penalizing the user who posted them, not the Facebook page that they were posted on, but it was a moot point in this case.)

Because of the word "Pornography" in the title of the page, it's also of course possible that a human at Facebook actually did review the complaints, but thought the word "pornography" meant the page was a porn-trading hub, without looking to closely at it. (It's also possible that the word triggered an automated filter at Facebook. Obviously, there is no filter pre-emptively preventing pages with words like "pornography" in the title from being created, since otherwise the page never could have existed in the first place. But it's possible that an automated algorithm does something like the following: If a page receives X complains within time period Y, and the page contains certain keywords in the title or the content, then shut down the page automatically.)

Previously I'd suggested an algorithm that Facebook could use to stop users from coordinating phony complaints in order to shut a page down. The gist was: If a page receives a sufficient number of complaints, have the page reviewed by a random sample (chosen by Facebook) of Facebook users who had signed up to review abuse cases in situations such as these. If enough of those users vote that the page was violating the TOS, the page gets shut down, but if not, then it stays up. What makes this algorithm difficult to abuse, is that in order for a "coordinated mob" to swing the vote of the jury, they would have to comprise a majority (or a significant minority) of the entire set of users that the randomly-selected jury could have been chosen from -- a difficult task if thousands of people have signed up as content reviewers. I offered a $100 prize to be split between readers who submitted the best suggested improvements or criticisms of the idea; their ideas were summarized in a follow-up article. A couple of readers commented that there was no point in debating the idea since I don't work for Facebook and have no influence there; they have a point, but the idea has to start somewhere. If engineers at Facebook are looking for a way to fix the problem, one thing that can be said about this suggestion is that it was posted to a large audience of smart people, and several readers suggested very clever improvements, while nobody found any obviously fatal flaws in it.

It seems pretty likely that a process like that for reviewing abuse complaints, would have saved the Pornography Harms page from being yanked from Facebook. Anybody who seriously reviewed the page's contents for more than twenty seconds would have understood the page's real purpose and seen that it was not actually distributing pornography or otherwise violating the Facebook TOS. In my experiences posting surveys on sites like Mechanical Turk, where you can pay users a penny apiece for filling out surveys or performing other tasks, I've gotten the impression that people will take such tasks seriously, even for zero (or virtually zero) pay, if they find them interesting. In the case of the Facebook "jurors" who are voting on whether a page violated the TOS, you're talking about users who voluntarily signed up to be jurors, after all -- not underpaid workers grinding through as many tasks as they can squeeze into their working hours.

Finally, it would be easy to point out the irony of a pro-censorship group being censored (and some people did, on the mailing lists where I saw this news announced), but I don't think that's really fair to Morality in Media, since even MIM doesn't oppose people's right to express their opinions in favor of pornography. Likewise, MIM presumably supports the use of Internet blocking programs in schools, even though their Facebook page (as well as the companion website PornHarms.com) would probably be blocked by default by most Internet blockers because of the word "porn" in the URL -- but even that is not as richly ironic as it would seem. Neither Morality in Media, nor almost anyone else, is in favor of political sites about pornography being blocked because of the word "porn" in the address; presumably they'd just want the error corrected by the blocking company, and if a left-wing site on the opposite side of the debate happened to be blocked because of the word "porn" in the URL, I have no reason to think that Morality in Media would be opposed to correcting that error and unblocking that site as well. So this really isn't a case of them being given "a taste of their own medicine."

No, the real irony in this particular case -- at least, if I did have a role in getting their Facebook page restored -- is that not only would I support their right to express their view (duh), I would support students' right to bypass their school's Internet blocker to view the page from school if they had to, and I would even support the right of under-18-year-olds to view the page even if their parents were specifically trying to block them from it. I highly doubt that even anyone at Morality in Media would go that far.

Slashdot regular contributor Bennett Haselton writes: "An anti-porn organization's Facebook page is disabled by Facebook, and then resurrected. Was the page the victim of a 'complaint mob,' and could the previously-discussed 'voting algorithm' have saved the page from being shut down?"

Speaking of Facebook pages being unjustly shut down, on Monday the anti-porn Facebook page http://www.facebook.com/PornHarms/, run by the non-profit Morality in Media, was abruptly disabled by Facebook. The page had 35,000 "likes" at the time the plug was pulled. Morality in Media CEO Patrick Trueman, who also ran the Facebook page, says he never received any warning from Facebook before the page was removed.

Some time on Wednesday, the page was restored. I had emailed a contact at Facebook to ask why the page was shut down, and he replied later to say that it had been deleted in error and the page had been restored. (He didn't say whether the page was on track to being restored anyway, or whether it would have remained down indefinitely if I hadn't pinged him.)

Facebook did not respond to inquiries as to why the page was removed, but as Evgeny Morozov has pointed out regarding political pages (and as many other users have heard from people's anecdotal experiences having pages pulled without explanation), it's common for pages on Facebook and YouTube to get removed that were almost certainly not violating those sites' Terms of Service. If enough users decide to file "abuse complaints" simultaneously against a piece of content on Facebook or YouTube, this has a good chance of getting the content removed, whether the complaints were legitimate or were simply part of an organized campaign of filing false complaints.

Meanwhile, I correspond with dozens of people every week on Facebook (usually people who use my proxy sites to get on Facebook at school or work), and about once a week I get an automated message from Facebook that says, "You have been sending harassing messages to other users," and goes on to sternly list the types of messages that violate Facebook's TOS. (Only twice has this resulted in my account actually getting locked, and it was unlocked after I bugged my friend at Facebook about it.)

I figure that these are either the result of users clicking "Report this message" accidentally, or parents hacking into their kids' accounts, reading their messages, and then trying to get the account shut down of the person who was talking their kid about proxy sites. In either case, I assume it's not the result of an "organized campaign," but perhaps your account gets locked if you're unlucky enough that two or three people file complaints within the same short time frame.

So I have no reason to doubt Mr. Trueman's claim that the PornHarms Facebook page never contained any content that violated Facebook's TOS. He says the page mostly contained links to academic research supposedly demonstrating the harmful effects of pornography, and that while the target audience was adult academics, there was nothing in the content that most parents would consider inappropriate for underage viewers. There was certainly no actual pornography on the page, not even in censored form with the fun parts blurred out (although I didn't check every single academic paper linked from the site to see if any of them might have used pixellated/censored porn for illustrative purposes). Trueman also says that they prevented third-party users from posting on the PornHarms page directly, and regularly monitored the page's content to remove any "inappropriate" comments that users had written in response to the officially authorized posts. (Of course, even if the page admins hadn't done this, inappropriate comments should be the basis for penalizing the user who posted them, not the Facebook page that they were posted on, but it was a moot point in this case.)

Because of the word "Pornography" in the title of the page, it's also of course possible that a human at Facebook actually did review the complaints, but thought the word "pornography" meant the page was a porn-trading hub, without looking to closely at it. (It's also possible that the word triggered an automated filter at Facebook. Obviously, there is no filter pre-emptively preventing pages with words like "pornography" in the title from being created, since otherwise the page never could have existed in the first place. But it's possible that an automated algorithm does something like the following: If a page receives X complains within time period Y, and the page contains certain keywords in the title or the content, then shut down the page automatically.)

Previously I'd suggested an algorithm that Facebook could use to stop users from coordinating phony complaints in order to shut a page down. The gist was: If a page receives a sufficient number of complaints, have the page reviewed by a random sample (chosen by Facebook) of Facebook users who had signed up to review abuse cases in situations such as these. If enough of those users vote that the page was violating the TOS, the page gets shut down, but if not, then it stays up. What makes this algorithm difficult to abuse, is that in order for a "coordinated mob" to swing the vote of the jury, they would have to comprise a majority (or a significant minority) of the entire set of users that the randomly-selected jury could have been chosen from -- a difficult task if thousands of people have signed up as content reviewers. I offered a $100 prize to be split between readers who submitted the best suggested improvements or criticisms of the idea; their ideas were summarized in a follow-up article. A couple of readers commented that there was no point in debating the idea since I don't work for Facebook and have no influence there; they have a point, but the idea has to start somewhere. If engineers at Facebook are looking for a way to fix the problem, one thing that can be said about this suggestion is that it was posted to a large audience of smart people, and several readers suggested very clever improvements, while nobody found any obviously fatal flaws in it.

It seems pretty likely that a process like that for reviewing abuse complaints, would have saved the Pornography Harms page from being yanked from Facebook. Anybody who seriously reviewed the page's contents for more than twenty seconds would have understood the page's real purpose and seen that it was not actually distributing pornography or otherwise violating the Facebook TOS. In my experiences posting surveys on sites like Mechanical Turk, where you can pay users a penny apiece for filling out surveys or performing other tasks, I've gotten the impression that people will take such tasks seriously, even for zero (or virtually zero) pay, if they find them interesting. In the case of the Facebook "jurors" who are voting on whether a page violated the TOS, you're talking about users who voluntarily signed up to be jurors, after all -- not underpaid workers grinding through as many tasks as they can squeeze into their working hours.

Finally, it would be easy to point out the irony of a pro-censorship group being censored (and some people did, on the mailing lists where I saw this news announced), but I don't think that's really fair to Morality in Media, since even MIM doesn't oppose people's right to express their opinions in favor of pornography. Likewise, MIM presumably supports the use of Internet blocking programs in schools, even though their Facebook page (as well as the companion website PornHarms.com) would probably be blocked by default by most Internet blockers because of the word "porn" in the URL -- but even that is not as richly ironic as it would seem. Neither Morality in Media, nor almost anyone else, is in favor of political sites about pornography being blocked because of the word "porn" in the address; presumably they'd just want the error corrected by the blocking company, and if a left-wing site on the opposite side of the debate happened to be blocked because of the word "porn" in the URL, I have no reason to think that Morality in Media would be opposed to correcting that error and unblocking that site as well. So this really isn't a case of them being given "a taste of their own medicine."

No, the real irony in this particular case -- at least, if I did have a role in getting their Facebook page restored -- is that not only would I support their right to express their view (duh), I would support students' right to bypass their school's Internet blocker to view the page from school if they had to, and I would even support the right of under-18-year-olds to view the page even if their parents were specifically trying to block them from it. I highly doubt that even anyone at Morality in Media would go that far.