OKCupid Experiments on Users Too

With recent news that Facebook altered users' feeds as part of a psychology experiment, OKCupid has jumped in and noted that they too have altered their algorithms and experimented with their users (some unintentional) and "if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work." Findings include that removing pictures from profiles resulted in deeper conversations, but as soon as the pictures returned appearance took over; personality ratings are highly correlated with appearance ratings (profiles with attractive pictures and no other information still scored as having a great personality); and that suggesting a bad match is a good match causes people to converse nearly as much as ideal matches would.

Comments

Flash panic

[LordLucless]

World discovers A/B testing
Freaks out
Until the next reality tv show comes on

Re:Flash panic

[stranger_to_himself]

World discovers A/B testing
Freaks out
Until the next reality tv show comes on

When we (academics) do experiments on people however trivial we usually have to go through ethical clearance, get informed consent etc. I think its skipping that part that people are uncomfortable about. Of course that happens every day in the business world (and even did before computer scientists rediscovered basic experiments and called it A/B testing), but in some of these cases it does start to look like an academic psychology experiment. Perhaps use of OK Cupid implies consent to be experimented on but I doubt that consent is collected in a transparent way.

Re:Flash panic

[DerekLyons]

When we (academics) do experiments on people however trivial we usually have to go through ethical clearance, get informed consent etc. I think its skipping that part that people are uncomfortable about.

The operative word being "usually", which implies there exist cases where you don't. The discomfort come from people not grasping the existence of the "usually", and that businesses are not academics and product testing is not held to the same standard.

Re:Flash panic

[bondsbw]

When we (academics) do experiments on people however trivial we usually have to go through ethical clearance, get informed consent etc.

Academic experiments have external results, publishing findings as scientific research. Business experiments have internal results, data mining with the goal of increasing profits (via providing better value to the consumer, at least in capitalist theory).

Well, at least, I can hope the results stay internal to the business. As with data mining in general, that's not always the case. But perhaps this becoming a mainstream topic will end with a framework on which to judge companies that release "experiment" data about their customers.

Re:Flash panic

[taustin]

It's hard to imagine how anyone could find this to be scientific experimentation, rather than some random crap done in hopes of finding a way to sell more advertising.

Re:Flash panic

[mattwarden]

Do academic demographers get "informed consent" before processing census data? What about crime statistics? Network security incidents?

Showing a page with and without images and then processing access_log is not the same as monitoring someone's eating habits and stress levels for a week. Just because you call something "an experiment" (a) doesn't mean it is one, and (b) doesn't mean it's the same as all other "experiments".

Re:Flash panic

[pipedwho]

The problem with most 'commercial experimentation' is that it isn't about getting better value for the consumer, but about how to to best convince the consumer to pay more for something, or buy something, that they otherwise would not have.

Loyalty cards are a way for a business to encourage a customer to return whether or not it is really in their best interest. Phone contracts, transaction 'fees' and 'licensing' are other ways to get people coming back for more of a beating. If you make the fine print and pricing structure too complicated to understand, while offering all sorts of shiny bling in the big print, marketers have found that they can significantly increase sales. Auto bank account debits are great in that the consumer starts to forget that they are continually paying for something, and may take a few extra months (and therefore payments) to cancel a service that they are no longer using - especially when you make it difficult to do.

Factory rebates are another example of sneaky marketing. They make it hard to claim the rebate, in some cases always 'losing' your first application, or finding something incorrect or incomplete in their overly complicated request form. In the end they pay out less than 1 in 5 rebates because most people give up trying to claim that $100. However, when buying the product the consumer factored in the rebate and probably avoided a more suitable competitor with a better more 'expensive' product.

All these techniques would have been arrived at by experimenting on consumers. It is simply about a business trying to get as close to the threshold of pain as possible to maximise profit. Too far and they go backwards - which is why they experiment on a small sample of their market before any large scale roll out.

Re:Flash panic

[Bing Tsher E]

That's weird, because here where I live 'Loyalty Cards' are just the only means by which you would ever want to frequent the store. Kroeger (groceries) Walgreen and CVS (both drugstores) all have loyalty card programs. Especially with Kroeger, the prices if you don't use the card are such that you'd just never go into the store with those prices. It doesn't really 'pull' you to Kroeger over any other store chain.

Re: Flash panic

[bondsbw]

Specifically regarding getting you to purchase something you wouldn't have, well, I don't see that as bad. You buy stuff because that stuff is worth more to you than the money is sitting in your wallet. That is what every honest transaction is...an exchange that favors both parties.

Shady practices like making it difficult to redeem a coupon are a different story, and frankly should be illegal.

Re:Flash panic

[pipedwho]

It encourages you to sign up for a loyalty card in the first place, otherwise you pay 'full price'. Once you have the loyalty card you start thinking about going to one of the shops that you are 'loyal' to rather than the corner shop down the street. (This may not be individually true for everyone, but on average that is how it works.)

This sort of 'discounting' is kind of like in industries without loyalty programs where there is an RRP and a much lower 'street price'. The shops don't actually expect people to pay the RRP (with the exception of someone that is either uninformed or desperate to buy a more difficult to source item immediately). But it does allow them a lot of 'discounting' and haggle margin to make the customers feel good about what they're paying.

Once the first few chain stores started doing this, the others followed suit. So now you have a sad state of affairs where the 'chain cartel' has a huge advantage when it comes to repeat customers. Your local corner store loses business and has to jack up their prices to pay the rent, causing even less customers to shop there. Less competition from small players. Big chain stores can get away with this because they always have one or two items that the other big chain stores don't and thus you end up signing up.

As a loyalty card holder, the retailers have far more detailed (marketable/mineable) information about you than they otherwise would have. Now they can run the numbers to see where its safe to jack up the price without losing customers.

With competition reduced, once a company has significant control over a market (either geographically or in a category segment), up go the prices to the maximum the market can afford.

So you may think you're getting a discount, but the discount is simply from an artificially inflated starting price. Or in some industries funded by the fact that you are now buying more than you otherwise needed to. It is all a numbers game. The big retail giants have increased profits year over year, so at no point have they been forced to release some of that profit back to either the suppliers by paying more, or to the consumers in the form of honest price reductions.

Re: Flash panic

[pipedwho]

I have to agree that buying something useful that you otherwise would not necessarily have bought in the first place is not a bad thing for either yourself or the economy as a whole.

Re:Flash panic

[LordLucless]

The two are not mutually exclusive. Something doesn't become unscientific just because you happen to think it's trivial.

Re:Flash panic

[stranger_to_himself]

I think the dividing line between when you need to get informed consent is when the experiment begins to make people do things they wouldn't have done anyway. Tweaking how people get paired up for dates is fine if they were looking for a date anyway. Forcing them to go on a date when they weren't planning to would require informed consent (and probably compensation).

Not really - even purely observational academic studies need ethical approval and informed consent. I really am confused about where the diving line should be between academic and commercial work.

Re:Flash panic

[stranger_to_himself]

Do academic demographers get "informed consent" before processing census data? What about crime statistics? Network security incidents?

You are talking about aggregated data which is a bit different. Its a current debate as to whether anonymous routine data should be available to researchers at the individual level without the explicit consent of the people involved. I would say it should be but many argue otherwise.

Re:Flash panic

[AmiMoJo]

According to TFA they correlated certain reactions with attractive photos, which means they must have looked at the photos to determine if they were attractive or not. It isn't clear how far they went into people's private profiles to do this. Maybe it was all innocent, but it seems creepy because they were not transparent about it and probably would have kept quiet if Facebook hadn't been caught.

I know what people will say, they uploaded their photos to a web site and have no expectation of privacy. Well, actually they do. A site like OK Cupid needs to maintain a good image and handle customer data with respect. If you went to a brick and mortar dating agency you would expect them to keep your profile secure and only show more than the front cover to people you authorized them to. You certainly wouldn't expect random members of staff to go through your file without telling you.

Re:Flash panic

[BlueScreenO'Life]

even purely observational academic studies need ethical approval and informed consent.

In what jurisdiction?

That's interesting, because a lot of "purely observational academic studies" have been done with no informed consent at all.

Examples: - Robert Levine's experiments linking a city population's average walking speed with their degree of helpfulness and their health (actually, this is not merely observational - parts of those experiments involved getting people to pick up dropped pens, return lost letters, etc). That was done in many parts of the world including the USA.

- In a car, sitting at a red light. Wait for it to go green, and deliberately fail to move off. Measure how long it takes for drivers behind to honk. Then do the same thing on a car with a foreign plate and compare results. They did that one in Western Europe, if I remember correctly. Unfortunately I can't find a link to that one, but I think I also read about that one on Quirkology.

Neither of those experiments (and many more, those two are just off the top of my head) were done with informed consent. That would have rendered them completely useless, obviously, due to bias.

Re:Flash panic

[EL_mal0]

For me, it's not the testing so much, but in Facebook's case, publishing those results as if the participants had given informed consent. I expect to be subject to usability testing. I don't expect to be the subject of psychological testing. I don't know where the line is, but there's a line somewhere in there that was for sure crossed in the Facebook case. It's less clear to me what side this OKCupid case is on.

Re:Flash panic

[mattwarden]

I think you mean de-identified data, not necessarily aggregated data. But I understand your point. I am not sure that the outrage of the interwebs turns on that, though, as there are plenty of examples of data collection that cannot be tied to a real-life identity that gets their panties in a wad.

Re:Flash panic

[mattwarden]

> which means they must have looked at the photos to determine if they were attractive or not

That is quite an assumption. I can think of a ton of ways they could have an attractiveness measure without themselves digging into people's personal profiles. In fact, I did 5 seconds of googling and found this, which clearly suggests that they are asking other members to rate attractiveness of profile pictures: http://blog.okcupid.com/index....

> I know what people will say, they uploaded their photos to a web site and have no expectation of privacy.

They certainly don't have an expectation of privacy from their photo being seen by other users of the site. That's why they uploaded it! So what's the beef?

Re:Flash panic

[Lennie]

The problem is: usability testing is psychological testing.

With usability testing you are testing what people would do in different situations. That to me sounds like a psychological test.

Re:Flash panic

[stranger_to_himself]

You are right. I just checked my School's ethical research code, and there is an exemption under the 'exceptional circumstance' that taking informed consent is not possible because of a need for concealment. These studies still require ethical review though and the ethics panel will decide whether or not the concealment is justified. I had not encountered this before in my own work and I should not have generalized.

Re:Flash panic

[EL_mal0]

I get that. I just think there's some important difference of degree between, "Let's see if the blue button gets pressed more than the green button," and "Let's see if people become sad if we show them sad posts from their friends." My opinion is that there is a line somewhere there. I just don't know quite where it is.

A/B Testing

[Mr Z]

Isn't that what A/B testing is all about?

Re:A/B Testing

[Mister Liberty]

My manipulation's as good as yours.

Re:A/B Testing

[Bite The Pillow]

People being manipulated, even if the manipulation is only demonstrated by the experiment, without knowing they are being A/B tested, is not what A/B testing is about.

Knowingly participating in an A/B test is kinda part of A/B testing. Is this lens better, or worse? Which of these televisions side by side looks better to you?

You are looking for love, your soulmate, or someone who will put up with your desire to have cough medicine inserted into your rectum by someone dressed as a Teletubby.

You don't find it because you are accustomed to the way people on dating sites work, and these people are not behaving your way. Cough syrup inserting Teletubbies usually post pictures, and no one fitting the description has. Or vice versa, I don't know.

Or, I have been treating this person who did not post a picture differently, even though they did.

This is much more about how people interact with people who post pictures, and misrepresenting people as picture posters or non picture posters. The misrepresentation is NOT part of A/B testing. Knowing that you don't know is part of an A/B test. Not knowing that you don't know is not.

So no, the answer to your question is no.

Re:A/B Testing

[Bite The Pillow]

Fuck me, I made the point using the wrong experiment. Otherwise, the argument still stands. People trust that the number is as correct as the website can be. Given that it doesn't know whether you like Teletubbies putting cough syrup in your ass.

Still, not A/B testing in any but the most ignorant sense of words.

Re:A/B Testing

[Anubis IV]

A/B testing, as a concept, is fine. The issue here is that A was "truth" and B was "deception", and that's something you shouldn't be A/B testing (at least not without getting ethics waivers signed). Facebook provided feeds that were not representative of what was actually going on and OKCupid flipped bad matches to good matches, both of which compromised their relevant services by misleading users or misrepresenting information. You can't do stuff like that in most (all?) ethical systems, and it may even open them up to legal trouble, since they're knowingly providing something other than the promised service.

At the very least, their doing so runs contrary to the categorical imperative, so for any deontological ethicists out there, it should seem pretty apparent that they were out of line. And if you subscribe to more consequentialist ethical thinking, such as utilitarianism (either the Act or Rule variety), it's trivial to point out that the users were going to obviously be worse off in several of these cases and that happiness was not maximized, nor would it be if everyone was misleading their users like this.

Again, A/B testing is a great tool, but it needs to be used ethically.

Re:A/B Testing

[Bing Tsher E]

Fuck me,

*cough* I only had the Actimates Teletubbie long enough to dissect it and investigate the interesting LED array. You should try on Craigs list, I suppose.

Re:A/B Testing

[PlusFiveTroll]

>OKCupid flipped bad matches to good matches

To be fair on this point is there any objective measure on what a good or bad match is? The entire system on OKCupid is made and defined by OKCupid, there is no objectivity. Therefore a good match == bad match == imaginary purple dinosaur. There are plenty of writeups online about just this subject. Human happiness in relationships is not as formulaic as OKC would like you to believe.

The FB thing is definitely more objectively definable. As in many bits of information was posted, only bits of information that fell under content X were shown to other users.

Re:A/B Testing

[Anubis IV]

You're correct that the matches are not objective, though that really doesn't matter in the end. If I say, "I'll make my best guess," and then knowingly provide you with the choice that is as far away from my actual best guess as possible, there's nothing subjective about the fact that I've intentionally misled you. My guesses may be subjective, but you were expecting my best one, and instead got my worst one. That's a lie.

Re:A/B Testing

[Lennie]

What OKCupid is doing: is checking if their algorithm works.

How else then changing variables do you check if it works ?

Re:A/B Testing

[Anubis IV]

There are ways to test the algorithm that don't involve flipping the results entirely. Suggesting, as you have, that this is the only means available to them to do so is a bit disingenuous. Moreover, if you are going to intentionally give people results that you believe to be incorrect, contrary to what you have promised them, you have an ethical responsibility to get their permission in advance via some form of opt-in release.

Re:A/B Testing

[Lennie]

I'm sure it's somewhere in the terms-of-service. ;-)

Re:A/B Testing

[Anubis IV]

As am I. :P

Even so, you get what I mean, and I doubt most people would consider it ethical to have a blanket ToS that covered all use, including experimental uses. Rather, they'd expect to have to do a separate opt-in to a beta or experimental version of the product, I'd wager.

Re:A/B Testing

[Lennie]

I honestly don't know.

They do say: opposites attract. :-)

Really: we don't know how well this online dating thing really works. Isn't really all that clear cut as people make it out to be. They are just guessing. And they know it this. So them trying out different approaches isn't as different as what they normally do as you think.

people are shallow

[asmkm22]

All they did was discover that everyone on a dating site places physical attraction (based on a photo) above everything else by a wide margin. Reported "compatibility" and profile data are largely irrelevant. Basically, Hot or Not should be as effective for online dating as eHarmony.

Re:people are shallow

[Anonymous Coward]

"People are shallow and place too much value on appearances," said the fat, ugly nerd. "They should be like me, and value what's in peoples' souls. For instance, I can tell Jennifer Lawrence and Scarlett Johansson are perfect for me, because they have such beautiful souls."

Re:people are shallow

[Bite The Pillow]

They did not control for people thinking they were similar, and looking deeper than they would have otherwise. They made hypotheses and passed them off as conclusions.

Your conclusion is therefore based only on confirmation bias, not on fact.

Re:people are shallow

[Falos]

Either you're lamenting in monologue or you're trying to shoehorn in some kind of distance to keep yourself safely apart and hopefully polar from a stereotype you're assembling ad hoc. Need to stay distinct to protect that fragile ego.

Happens during vilifying too.

Re:people are shallow

[Bing Tsher E]

Possibly they're just lampooning, though, which renders your socio-psych judgment into something kinda silly. And which points to the inquiry: what provoked your rather strong response?

Re:what?

[nospam007]

"No. It's what some unethical douche bags do."

There are ethical douche-bags?

OKC started as a science project

[slaker]

The people who run OKC were a bunch of statistics nerds. It runs (ran, anyway) on a custom web server that performs a lot of real time analysis. Their blog is chock full of incredibly detailed information about their users. This shouldn't be news to anyone who has even the slightest clue as to how OKCupid actually works.

Re: OKC started as a science project

[Anonymous Coward]

OKCupid was sold to the owners of match.com a long time ago. When that happened the best blog post that they wrote (Why you shouldn't pay for online dating) was taken down, and the blog itself hasn't been updated (not counting the entry this articles about) since April of 2011.

Re: OKC started as a science project

[Anonymous Coward]

Here you go:
https://web.archive.org/web/20...
Yeah the blog title sais everything: Communist Inc, which doesn't want to make money, gets swallowed by Monster money Corp, which makes money.

Re:what?

[thieh]

Thsi is how the internet works, not just web sites

Re:what?

[vux984]

No. It's what some unethical douche bags do. it has nothing to do with how websites work, asshole.

Anyone who has ever:
a) taken any metrics about there site
a) altered their website in any way
b) measured whether or not it made any difference

Change the font? Rewriting the sales pitch? Moving the photo to the left? Changing the checkout sequence? Showing more or fewer related products? Added bitcoin as a payment option? Offered a discount? Let you checkout without registering? Adjusted your online advertising budget or changed the keywords you were paying for or targeted a new demographic or region...

Do any or all of those one at a time, checking whether sales increased or not... congrats you effectively "experimented" on your users.

Whether or not it is insidious or unethical doesn't depend on "did you or did you not experiment" it depends on what EXACTLY you've been doing.

Me, I've noticed that people tend to click on articles that are finite lists of things. Hypothetically take an article called "Retirement Savings Strategies Everyone should know" gets fewer clicks than "7 retirement savings strategies everyone should know".

The only change is the addition of the number 7.

The internet has gradually been replaced by "X Y's" articles, because it gets more clicks, as this has become increasingly "discovered" by people "experimenting" on users with different headline styles.

The only upside is that I can safely ignore any "news" site with more than 1 article that starts with a count in the title, as containing nothing more than processed brain diarrhea.

OKC's match algos suck

[thisisauniqueid]

Findings include that ... suggesting a bad match is a good match causes people to converse nearly as much as ideal matches would.

All this means is that OKC's match algorithms suck: there's only a weak correlation between match scores and real-world compatibility (like with every other dating site).

Re:OKC's match algos suck

[penguinoid]

Findings include that ... suggesting a bad match is a good match causes people to converse nearly as much as ideal matches would.

All this means is that OKC's match algorithms suck: there's only a weak correlation between match scores and real-world compatibility (like with every other dating site).

No, it means that:
1) People trust OKCupid's rating system enough to try harder when it suggests a good match
2) OKCupid has to take into account their stated match rating, not just length of conversation, when trying to use conversation length as data to improve their algorithm.

Re:OKC's match algos suck

[Impy the Impiuos Imp]

It's called the "tyrrany of dimensions". The more variables you have, the more data points you need exponentially to derive meaningful partitioning analysis from it, regardless of how clever your distance algorithms are.

And they have hundreds of questions when a dozen would be about all the entire population of Earth could support.

Re:OKC's match algos suck

[penguinoid]

edit: 1) Users of OKCupid trust OKCupid's rating system enough to try harder when it suggests a good match

Re:OKC's match algos suck

[wvmarle]

Well, of course. They're digging too deep in that. In the real world, I believe that there is no such thing as "the one" or "the perfect match". Maybe it feels like it, but that's in part thanks to the "pink glasses" effect of being in love and because both parties tend to adopt to one another, especially when a relationship lasts long (years, decades).

People probably can form lasting romantic relationships with a large number of other people, after the following basic matches are followed (assuming heterosexual relations but some will apply for homosexual relations too):

• Geographic proximity.
• Speaking the same language, or at least share a second language.
• Similar age, preferably the female 1-5 years younger than the male.
• Similar educational level, or the male having higher education than the female.
• Similar political/religious views (left/right wing, Muslim/Christian/Buddhist/etc).

The above are true for the vast majority of heterosexual relationships. Another major factor in partner choice is also the availability of the person, as in, that s/he is not in another relationship already. The fact that someone is active on sites like OKCupid fulfils that requirement. Coincidence plays a great role as well: whether you meet a person now (when he's single and looking) or in half year (when he's just got a girlfriend). Whether you meet the person at all. He may be a perfect match for you on all fronts, yet unobtainable due to living 1,000 km away.

Re:OKC's match algos suck

[Too Much Noise]

It's called the "tyrrany of dimensions". The more variables you have, the more data points you need exponentially to derive meaningful partitioning analysis from it, regardless of how clever your distance algorithms are.

Indeed, but only if you insist on carrying along in your analysis all the irrelevant and correlated dimensions.

And they have hundreds of questions when a dozen would be about all the entire population of Earth could support.

So do surveys, for significantly smaller sample sizes. I wouldn't be surprised if a non-trivial percentage of those questions are intentionally redundant - you know, to check *ahem* consistency, improve accuracy, etc. If, say, you have 100 questions grouped into 10 categories with 10q/cat, you have just dropped the dimensionality significantly while at the same time having more confidence in your data. A rule of thumb in surveys is don't trust the user^W^W^W^W *ahem* trust, but verify.

Ok Cupid....

[bobbied]

That was STUPID!

Conduct your little experiment if you have to, just keep your mouth shut about it.... At least until you have notified ALL your users that such experiments *might* be taking place (Or if you intend to issue refunds from the resulting class action suit.)

Re:Ok Cupid....

[penguinoid]

As opposed to just randomly matching people so as to not have to learn things about people in general nor their users in particular?

Re:Ok Cupid....

[joe_frisch]

The particularly stupid part was messing with their match algorithm. If they imply that their algorithm has any value, then their users will feel at least ripped off (since the algorithm doesn't seem to work well), and possibly angry because they were given incorrect information .

Blocking pictures was visible to users and I don't have any problem with that .

Re:Ok Cupid....

[PlusFiveTroll]

>The particularly stupid part was messing with their match algorithm.

You are making a false assumption because you are dealing with a biological system.

If you made a screen to sift sand, that screen will reliably sift sand of a certain size because they sand has no choice in the matter and does not evolve.

On the other hand if you make an antibiotic that kills bacteria X you will quickly find out that in just a few generations almost all of bacteria are resistant to the antibiotic.

Culture evolves, religious views change, human relationship standards evolve. 50 years ago it was probably a bad idea to pair a black guy and white woman, not so much so these days (in most places). Saying that a sorting process is going to stand up to that change, when that sorting process itself feeds back in to the system is pretty unrealistic.

Re: what?

[Anonymous Coward]

Yes, they're divorce attorneys.

Re:what?

[Bing Tsher E]

There are ethical douche-bags?

Believe it or not, there are FDA approved douche-bags which are produced by ethical companies, and sold by ethical retailers.

This is different from what Facebook did

[nikhilhs]

FB's experiment was to see if they could alter the mood of their user. OKC tried to see if they could get more conversations going. Intent matters. OKC's is fairly harmless. FB's experiment could have a ripple effect and cause negative consequences.

Re:This is different from what Facebook did

[thegarbz]

FB's experiment was to see if they could alter the mood of their user.

They should have hired the DICE Beta department.

Re:This is different from what Facebook did

[IamTheRealMike]

Er, why? Facebook presumably wants users to feel happy when they visit their website - exploring how to do that doesn't seem substantially different to OKCupid wanting to make lots of successful relationships.

Re:This is different from what Facebook did

[csha]

Actually, Facebook did the exact opposite. They specifically removed positive news from people's feeds to produce a negative affect. They were literally trying to make people unhappy. Also, "facebook wants users to feel happy" is not true. There's already been research showing that Facebook tends to make people less happy, and that less happy people spend more time on Facebook. Knowing this, it's in Facebook's interested to ensure people are unhappy.

Re:This is different from what Facebook did

[phorm]

Moreover: OKC's experiment is related to improving the user experience. FB's... not so much.

Huge differences

[jimmyswimmy]

There's a huge difference between A/B testing, designed to optimize your website with the direct intent to improve sales, and performing experiments on how different news feeds affect your users' moods. A/B testing typically comprises changes in button size and color, website layout, font variations, etc; should we lead with the price, or with the benefits, or with something else? On the other hand, what FB and OKC are doing - admitting to, and proudly! - amounts to wholesale experimentation on their users, with undisclosed intent - perhaps to make the users come back more frequently for another hit.

This seems akin to me to cigarette companies manipulating the nicotine content of their products. That didn't go over well when it was finally disclosed.

You can't just tell people you "might" experiment with them, they have to know and understand that they are part of an experiment. They don't have to understand the goal, they just need to know what they are part of, and they have to consent to that experimentation. One could argue that A/B testing should submit to the same level of scrutiny as other psychological experiments, but I think people generally understand and accept corporations' profit incentive. We don't accept the idea that a company might wish to screw around with our mood or set us up on a date when they know it won't work out.

Marketing

[Bo'Bob'O]

The fact is that the experiment they Facebook conducted was mild to what other corporations do every day under the umbrella of "marketing".

They use control groups and try every trick they can to manipulate your mood, feelings, impressions of their products. They carefully script interactions to take advantage of your feelings and social norms. Also take the recent example in the past few weeks of the scripts that Verizon's 'account retention' departments use to try and wedge people into keeping their account longer. Those weren't just thrown together, those were made with careful research and years of experiments on customers and focus groups.

The only difference with what Facebook did and the rest do is that they shared their results with everyone. Was Facebook Unethical manipulating people the way they did? I think so, and I'm only less interested in the service after that scandal, but what they got them in trouble was sharing it with the rest of the world in a way that might have also done some honest good. Now they will learn from their mistakes, keep it to themselves, and use that research purely to manipulate people for higher profit and no one will say a thing.

Re:Marketing

[DNS-and-BIND]

There are three professions where being untruthful is the key to success: Lawyers, salespeople, and marketing. All three are hired to portray their client in the most favorable light possible, and the very best ones lie through their teeth. The worst of these three are the marketers because they have legions of psychologists and scientists trying to figure out the best way to lie to people.

Re:what?

[geekoid]

Yes.
I know a physicist who is ethical, but when he speaks outside his expertise, he becomes a douche. Really irritating.

Re:what?

[geekoid]

Since douching is harmful and not needed, I would disagree with your statement.

Re:what?

[geekoid]

"congrats you effectively "experimented" on your users."

That's not experimenting. generic research on overall performance is not the same as selecting a sample of users, and conducting tests on them specifically to change their response.

You're definition is so loose it's useless.
Now, I need to experiment ans see if I can get home from work.

Your grocery store experiments on you ...

[perpenso]

That probably depends upon whether you consider the terms of use of the online service, grocery store loyalty card, casino player's card, etc to be transparent. Those terms of use that no one reads.

There is also consent by action. The casino does A/B testing by offering some a $40 steak dinner plus $40 in chips while it offers others $80 in chips. You clicked on the advertisement/offer, or you opened the envelope that arrived in your postal mail, etc.

Similarly the coupons a grocery store offers you are often part of an experiment. Hell, changing the items on the isle end caps are sometimes part of an experiment.

My marketing processor thought that grocery store loyalty cards were the greatest invention ever in the history of marketing. The data collected and opportunity for experiments enormous.

Re:Your grocery store experiments on you ...

[mwvdlee]

One of those standards involves informed consent.

Which instantly makes any kind of unbiassed behavioral research impossible.

Re:Your grocery store experiments on you ...

[Xest]

"There is also consent by action. The casino does A/B testing by offering some a $40 steak dinner plus $40 in chips while it offers others $80 in chips. You clicked on the advertisement/offer, or you opened the envelope that arrived in your postal mail, etc."

Well I think this is the difference, when you sign up to OKCupid you're signing up to a service that's explicitly designed to optimise your chance of meeting someone, so almost by definition you're going to expect them to play around with your profile, their matching algorithm and so forth to optimise that goal.

I'm pretty sure however that no one signing up to Facebook did so with the belief that Facebook would try and play with their emotions to make them unhappy.

Re:Your grocery store experiments on you ...

[Anonymous Coward]

There is also consent by action. The casino does A/B testing by offering some a $40 steak dinner plus $40 in chips while it offers others $80 in chips. You clicked on the advertisement/offer, or you opened the envelope that arrived in your postal mail, etc.

In the academic world, we have decided that "consent by action" is not sufficient. You have to tell the (potential) volunteer what information you're hoping to learn from their participation, you have to explain what the benefits to society and the possible risk to themselves is. In the academic world, you would have to explain to a (potential) subject that you hoped to evaluate the relative importance of food and money before you even show them one of the advertisements. You would have to explain that accepting "free" gambling chips carries a risk of gambling addiction, that a steak dinner carries a very small risk of asphyxiation, and that exposure to a colorful, flashing image carries a small risk of inducing seizure in specially sensitive groups. Only after these risks, and the rationale for the "study" had been understood and accepted by the (potential) subject, could you then display one of the two advertisements and see which garnered more clicks.

All of this stuff you call "A/B testing" would be completely unacceptable in an academic setting. The results should be unpublishable in any journal that requires compliance with the Declaration of Helsinki. Unless: that data had been collected for commercial business purposes, and only after it was used for such commerce were researchers allowed, post hoc, to mine the data. That's totally within the bounds ethical research.

Re:Your grocery store experiments on you ...

[tomhath]

That probably depends upon whether you consider the terms of use

I use depends, you insensitive clod.

Re:Your grocery store experiments on you ...

[perpenso]

That probably depends upon whether you consider the terms of use

I use depends, you insensitive clod.

We know, we have your supermarket data. :-)

Re:Your grocery store experiments on you ...

[phorm]

Not really. I'm sure many companies just bury it in the ToS. The only bias there is towards the type of people that
a) Use the service
b) Don't read ToS's (or don't care).

Re:Your grocery store experiments on you ...

[stranger_to_himself]

... and your testing is useless, because you've just tainted the results by revealing far too much about the scenario. All of your subjects now act differently knowing they're being judged on their responses.

In other words, academics are jealous that real companies get to do more science than they do.

Ha! Actually yes I am jealous that Google has better health data than we do and seems to be allowed to use it in ways that would not be 'ethical' or even legal for academic health scientists.

Re:At least it isn't reddit or Hacker News tyranny

[cavreader]

It's obvious you do not have a clue about what real "censorship" is. So a website rejects posts that do not meet their basic and usually very low standards you agree to when posting there, BFD. On the other hand under real censorship the site would not even exist in the first place and if you tried to start one in some countries you would have state security knocking on your door.

Re:Shallow people will be shallow

[viperidaenz]

or perhaps all people are shallow.

Re:what?

[vux984]

selecting a sample of users, and conducting tests on them specifically to change their response.

How does changing something about your website to get them to spend more money not qualify as "selecting a sample of users, conduncting tests specifically to change their response"?

So what if the 'sample of users' is everyone, and the A/B test occurs over the same users in two non-overlapping timeframes? If I make the changes to my regionalized .CA website to test the impact on "Canada" before making it to the global site? Does that qualify? Because pretty much all sites do that sort of thing too.

You're definition is so loose it's useless.

That is PRECISELY my point. Getting in a huff about "experimenting on users" is absurd, because the definitions in play ARE uselessly broad. What did OKcupid or facebook do, SPECIFICALLY, that crossed a line that any other website wouldn't do to increase whatever metric they were looking at.

Because, to my view, they haven't done anything different from any other site, at all. So this all really is much ado about nothing.

Dating sites simply lie.

[Kris_J]

Try this the next time you want to try an online dating site: Create two profiles, a "real" one and a fake perfect match to your real profile and see how long it takes for the site to claim that your fake perfect match has attempted to contact you and for only $4.95 you can sign on to the paid service and reply.

Re:Dating sites simply lie.

[Ksevio]

Funny you should mention that since this very blog had a length article discussing that topic. It's gone now that match.com bought them out, but you can still find it in archives.

Re:Dating sites simply lie.

[Anonymous Coward]

That's not actually how okcupid works. You aren't charged for talking to your perfect match. You are charged if you go over the limit of 100 messages and don't want to delete any. And, you are charged if you want to remove advertisements. If you use adblocker their ads actually ask you to turn it off for their site because they get most of their revenue that way.

Re:Dating sites simply lie.

[Cammi]

You forgot to link the blog ....

Re:Dating sites simply lie.

[Ksevio]

The blog is in the summary...

Re:what?

[jxander]

I see your point, but I can't equate a change of font or layout with lying about compatibility on a dating site.

Especially without any kind of warning to the users in question. Double especially when it's a potentially-paid service (a quuick google search says they OKCupid has both free and paid options... though I've never used the site myself)

On the flip side, if people knew that such shenanigans were afoot, we probably wouldn't get any decent results. Still, it seems like there should at least be a "we are altering our algorithms regularly to try and optimize the compatibility... blah blah blah" stated fairly clearly when you sign up.

This is outrageous

[mysidia]

I think Firefox should boycott the site.... display a message about it being possibly malicious/dangerous to all users attempting to visit OKCupid, showing a link to the article as a warning message in bright red... (Just kidding <EG>).

Re:This is outrageous

[Bing Tsher E]

The problem with that is, the folks who remain now at Firefox bought into the same sub-ethical slime ethos. They were basically part of the same chorus on that earlier issue as these fine folks at OKCupid. Those who would enjoy the irony are gone or at least thoroughly stifled within the present organization.

Re:Shallow people will be shallow

[ColdWetDog]

"A walk through the ocean of most souls would scarcely get your feet wet".
          - Deteriorata

Re:At least it isn't reddit or Hacker News tyranny

[OhPlz]

Do you think the donation information was leaked accidentally? If you do, you probably think that the IRS also accidentally lost months worth of email that may have contained evidence pertaining to that agency's targeting of conservative groups during an election season. It won't be state security knocking on your door, but the feds are still going to arrange for someone to come after you.

"That’s how websites work."

[Cammi]

"That’s how websites work." Whoa so OKCupid was retarded enough to hire someone who have NEVER, EVER been on the internet? Some heads need to roll ...

business is all about experimentation

[mattwarden]

very few successful businesses are doing what they were originally founded to do. business is all about experimentation. you tweak and reset and change and reset again until you see the numbers going in the right direction at the desired speed. unsuccessful businesses usually do the same thing, too; they just don't ever find a combination that works.

see also: "Getting to Plan B: Breaking Through to a Better Business Model" http://www.amazon.com/Getting-...

"To succeed, you must change the plan in real time as the inevitable challenges arise. In fact, studies show that entrepreneurs who stick slavishly to their Plan A stand a greater chance of failing-and that many successful businesses barely resemble their founders' original idea. ... Testing those assumptions and unearthing why the plan might not work."

Re:At least it isn't reddit or Hacker News tyranny

[cavreader]

If someone is secretly manipulating or shaping information to push a preferred outcome it first needs to be secret to have any true effects. Without the secrecy you are free to evaluate the posted information with the knowledge that someone is trying to influence your opinion by excluding certain pieces of information or posts in this particular case. If you recognize this pattern you are free to go to another source for information. Unfortunately there are far to many news outlets or websites pushing their own agendas and partisan editorial lines instead of facts. A lot of folks can not recognize fact from opinion and tend to gravitate towards sources that publish information that validates their pre-determined opinion while ignoring any information that contradicts their stated opinion. You have the far right and far left and everything in between supposedly reporting on or describing the same thing but the information they publish turns out looking like the people providing the information all live in their own little universe. Web forums are notorious echo chambers where facts tend to get in the way. "Winning" the argument comes before facts. Most popular news sources and web sources are becoming adept at using "lies of omission" to shape their stories. This allows them to state that everything they published was factually correct which in a sense would be true but the information omitted could have put a whole different slant on the argument.

What next?

[AlCapwn]

OkCupid will start pairing everyone with the worst possible match. Once and for all, we'll be able to prove that opposites attract!

Re:Enough already!

[Opportunist]

Yeah, the good ol' times. When it was other users that trolled you, not the page owners themselves...

Experimentations or shady practices?

[ruir]

I have a completely different opinion of online dating services at all. When you are trying them in the trial period, which of trial is a waste of time, as just it lets you browse the public profiles and receive messages, you are most certain to receive one or two messages, often in english, no matter what your mother tongue, of someone VERY INTERESTED in meeting you, just to make sure you sign up for the service. Do those people think we are dumb?

Facebook evil, OKC less bad experiments

[billstewart]

Facebook's experiments bother me more than OKCupid's. They're deliberately manipulating which news stories their readers see in order to affect their mood, and seeing how that affects the readers' behavior. That seems mean and dishonest. (Of course, I didn't know Facebook had news, so I'm not in their target market anyway, but it still seems mean.)

OKCupid's a dating site, which means that all their "compatibility" scores are pretty much guesswork anyway, assisted by a lot of measurement, so an occasional suggestion of "maybe you two should see if you want to date" to people they normally wouldn't match up isn't that much perturbation of their approach anyway, and "whoops, pictures are broken, why don't you try talking first instead of just looking at pictures" is just fine, and both of them give them a bit of data outside the ranges they'd normally be collecting from - perhaps there are people that would get along well who they haven't been matching up. (I'm not in their market either, fortunately.)

Re:Facebook evil, OKC less bad experiments

[stranger_to_himself]

Facebook's experiments bother me more than OKCupid's. They're deliberately manipulating which news stories their readers see in order to affect their mood, and seeing how that affects the readers' behavior. That seems mean and dishonest. (Of course, I didn't know Facebook had news, so I'm not in their target market anyway, but it still seems mean.)

I agree that the Facebook incident is much more worrying than what OKCupid is doing.

OKCupid's a dating site, which means that all their "compatibility" scores are pretty much guesswork anyway, assisted by a lot of measurement, so an occasional suggestion of "maybe you two should see if you want to date" to people they normally wouldn't match up isn't that much perturbation of their approach anyway, and "whoops, pictures are broken, why don't you try talking first instead of just looking at pictures" is just fine, and both of them give them a bit of data outside the ranges they'd normally be collecting from - perhaps there are people that would get along well who they haven't been matching up. (I'm not in their market either, fortunately.)

Your post raises another interesting (and IMO ridiculous) issue though, which is that just because providing either of two different services (either pictures or no pictures) to all of your clients is perfectly fine - randomizing people to receiving one or the other is often considered not to be ethically sound.

Re:Enough already!

[PlusFiveTroll]

You must not have been online during those times, as SYSOP I trolled the living shit out of people I didn't like.

Re:what?

[PlusFiveTroll]

>lying about compatibility on a dating site.

Here's the gist of it, they already were lying about compatibility, or at least what you think of as compatibility. Different cultures have distinctly different criteria for selecting mates and it evolves over time. There is no golden rule, no algorithm, no magic. They throw a bunch of different shit at the wall and see what sticks. Why they look so good at finding matches is not actually finding matches but weeding the unmatchable out. Take them out, and most other people can date a pretty wide range of other people with just a few points of similarity.

The fact you don't think that their matching changes over time boggles my mind. Culture evolves and changes, technology evolves and changes, communication evolves and changes, to think some kind of static algorithm could possibly work at matching people under those influence is insanity.

Re:what?

[mwvdlee]

I guess somebody who points out something that is technically correct but unnecessary to point out, could be called an "ethical douche-bag".

Old news

[Anonymous Coward]

OKC have always done this kind of thing: for some time they've had articles about what they've found by datamining their users. The difference is they've been pretty open about it and the stuff they publish is interesting and helpful to users, so people are OK with it.

But the same will be unavoidably true of any internet company that uses algorithms. Those algorithms aren't handed down by God on tablets of stone, they don't appear by magic. Companies have to develop them and improve them, and that will necessarily involve bringing algorithms out of the development house and taking them live, and then seeing how they work out.

Re:what?

[gl4ss]

it's not really beta stuff though.

what they do is serve a portion of the visitors version A.

then they serve a portion of the visitors version B. then they see in which one resulted in more sales/longer engagement.

neither one is technically beta, but more like an experiment about what works to get you to do the wanted outcome.

there's plenty of tracking/logging companies providing easy facilities this and if you work in startups you'll see it pushed on your face, even if the a and b versions don't have enough difference and the sample sizes are too small to make any conclusions on. but it gives people to do something when they don't know how they should further the sales/engagement and it's on the "book" of marketing.

if you're not paying for it...

[Cardoor]

if you're not paying for it... you're the product.
an oldie, but goodie.

No, it isn't and they don't

[jd]

The Internet is not powered by experiments on humans. Not even in the DARPA days.

No, websites do NOT experiment on users. Users may experiment on websites, if there's customization, but the rules for good design have not changed either in the past 30 years or the past 3,000. And, to judge from how humans organized carvings and paintings, not the past 30,000 either.

To say that websites experiment on people is tripe. Mouldy tripe. Websites may offer experimental views, surveys on what works, log analysis, etc, but these are statistical experiments on depersonalized aggregate data. Not people.

Experiments on people, especially without consent, is vulgar and wrong. It also doesn't help the website, because knowing what happens doesn't tell you why. Early experiments in AI are littered with extraordinarily bad results for this reason. Assuming you know why, assuming you can casually sketch in the cause merely by knowing one specific effect, is insanity.

Look, I will spell it out to these guys. Stop playing Sherlock Holmes, you only end up looking like Lestrade. Sir Conan Doyle's fictional hero used recursive subdivision, a technique Real Geeks use all the time for everything from decision trees to searching lists. Isolating single factors isn't subdivision because there isn't a single ordered space to subdivide. Scientists mask, yes, but only when dealing with single ordered spaces, and only AFTER producing a hypothesis. And if it involves research on humans, also after filling out a bloody great load of paperwork.

I flat-out refuse to use any website tainted with such puerile nonsense, insofar as I know it to have occurred. No matter how valuable that site may have been, it cannot remain valuable if it is driven by pseudoscience. There's also the matter of respect. If you don't respect me, why should I store any data with you? I can probably do better than most sites out there over a coffee break, so what's in it for me? What's so valuable that I should tolerate being second-class? It had better be damn good.

I'll take a temporary hit on what I can do, if it safeguards my absolute, unconditional control over my virtual persona. And temporary is all it would ever be. There's very little that's truly exclusive and even less that's exclusive and interesting.

The same is true of all users. We don't need any specific website, websites need us. We dictate our own limits, we dictate what safeguards are minimal, we dictate how far a site owner can go. Websites serve their users. They exist only to serve. And unlike with a certain elite class in the Dune series, that's actually true and enforceable.

Just Ask!

[ememisya]

Firstly, I'm not convinced Facebook can be compared to OKCupid as far as their website identity is concerned.

I think the problem lies in the amount of uninformed experimentation in a given time period and how it effects the individual.

To some people, a barking dog at night is mental torture, or shining a light in someone's house really bothers them. Others can easily ignore lights and dogs and have a great night's sleep. It becomes an issue when a single individual who is bothered by it has to deal with 4 constant lights, 12 constantly barking dogs, and 1 person following you around scratching their nails on a chalk board all day (Got the phone app?). None of these sites bother to ask if anything bothers the user before testing it on them. Imagine the shittiest day you ever had, relative(s) died, lost that business deal, divorced all on the same day and say you also sat on your cat and smushed it.

You sit in front of Facebook and it starts reciting, "Nevermoore!" you might want to kill yourself, and Facebook would have been the last straw.

People don't expect such behavior from websites, because they misrepresent themselves. They keep these interactions as secrets, even porn sites have the decency to ask you, "Do you like man on man action?" before shoving a dong picture in front of your face.

Re:Shallow people will be shallow

[secondhand_Buddah]

People who are perceived to be attractive have an evolutionary advantage, because people are attracted to them
So for your genes to survive, it is better that you find an attractive mate.
Over time the traits we find unattractive will disappear from the gene pool.

Apples and oranges

[Shirley Marquez]

OK Cupid's experiments don't bother me as much as Facebook's do because online dating is already a big social experiment.

Re:Shallow people will be shallow

[peterantonrev]

Consider that complex human beings are trying to relate to one another on a machine with limited multiple choice descriptions. I much prefer face to face interactions where I can find them. And not use a machine to do that.

Re:Shallow people will be shallow

[viperidaenz]

So why are there still so many ugly people?

Re:Shallow people will be shallow

[secondhand_Buddah]

The norm shifts over time. Our perception of ugly is very different to the ugly of 500 years ago.