Microsoft Tracking Behavior of Newsgroup Posters

theodp writes "Ever get the feeling your Usenet newsgroup list is being watched? By Microsoft? If so, consider yourself right. An interesting but troubling CNET interview with Microsoft's in-house sociologist goes into how the software giant is keeping a close eye on newsgroups and other public e-mail lists, tracking and rating contributors' social habits and determining "people who the system has shown to have value." Those concerned that it's not a good idea for computers to track their belongings and whereabouts are advised that they may ultimately have to fragment their identities, keeping multiple IDs and e-mail addresses."

Comments

What's wrong with this?

[Eric Ass Raymond]

Did you guys really think that Microsoft's not profiling the Slashdot users, or the Linux kernel contributors or anyone they deem as a valuable target?

My god, you are so naive.

Re:What's wrong with this?

[crazyphilman]

It's not just Microsoft. There are companies called "clipping services" which have been around for decades. They employ warehouses full of people, reading newspapers from all around the country, clipping out articles that relate to a client or set of clients. The client would get a notice mailed to him/her including the clipping and some time/date/source info. Just about every newspaper in the country is monitored.

Over the years, the clipping services expanded out, adding AP feeds, Newswire, etc. They suck articles right off the wire and store them for their customer's perusal. Then they added newsgroup and chatroom monitoring, and of course web monitoring. They use web spiders to capture the info, and databases to store it.

This is very widespread, and it's been going on for years. Do a google search on "clipping services" if you don't believe me. Anything you write online about a company who can afford the service is noticed, printed out, and sent to them by a flunky.

Why worry? It's not like you can do anything about it. So Microsoft knows that I think they suck. Big deal. Hey, Gates! You suck! See? No hitmen busting down my door, no guidos breaking my leggaggdafsafal;nfdasl'(MESSAGE TERMINATED)

I read the article!

[teamhasnoi]

It sounds like interesting and useful tech for USENET, but there is the question of MS doing it. I'd have far less reservations about it if Google was behind it.

The AURA just sounds like the CueCat Digital Convergence people who wanted to put a bar code on everything. Again, MS is not the company I'd like to see doing this.

*Rather Offtopic - but Digital Convergence used to advertise the CueCat with an 'Angel coming down to earth from heaven to barcode everything' and the well-known Digital Angel RFID people have also made a religious reference in the company's name. The hue and cry of Christian's 'the number of the beast' references beg the question:

Who the hell is doing marketing for these people? I remember getting an icky feeling when I saw the 'infomercial' for the CueCat, and similarly the Digital Angel website. And I'm not the 'churchy' type. I can only imagine what the fundies think...

* This idea is copyrighted. Use of this idea may not be used to more attractively market 'evil' technology, or put a chip in my head. Thanks.

Re:I read the article!

[TheGrayArea]

A big reason for this type of research in MS is to push the community support model. If MS can create a scenario where many questions get answered in a community model like newsgroups by unpaid volunteers/posters, it lowers the overall cost of product support for MS. Newsgroup support is becoming a big thing around Microsoft Product Support. There are actually engineers whose sole job is to monitor and respond to newsgroup postings.
It's all about support costs. Supporting newsgroups is very cheap and also very easy to farm out overseas to folks who really do nothing but paste in answers from scripts.

On the Internet ...

[burgburgburg]

Microsoft knows you're a dog.

Woof.

so what?

[acvh]

Don't tell me that you post on Usenet and expect those posts to be "private"! Give me a break. If ANYONE wants to read and study how people interact on this most public of forums, I fail to see how anyone can object.

Since the early days of netnews...

[sphealey]

Since the early days of netnews (now Usenet) is has been fairly clear that everything you post is being saved, and anything you post if fair game to be responded to, analyzed, and/or held against you at a later date. If this disturbs you, don't post in public forums.

And if Microsoft weren't doing this, wouldn't there be articles appearing with titles such as "Microsoft ignores valuable customer feedback available free on Internet"? I am no big fan of Redmond, but I think they are almost forced to do something like this to avoid being blindsided.

sPh

Slashdot Karma or Google PageRank

[peterdaly]

So it's like Karma on Slashdot, but on a more stealth level, like Google PageRank.

It's more like a Google PageRank implemented Newsgroup posters instead of Web Sites, and run by Microsoft instead of Google. Microsoft is just adding true statistics and tracking to the already existant "human credibility" of posters.

Newgroups posts are public.

I don't see this as a problem.

-Pete

Paranoia

[heironymouscoward]

Is a suitable state of mind when large and powerful groups decide they want to spy on you.

I'm sure MS already spies on Slashdot and tracks every profile here. I have four, and switch between them carefully, unt sometimez I speek in forin lanjuajes just to confooze them.

On the other hand, this reminds me strangely of a scene from Dilbert.

Serf1: Boss, I need to monitor newsnet.
PHB: why?
Serf1: So we can track our competitors, manipulate public opinion, and run smear campaigns against political opponents.
PHB: sounds fine...
Serf1: It will take nine months, that's ok?
PHB: yes, get someone to help you if you need it.

later..

Serf2: So, did you get it?
Serf1: Yes, we're now official newsnet spies.
Serf2: porn on, dude!!! alt.binaries, here I come.
Serf1: I've asked for some new hard drives too... :)

Troubling?

[110010001000]

How is this "troubling"? They are researching a way to make USENET and such more effective. They aren't interested in the fact that cmdrtaco@slashdot.org posted to alt.sex.unicorns 10 times last month.

This is good valid research, the type that applied research CS programs should be doing. Thismay actually make a difference in a deployed product.

I think we should tone done the M$ and SCO crap for a while.

SCO IP Infringer List

[McBride, Darl]

Here at SCO, we've been doing something similar to this for months. We've been tracking user comments on slashdot to compile an extensive list of Linux zealots to go after once our lawsuit against IBM is successful.

Bide your time well, Linux zealots, for the mighty power of SCO's IP will reign down upon thee!

No problem

[smatt-man]

I figure we have nothing to worry about. If Microsoft wrote the tracking software, then it probably doesn't work anyway.

Give it a break

[The Bungi]

• This monitoring goes on exclusively in the msnews.microsoft.com domain, plus a few others that are also run by the company. While NetScan is sometimes pointed to MS-oriented news servers (news.devx.com is an example), Microsoft is not "monitoring USENET".
• Marc Smith is a very sharp guy who has done a lot of interesting work with the social dynamics of online communities. Goggle him for more info. And if you have questions about what NetScan does, give it a whirl and form your own conclusions.
• At the moment, NetScan is used by the MVP program to follow members' posting history. The MVP program is not exclusive to NNTP, however.
• I can't see how this goes into the "YRO" section - if Microsoft is monitoring the news servers it operates and that bothers you - don't post there. This is hardly the land of the Microsoft advocate or even user for that matter. This is like reporting that I'm painting my bedroom bright red - WTF do the neighbors care about that?

Yet another hysterical ad revenue generating headline, brought to you by the Slashdot "editors".

I've seen some of the tracking ...

[galego]

The University of Maryland HCIL had a conference earlier this year. I was in the same hand-held session this guy was in. I checked his picture, and that's him for sure.

The next day he was showing Ben Schneiderman some of this stuff at the open house. A bunch of us looked on as they chatted, planned visits, golf outings and talked about how it all worked.

Depending on the queries he gave it, this one program would chew through data from usenet. and give back all kinds of stats and then draw relationships It even did graphical representaitons of users' actvity. Density of posts in a single thread versus starting new threads, frequency of posts, replies vs. new messages etc would be denoted by distance from the main timeline, darkness and width of the circel and so forth. You would look at a wide but faint circle and say (and I may be off in how the key worked, but ...) "This guy sticks to the topic over a long period of time" or you could denote the flame warrior or the vagrant by their graphical representation and so forth. The way the data was processed was really cool and how quickly you could start to decipher the keys was really interesting.

The Big brother implications ... well that's a whole 'nother thing there too isn't it?

Pay attention kids....

[NightSpots]

Now that they've confirmed that they do this, there's only one thing to remember:

Before you fly up to Redmond for your interview, make sure you post a year's worth of insightful commentary on major relevant newsgroups, with your name and email attached

Re:Multiple addresses wont work

[pclminion]

Bayesian analysis can match writers to messages regardless of the email address.

You just pulled that out of your ass, and you know it. There are so many gigantic misunderstandings underlying that statement that I can't even begin to attack it, so suffice it to say, a simple Bayesian analysis more than likely cannot identify people based solely on what they write.

Ok, I'll give you a hint. Suppose we apply this method to Slashdot. There are about 650000 Slashdot readers. You are talking about calculating the class-conditional probability for every user on Slashdot. The differences in class-conditional probability (per user) are going to unbelievably small -- so small that any results you achieve are going to be statistically meaningless.

Bayesian techniques work okay for classifying when you've only got two or three buckets. But when you try to apply it to say, thirty buckets (much less 650000!!) it breaks down really quickly.

Also, remember that the true name for the technique is "Naive Bayesian inference." In this case (heh, in most cases) the term "naive" doesn't mean "clever and infallible."

Yes, I do research on text analysis algorithms with applications to anti-spam filters, so I do have some clue what I'm talking about.

Re:Huh?

[OneIsNotPrime]

Yes, who among us doesn't already keep multiple Slashdot accounts to mod ourselves up as "Insightful" every once in a while?

Err... never mind.

Well I think I deserve it.

I don't

[Barlo_Mung_42]

I am Barlo Mung. Barlo Mung is me. It's my email address. It's my counter strike nic.
I'm not going to pretend to be anyone else.
Want to dredge up all the postings I've made anywhere on the internet? Go ahead. WTF do I care. If I didn't want people to read it and know it came from me, Barlo, I would not have posted it.

Re:Huh?

[gregmac]

get your own domain and your own email server and go that way.

I set up a subdomain from one of my domains, that forwards all mail to one of my real addresses. Everytime I have to use my email, I use something at that subdomain, for example, slashdot@catch.domain.com. If I get spam to that address, 1) I can block the address without affecting anything else, and 2), I know who got my name on the list.

Particularily useful when you have to register to get access to download or use something. I'm careful about giving out those addresses anyways, and always "opt-out", so I get a surprisingly small amount of spam to them. I've yet to recieve spam for an address I gave to a company that said it wouldn't spam me.

I'm glad that the majority of posters...

[bmajik]

are seeing this for what it is: "No big deal"

This is NOT big brother. This is about building valuable meta information on top of usenet. Why ? Because one of the things MS heard long ago is that people liked linux because they could go to a newsgroup and get help with it, often from the people that wrote the component in question ? What did MS do ? They responded - MS employees now monitor the microsoft.public news groups. We respond to posts, try and solve problems for people, answer questions, debug code, etc etc. I myself can be found occasionally posting in the Visual Basic newsgruops (where we have lots and lots of non-full-time or beginning programmers that really need just a little bit of help to get them going).

The people that _write_ the VB compiler are now monitoring VB newsgroups to try and help connect with real customers and to really understand how people use and dislike MS products.

Managing and making sense out of the whole mess that is usenet is a nightmare, and MS Research is doing some good work in this area. MS has some internal software that treats usenet posts as "issues" and determines if they've been resolved or not, if they need followup, etc etc. One interesting thing we've found is taht there are many issues resolved by "the community", i.e. non-MS employees that are subject matter experts. I don't know the details on this but I think we make an effort to track who is and isn't a great contributor and maybe they get some sort of compensation or recognition or something.. like i said i don't know the details of that at all..

In any case, the point of this usenet data mining is to try and analyze the incredibly huge sea of usenet. We want to figure out what kinds of problems people have, what people are causing noise, what people are really helping other, etc etc. There is no nefarious invasion of privacy here, the only thing that is analyzable is what people explicitly post to a public forum...

Look at my userid - i was a slashdot reader long before i work where i currently do. Back then, the MS bashing and second guessing definitely took place, and i even participated. I'm still a slashdot reader but I do get awfully tired of the sheer volume and irrationality of negative-MS stuff that happens here.

When I started at MS, I found out awfully fast that many of my arguments against MS were speculative, but mostly it was me being factually wrong and talking out of my ass. I remember in my original interviews i was trying to lecture an NT developer about how putting GDI in kernel for NT4 was stupid because it would lead to crashes. How pompous of me! It was something I read on some stupid website or industry rag. Later I found out (from reading Inside W2k -- excellent book) that it was irrelevant because if the session manager sees that the GDI user-land process exits /crashes for some reason, it reboots the box anyhow, i.e. a problem with GDI reboots the box either way.

So after 8+ years of hating MS and talking out of my ass, followed by 3+ years of working at MS and realizing how much i was talking out of my ass, I'm doing two things:

1) talking out of my ass less
2) telling others that are clearly talking out of their ass that they are doing so, so that they can
2a) stop spreading misinformation
2b) have their eyes opened that nobody is impressed by their incorrect speculations and their emotional campaigns of disinformation

I know im not preaching to a sympathetic audience here, but honestly, the speculation, questions, etc people have about MS could be answered truthfully and honestly if some of you would bother to ask, or do some research. But unfortuneately i know all to well (because i used to do it) that its easier, and certainly more fun, to beleive everything you _want_ to beleive about MS that bolsters your own predetermined mindset. If, for example, you find yourself referring to an article that The Register wrote, please stop and ask yourself what the hell the regis