Slashdot Mirror
Microsoft's Asimov System To Monitor Users' Machines In Real Time
SmartAboutThings writes Microsoft will monitor users in the new Windows 9 Operating System in order to determine how the new OS is used, thus decide what tweaks and changes are need to be made. During Windows 8 testing, Microsoft said that they had data showing Start Menu usage had dropped, but it seems that the tools they were using at the time weren't as evolved as the new 'Asimov' monitor. The new system is codenamed 'Asimov' and will provide a near real-time view of what is happening on users' machines. Rest assured, the data is going to be obscured and aggregated, but intelligible enough to allow Microsoft to get detailed insights into user interactions with the OS. Mary Jo Foley says that the system was originally built by the Xbox Team and now is being used by the Windows team. Users who will download the technical preview of Windows 9, which is said to get unveiled today, will become 'power users' who will utilize the platform in unique scenarios. This will help Microsoft identify any odd bugs ahead of the final release.
From Windows 98 onward you had the shortcut bars which you could create on the task bar - thats where the majority of my most often used applications were started from.
That morphed into pinning applications to the task bar in Windows 7, and became much more useful as pinning an application and running that same application took up no more room on the task bar, so you could have more.
These days I pretty much have all my applications pinned to the task bar, and I hit the start menu probably once or twice a week, if that. I can lock the computer, minimise all windows, start applications, open task manager, get to the control panel and lots of other things via either interaction with the task bar itself or via keyboard shortcuts, where as before I had to use the start menu for a lot of that.
I remember when they were talking about this research at the time. If I remember correctly, they found that most people rarely hunted through the start menu "Programs" menu. They pinned applications to their task bar, or they put shortcuts on their desktop. If they used the start menu, they usually either used the search function or the list of applications that were pinned to the start menu.
This lead them to think that the Windows 8 UI would be fine, since you could still search, and you could still pin applications to the Start screen. It seems they figured, if most people aren't using the other features of the Start menu, we can provide a solution that only includes the two features people do use, and everyone will be happy for the simplified solution. Apparently they are now admitting that their approach was flawed or insufficient.
Lots of other companies manage to produce a great UI without telemetry. It's pretty sad that a company of Microsoft's depth needs telemetry data to break the management deadlocks that are contributing to the 'designed by committee' feeling of Windows 8. Talent and balls seem to be absent in these decisions.
- The Kessel run is for nerf herders. I can circumnavigate the entire Central Finite Curve in a lot less than 12 parse
"How does one obscure data to the point where you can't identify the user, but still have meaningful data? Haven't we heard this all before?"
Easily, if all you want is to figure out things like "How long does it take a user to find the application they want in the Start Menu" then all you're doing is timing from the moment they click start, to the time they click a start menu option. You don't need to know who the user is, or even what IP the data was submitted from and when you have a lot of this data it's trivial to tell if the mean time users take to find an application has increased or decreased after you made a change in an update, or after they changed a configuration setting.
If all you're doing is getting metrics on millions of users as to how they use things like this then it's trivial to keep it anonymised and non-identifying. I don't care how long it takes John Smith from Outer Mongolia specifically to find Microsoft Word in his Start Menu - I don't need to take information about who he is, where he lives or any such thing, I just want to know how long on average it takes a sample of users to do so for example.
Though of course, this isn't to say that I trust Microsoft to do just this, I don't for one moment imagine they'll be able to resist the urge to keep the data anonymous and/or only collect data that is non-identifying, but that doesn't change the fact that it's trivial to come up with useful metrics they may choose to gather without it being identifiable - what they're claiming is certainly possible, realistic, and even helpful to them (and arguably users too if they get a better product out of it) but whether they'll stick to what they're claiming or not? that's what's troubling here.
Disclosure:
I work extensively with Microsoft customer usage data (although on Visual Studio, not Windows)
Odds are, unless you've been very intentional about ticking the checkboxes the right way, Microsoft is already collecting usage data from you -- for a variety of products. Never without your consent, of course.
The issues around anonymizing your data and removing PII are taken very seriously. It's damn frustrating, because I often look over the data for user 234209342349 and think, "I wish I could email this guy and ask why the hell he is doing that". But there is no way for me to recover PII for VS client customers.
For the Visual Studio products, a typical approach is that data that might have a PII impact is one-way hashed on your local machine, so that PII never goes over the wire and never gets to Microsoft to begin with.
You can use tools like filemon to see where VS dumps the usage data files it generates. I don't remember if these look like binary mess on disk or not, but they get written to disk, and then you can see them go over the wire some time later. You could of course use a packet sniffer to see the on-the-wire format, and if it differs from what is stored on disk.
The data we scrub in VS covers the obvious things -- account names or email addresses -- but also some more subtle things -- like file paths (because these could contain your username, or a company name, or anything else), and even thing like VS Project Type names (because Company Foo can create their own Project Type, and might put their company name in the Project Type Name)
So anyway, there's actually not much of a story here. I can't comment on the truth or accuracy of what MJF is saying. However, what she is saying is that, in effect, the latency between usage data being locally captured/calculated, and that data being sent to Microsoft (assuming the user has allowed usage data to be sent), is now much lower than it was in the past.
For VS, at least, I know what data we have available to us. I opt-in to all of the MS data collection stuff, because I see no evidence of it being used inappropriately, and, because I know that we use it to try and understand what users are doing and why they are doing it.
Opting into the data collection stuff effectively gives you "a vote" in how we do things in future releases.
My opinions are my own, and do not necessarily represent those of my employer.
The trick of course is knowing whether there's a secondary channel that they use to send the PII and associated hash that they wouldn't generally provide to anyone except say the NSA.
Of course a packet sniffer would find that out easily enough, and I'm guessing that someone would have already done so and let the world know if that was the case (and thus its probably not,) but simply being anonymized in the data you have doesn't directly imply that there isn't additional data somewhere capable of de-anonymizing it.