Slashdot Mirror
How Can You Measure a Wiki's Worth?
moldar writes "I have been involved in a major project to migrate documentation from multiple sources to a wiki (Media Wiki, if you must know). Now, the PHBs are all asking questions about how organized the data is. I've already googled for various wiki metrics ideas, but mostly they focus on page counts, average page sizes, rates of edits, etc. Can anybody suggest better ways of measuring the quality of a wiki? Things like uncategorized pages, articles that are too small, etc? Any help or fresh ideas would be appreciated."
See how many phone calls/emails you get complaining.
I really don't know the best way to go about measuring a wiki, but I'd suggest not taunting it. It might get pissed and rip your arm off.
"Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
PHBs will always insist on metrics for things that can't properly be quantified.
What I would do is suggest that he task a team from another department to run a survey on satisfaction amongst wiki users. Has the wiki helped them? How often do they use it? What would they like to see changed.
This should distract the PHB for a while, while diverting your responsibility to come up with a metric.
http://blindscribblings.com - Tasty pop-culture in conceptual fashion.
It depends a lot on what your wiki is for, but usage pattern statistics might be a good metric. For example, repeat visitors are people who are getting some value out of the wiki. Trampoline pages (pages people hit and then bounce elsewhere from a link in the page) might be another good thing to track. As well as abort-and-retry pages, where instead of following a link the user goes to the search box and tries something else.
You get the idea.
If the wiki is valuable to the people who are using it, you should be able to tell this from the logs--even if no one ever mentioned wikipedia in public, they could tell how much we care by noting that we keep coming back.
--MarkusQ
You cannot "measure" anything until you have defined measurement criteria. So until you define the "worths" of a wiki, you cannot measure them.
Modding Trolls +1 inciteful since 1999
The whole point of a wiki is that it doesn't have to be organized. It doesn't have a table of contents, it has a page list -- which are not organized in any sort of temporal or incremental way, but rather alphabetically.
You don't read a wiki start to finish like most PHB's think documentation should be read. Instead, try to make him understand that relevant links (internal link count metric) and search indexing (search count metrics) are what make a wiki organized.
Which is the whole point. You go from an organized list of things describing X and Y in word documents, to a wiki describing X with subpoints X{a,b} and Y with clarifications to Y{c,f,g,h}, etc. Some people don't care about Y{f}, but some people actually do. And since everything describing Y should be in the same wiki, then all of the users can go to their respective pages.
Its more of a culture change than anything. [Now, talking about wiki uptake... that should be something interesting.]
by using metrics of course
-- Sex is the antonym of pringles. Once you pop it's time to stop.
Many Moons (I apologize for the hosting company in advance, the site is not mine) is a childrens' story everyone in IT, Sales, and Marketing should familiarize themselves with.
The truth of the matter is: You're asking the wrong people. You should be asking the suits what they would imagine in a report and what kinds of numbers they're looking for. If they're general and obtuse, tell them straight up they'll get a general report, or just make up numbers to keep them happy. Create a widget they can access from their desk that shows them numbers generated from the database that hardly matter. Does it matter what you deliver if they don't know what they want?
They're trying to imagine some great power-point presentation you'll be showing with pie charts and red/green/yellow/blue graphs popping out and wowing them? Make it so. They're trying to imagine a spreadsheet? Easier and more concise. They just want a status report at the end of the day on what percentage of the documentation has been migrated? You can probably get away with a few page count written on a napkin.
Remember the Alice's conversation with the Cheshire cat:
"Would you tell me, please, which way I ought to go from here?"
"That depends a good deal on where you want to get to," said the Cat.
"I don't much care where -" said Alice.
"Then it doesn't matter which way you go," said the Cat.
I am the richest astronaut ever to win the superbowl.
eBay
Better known as 318230.
Depending how much of a BOFH you want to be you could ask users to rate the wiki, either a rating for any page they use (hard to enforce): MS have something like this as too do IBM, IIRC, "rate this article on ...".
Alternatively you could do an exit survey, when a user leaves teh site you pop-up a survey to ask about their experience.
Perhaps you could simplify adding templates and get users to add class markings (I think that's what they call them) for things like "this is a lame article", "this article needs editing for brevity", etc. and markings of "this is a B grade article", etc., then run a test of how many of what grades are given and how many templates are used.
Note that templates describing failings will be more prevalent as the wiki gets more use (per user and overall user count) a PHB might not twig this and will think quality is going down.
You could also measure uptake versus the previous sources.
I'm actually in a situation which is very similar to the OP's. The company I work for has a number of training/procedural documents which currently exist in .mht format (of all things), and we're in the process of migrating those documents into the wiki which comes built into MOSS (the latest version of SharePoint). I won't go into detail about how horribly crippled the MOSS wiki is, except to say it is. Anyway, I was asked to write a "Wiki Best Practices" document and I tried to include a few ideas on what makes a successful wiki because I knew the business users were pretty clueless about them (not that I claim to be an expert here). These were some of the points I tried to address:
First and foremost are the users able to find what they are looking for? If they can't find it in the wiki they'll go somewhere else for it, and the wiki will fall on its face.
Is the information accurate? If you're not meeting these two criteria your wiki is probably worthless, and your users will almost certainly resort to another source of information.
Does your wiki exhibit signs of growth? By this I mean new pages/sections/categories being added when new information becomes available, and edits being performed to improve the quality of existing information. I think one of the biggest hurdles companies will have to overcome for this criteria is the fact that there tends to be a very top-down approach to the distribution of new information (This is definitely the case in my company). In order to allow the wiki to grow there must be enough users with create/edit privileges to keep it up to date. The more users there are who are actively involved in the wiki, the more likely it is to be successful. Of course, if management doesn't want to allow the user populace to be able to edit the wiki the task will probably fall to a select few, and they will inevitably become involved in their other tasks at which point the wiki will fall by the wayside and die a quick death. This is the battle I'm currently fighting, and I'm not entirely sure I'll win.
Ask your users what they think. This one is pretty obvious, but easy to overlook if you're doing things like checking page hit counts and sizes. Put together a survey and ask them if they find the wiki to be easier/more useful than the previous method, if it saves them time (and if so how much), and what they think should be done to improve it. Ask them if the information they find is accurate or not. Finally, ask them what they did/didn't like about the previous method, and whether or not that issue has been addressed by the wiki.
God, schmod. I want my monkey man!
The best measure of a wiki is connectedness, in my opinion. Do all pages have incoming links from relevant articles? Can all the pages be found by casual clicking, or are there some that only have links buried in the middle of paragraphs? Is everything categorized? You should have at least one well organized category hierarchy that contains everything, then add further categorizations from there.
Look, the pointy haired boss doesn't really want or expect a numerical metric of organization. Immagine asking him if he has a function in Word that will tell him how organized an outline or document is. He doesn't, and if he never needed it for Word, he doesn't need it for a webpage.
Possibility 1: he wants to kill the project, and came up with this as a way to find an excuse. The best response to this is to get your resume out there on monster right now, and walk into his office tomorrow morning and say "do you want to kill the wiki, and is this metric a way to get an excuse ?" . Let that conversation go where it will.
Possibility 2: he read in USA Today that anyone can add stuff to wiki's and they are chaotic, so he's worried. He's never actually looked at any wiki including this one in his life. This is the most likely possibility. If you still have a job after asking the question in Possibility 2, go for this, or maybe start with this.
Here, you want to talk to him for about 5 minutes using the following words as much as possible, without actually lying: "review", "control", "process" and "catagorize". If you want to can reveiw everything he has ever said and come up with your own set of buzzwords, based on the words he himself likes to use, but these will do. Then you say things like "We have a PROCESS to CONTROL the QUALITY of employee submissions. I occasionally REVIEW the most visited pages in the wiki, and make sure they are ACCURATE and properly CATAGORIZED. I also REVIEW the least visited pages, and if they are not visited because they are improperly CATAGORIZED and linked, I fix that." Actually talk like that, and kind of shout the buzzwords at him. He's either stupid enough that he needs it, or smart enough to figure out what's going on and move on to something important.
All the geeks in this discussion who started actually talking about measuring connectedness of graphs and crap are totally off in the weeds.