Slashdot Mirror
How Can You Measure a Wiki's Worth?
moldar writes "I have been involved in a major project to migrate documentation from multiple sources to a wiki (Media Wiki, if you must know). Now, the PHBs are all asking questions about how organized the data is. I've already googled for various wiki metrics ideas, but mostly they focus on page counts, average page sizes, rates of edits, etc. Can anybody suggest better ways of measuring the quality of a wiki? Things like uncategorized pages, articles that are too small, etc? Any help or fresh ideas would be appreciated."
See how many phone calls/emails you get complaining.
I really don't know the best way to go about measuring a wiki, but I'd suggest not taunting it. It might get pissed and rip your arm off.
"Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
It would be worth whatever you can get someone to buy it for.
Nice editing on the part of whomever took the firehose article. Changing the title changed how anybody is going to read the article.
Getting a single metric for the quality of a wiki is going to be really hard, since the number would just about be made up.
If you want to say what percentage of pages are uncategorized, below a certain size, or orphaned, then do that.
If I have nothing to hide, don't search me
your wiki measures you!
I drink to make other people interesting!
PHBs will always insist on metrics for things that can't properly be quantified.
What I would do is suggest that he task a team from another department to run a survey on satisfaction amongst wiki users. Has the wiki helped them? How often do they use it? What would they like to see changed.
This should distract the PHB for a while, while diverting your responsibility to come up with a metric.
http://blindscribblings.com - Tasty pop-culture in conceptual fashion.
It depends a lot on what your wiki is for, but usage pattern statistics might be a good metric. For example, repeat visitors are people who are getting some value out of the wiki. Trampoline pages (pages people hit and then bounce elsewhere from a link in the page) might be another good thing to track. As well as abort-and-retry pages, where instead of following a link the user goes to the search box and tries something else.
You get the idea.
If the wiki is valuable to the people who are using it, you should be able to tell this from the logs--even if no one ever mentioned wikipedia in public, they could tell how much we care by noting that we keep coming back.
--MarkusQ
You cannot "measure" anything until you have defined measurement criteria. So until you define the "worths" of a wiki, you cannot measure them.
Modding Trolls +1 inciteful since 1999
The whole point of a wiki is that it doesn't have to be organized. It doesn't have a table of contents, it has a page list -- which are not organized in any sort of temporal or incremental way, but rather alphabetically.
You don't read a wiki start to finish like most PHB's think documentation should be read. Instead, try to make him understand that relevant links (internal link count metric) and search indexing (search count metrics) are what make a wiki organized.
Which is the whole point. You go from an organized list of things describing X and Y in word documents, to a wiki describing X with subpoints X{a,b} and Y with clarifications to Y{c,f,g,h}, etc. Some people don't care about Y{f}, but some people actually do. And since everything describing Y should be in the same wiki, then all of the users can go to their respective pages.
Its more of a culture change than anything. [Now, talking about wiki uptake... that should be something interesting.]
by using metrics of course
-- Sex is the antonym of pringles. Once you pop it's time to stop.
Many Moons (I apologize for the hosting company in advance, the site is not mine) is a childrens' story everyone in IT, Sales, and Marketing should familiarize themselves with.
The truth of the matter is: You're asking the wrong people. You should be asking the suits what they would imagine in a report and what kinds of numbers they're looking for. If they're general and obtuse, tell them straight up they'll get a general report, or just make up numbers to keep them happy. Create a widget they can access from their desk that shows them numbers generated from the database that hardly matter. Does it matter what you deliver if they don't know what they want?
They're trying to imagine some great power-point presentation you'll be showing with pie charts and red/green/yellow/blue graphs popping out and wowing them? Make it so. They're trying to imagine a spreadsheet? Easier and more concise. They just want a status report at the end of the day on what percentage of the documentation has been migrated? You can probably get away with a few page count written on a napkin.
Remember the Alice's conversation with the Cheshire cat:
"Would you tell me, please, which way I ought to go from here?"
"That depends a good deal on where you want to get to," said the Cat.
"I don't much care where -" said Alice.
"Then it doesn't matter which way you go," said the Cat.
I am the richest astronaut ever to win the superbowl.
eBay
Better known as 318230.
The quality of the wiki is not the same as how organized it is, and organizational quality is human-measurable, but in the absence of good, labelled training data - or probably even with it - that's not computable. But you could apply metrics from graphing theory (pages as nodes, links as edges) and easily fool a PHB with jibberish.
Depending how much of a BOFH you want to be you could ask users to rate the wiki, either a rating for any page they use (hard to enforce): MS have something like this as too do IBM, IIRC, "rate this article on ...".
Alternatively you could do an exit survey, when a user leaves teh site you pop-up a survey to ask about their experience.
Perhaps you could simplify adding templates and get users to add class markings (I think that's what they call them) for things like "this is a lame article", "this article needs editing for brevity", etc. and markings of "this is a B grade article", etc., then run a test of how many of what grades are given and how many templates are used.
Note that templates describing failings will be more prevalent as the wiki gets more use (per user and overall user count) a PHB might not twig this and will think quality is going down.
You could also measure uptake versus the previous sources.
1. Pretend to invent your own metric by lack of pre-existing satisfactory metric
2. Describe it using esoteric terms you know your audience won't grasp/buzzwords
3. Make up phony results
4. ???
5. Ask for a raise, errr, I mean, profit!
You just got troll'd!
I was on the same boat not that long ago; here are a few ideas:
1. Wiki documentation to non-wiki documentation ratio. Compare how much documentation there is between the wiki itself vs. other project documentation like your word documents and pdfs (assuming they are not just copy/paste stuff from the wiki). A simple word count as measure will probably please your PHBs.
2. Assuming that this is a programming project that you are in, you can also compare the % growth of the wiki vs. the % growth of the codebase. It somewhat implies how much people are contributing to the wiki vs. doing the actual project.
3. Assuming that people properly tag pages; you can try to see the average time it takes for a tagged article to become untagged. I don't know exactly how to do that in mediawiki but I am sure you can write a complicated SQL script in order to get such information (assuming you have direct access to the database).
4. Other people mentioned activity as a good measure. I would also include average number of edits per page (people are actually reading and editing pages instead of dumping things in) and also average time between edits (people are constantly updating pages).
I'm actually in a situation which is very similar to the OP's. The company I work for has a number of training/procedural documents which currently exist in .mht format (of all things), and we're in the process of migrating those documents into the wiki which comes built into MOSS (the latest version of SharePoint). I won't go into detail about how horribly crippled the MOSS wiki is, except to say it is. Anyway, I was asked to write a "Wiki Best Practices" document and I tried to include a few ideas on what makes a successful wiki because I knew the business users were pretty clueless about them (not that I claim to be an expert here). These were some of the points I tried to address:
First and foremost are the users able to find what they are looking for? If they can't find it in the wiki they'll go somewhere else for it, and the wiki will fall on its face.
Is the information accurate? If you're not meeting these two criteria your wiki is probably worthless, and your users will almost certainly resort to another source of information.
Does your wiki exhibit signs of growth? By this I mean new pages/sections/categories being added when new information becomes available, and edits being performed to improve the quality of existing information. I think one of the biggest hurdles companies will have to overcome for this criteria is the fact that there tends to be a very top-down approach to the distribution of new information (This is definitely the case in my company). In order to allow the wiki to grow there must be enough users with create/edit privileges to keep it up to date. The more users there are who are actively involved in the wiki, the more likely it is to be successful. Of course, if management doesn't want to allow the user populace to be able to edit the wiki the task will probably fall to a select few, and they will inevitably become involved in their other tasks at which point the wiki will fall by the wayside and die a quick death. This is the battle I'm currently fighting, and I'm not entirely sure I'll win.
Ask your users what they think. This one is pretty obvious, but easy to overlook if you're doing things like checking page hit counts and sizes. Put together a survey and ask them if they find the wiki to be easier/more useful than the previous method, if it saves them time (and if so how much), and what they think should be done to improve it. Ask them if the information they find is accurate or not. Finally, ask them what they did/didn't like about the previous method, and whether or not that issue has been addressed by the wiki.
God, schmod. I want my monkey man!
The problem is that he's trying to measure the wrong thing. He's trying to get metrics from the wiki itself. He should be getting them from the users.
So the simplest solution is to do a user survey about what's good and what needs improvement.
The best measure of a wiki is connectedness, in my opinion. Do all pages have incoming links from relevant articles? Can all the pages be found by casual clicking, or are there some that only have links buried in the middle of paragraphs? Is everything categorized? You should have at least one well organized category hierarchy that contains everything, then add further categorizations from there.
There are metrics that can be applicable to the wiki itself, but probably for what they want to know what they have to measure is the community, the people that will feed it, not the tool. Is like measuring the quality of a book by the quality of the paper or ink that was used on it.
You can use page size, amounts of edits, amount of pages, etc, but that will not imply that things are right or wrong. Sometimes a single word in the right context could say all you need. I remember that a page on wikipedia had a lot of edits (in the date of birth of someone?) because that info wasnt known for sure. A page that had a single author once is better or worse than one that had a hundred? All depends, but more important, is not there where is the answer they want.
I'd say the best measure of a wiki's effectiveness is probably a thorough analysis of the access logs for the web server where the wiki lives. Run them through something like awStats (personal favorite) and see what you get. Are people in your organization actually visiting the wiki? Which pages are most frequently accessed, and by whom? How many users are hitting the Edit or History portions of a page, as opposed to simply viewing static content? These are solid, quantitative numbers, and presented in a "dashboard" style format like you can get from awStats, that will probably satisfy the PHBs.
I try to avoid orphaned pages altogether. Every page in your wiki should fall under some umbrella or hierarchy, even if it's just a catch-all like "Miscellaneous," so that it can easily be found by people who didn't know it was there.
In my experience, people tend to use wikis one or both of two ways: either they do a title/text search to find a particular topic, or they just start exploring and reading what they have access to. The latter can be a great way to get new hires up to speed, or for people to gain knowledge about various business processes in their downtime. Orphaned pages are only accessible by search and IMO defeat half the purpose of having a wiki.
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
Nonsense, a wiki is at least 2-3 LoC's, maybe 4 at the max..
I can understand why you'd be confused - trying to measure the quality and worth of information is a big problem (just look at Google and AI), let alone how it is organized.
There are a lot of problems with Wikis - even the benefits, that you can grow your base of information, result in problems: who is going to reorganize it so it is usable again? What are good lists and bullet points to divide gobs of text into into usable chunks of information?
Someone has already mentioned accurateness, but what that really means is correctness + point of view + timestamp of information, and these things are rarely if ever tracked normally, let alone in a wiki. Sure wiki's usually track all changes, but that doesn't mean the information is correct at the time of entry, nor that you necessarily know who's point of view is being described. And that's only if you take the time to go through all the logs! These details should be immediately apparent when viewing a page of valuable information: do you realize how many websites have articles and reviews without ever showing a timestamp of when they were written, submitted, or published?
In the ideal environment wikis would self-organize, ie. someone sees a page that's gotten too big and they take the time to divide and conquer. But that hasn't been my experience so far, and about half of the few times someone has rearranged them they end up using page titles that are awkward, and wiki's don't usually have a tools to rename a page or to list all links pointing to a page and selectively updating them (half to the new page, half to the old one with a new name...). Wiki management can be a real nightmare. Just like any large company's file repository.
Wiki SEO (especially within the wiki program itself) doesn't usually seem to be dealt with either - or if it is... I haven't seen it. I rarely have a wiki's search function turn up what I'm looking for even if I created the pages I need. So there's some other relationship to information here too that's missing some key features or concepts in our lives so far.
Anyone want to help?
I mod you -1 for american bashing. Why? Because ... uh ... I'm american! And I dislike criticism!
People are very resistant to pay for something that they feel should be free. The same people that would pay a hundred dollars or more for MS Office would feel screwed if they paid $10 for OpenOffice.
Value to the economy is very different to the value marketeers attach to something. Wikipedia, Linux and many other free products are very valuable to those that use them, but taht does nopt mean people would willingly pay for them. Marketeers would see that these products have no value because you cannot make money out of them.
Engineering is the art of compromise.
Look, the pointy haired boss doesn't really want or expect a numerical metric of organization. Immagine asking him if he has a function in Word that will tell him how organized an outline or document is. He doesn't, and if he never needed it for Word, he doesn't need it for a webpage.
Possibility 1: he wants to kill the project, and came up with this as a way to find an excuse. The best response to this is to get your resume out there on monster right now, and walk into his office tomorrow morning and say "do you want to kill the wiki, and is this metric a way to get an excuse ?" . Let that conversation go where it will.
Possibility 2: he read in USA Today that anyone can add stuff to wiki's and they are chaotic, so he's worried. He's never actually looked at any wiki including this one in his life. This is the most likely possibility. If you still have a job after asking the question in Possibility 2, go for this, or maybe start with this.
Here, you want to talk to him for about 5 minutes using the following words as much as possible, without actually lying: "review", "control", "process" and "catagorize". If you want to can reveiw everything he has ever said and come up with your own set of buzzwords, based on the words he himself likes to use, but these will do. Then you say things like "We have a PROCESS to CONTROL the QUALITY of employee submissions. I occasionally REVIEW the most visited pages in the wiki, and make sure they are ACCURATE and properly CATAGORIZED. I also REVIEW the least visited pages, and if they are not visited because they are improperly CATAGORIZED and linked, I fix that." Actually talk like that, and kind of shout the buzzwords at him. He's either stupid enough that he needs it, or smart enough to figure out what's going on and move on to something important.
All the geeks in this discussion who started actually talking about measuring connectedness of graphs and crap are totally off in the weeds.
Unless you work for yourself, you have to *learn* to put up with PHB bullshit. That's just life, dude.
The PHB's at our pad wanted to give a group of people web cams so they could watch their faces in conference calls. The users for the most part don't really want them and don't want to mess with them. But if the big-wigs want it, they have to do it.
Table-ized A.I.
The PHB's at our pad wanted to give a group of people web cams so they could watch their faces in conference calls. The users for the most part don't really want them and don't want to mess with them.
I can see why. That sounds kinda perverted. I'm guessing that it's not two way...
If it's not voyueristic, it's at least a bit extreme. Are they trying to fix a problem, or create one?
it must be a strange situation...
Install some analytics tags and look at the data. How many people are using it? How many pages are accessed every day/week/month? etc.
If it's public-facing, put ads on it and see how much money you make. :) Can't beat that for a "how much is this worth?" question.
Comment of the year
... your bosses are demanding citations, and you're pissed about their lack of NPOV.
Slug it out in a good ol' fashioned edit war.
'a';DROP TABLE users; SELECT * FROM DATA WHERE name LIKE '%'... if you're reading this, it didn't work.
As the person who started a wiki which has a small potential audience (10 million or so worldwide by some estimates) is that after a growth spurt at the start, it has now slowed down. Currently it gets about 100 hits per day - 3 or 4 of those are spammers adding junk pages and the same number is me deleting them, so around 90 hits per day on the front page. The metrics I use are : is the number of contributors increasing?, Are any pages being edited and how often? Are people returning? Is the number of pages growing? Whilst the accuracy of information is important, it is not in my opinion critical because the spirit of a wiki is that it can be made more accurate by another contributor.
I manage a large intranet wiki at a university, based on TWiki. We also do some research on its use. The main metrics I would suggest :
* number of pages (cumulative graph over time, and per month/day)
* number of edits, idem
* number of views , idem
* Number/size of attachments, idem
* a histogram of edits/views per person
* a histogram of people who only read and never contribute.
you can obviously anonymize the graphs if needed. A little straightforard programming-fu and access to the server logs goes a log way.
Before you show the histograms to the bosses, you need to tell them about the http://en.wikipedia.org/wiki/Power_law , as you will be looking at one, guaranteed. 10 % contribute, 90% read. That is fine and natural, but they must understand that. You can demonstrate the fact that people read it a lot and that it is a resource they use (use the the views/views and no edits graph), and the (relatively few) people who actively contribute must be encouraged to do so, and maybe even given an explicit task to do so.
You can also plot a degree distribution histogram per page ( http://en.wikipedia.org/wiki/Degree_(graph_theory) ), where the degree of a page is the number of outgoing links to pages + number of pages referencing back to it. If you get a power law again, you have a few important portal pages and lots of content. More uniform distribution suggests something in the lines of an encyclopedia/storage type of content. I would suggest that having a power law is better, as people find stuff more easily ( bar a good search engine...) You can also use any of the graphing tools (http://vlado.fmf.uni-lj.si/pub/networks/pajek/ or http://jung.sourceforge.net/ ) to draw all pages out, and show different clusters of content. AGain, you can do this over time
Oh, and while you are at it, list the top/bottom 10 (20, 30) most used pages, and explain why. Observe how they change over time.
Im sure that there is enough suggestions here to get a nice progress metric for the boss. Good luck.
Gori
Complexity is a measure of our ignorance...
Sell it. The money you get is the real world worth of that Wiki...
When a user leaves your site, just pop up one of those irritating survey windows with a 'how useful was this wiki to you?' (You might have to say 'web site' if 'wiki' means nothing to them)..
Log some answers, or just plain old make them up, and put them in an Excel spreadsheet which you then print out and give to your PHB.
Use mind mapping software... they make professional mind mapping software for businesses.
http://www.thebrain.com/
And it organizes information much better.
If your organization is anything like mine, the most useful part of the wiki is how much simpler it makes life for the support team. Anytime a particular issue reaches a critical mass, a wiki article can go up and enter the revision cycle, and from that point forward all calls related to that issue can be answered with a single hyperlink.
I'd look at a few measures:
Also, institute some meetings where people go through the Wiki and try to eliminate "old news" that has only historic value. This keeps the info fresh and useful.
In addition allow for automatic discovery of new info. For example a subscription to a weekly query for chosen keywords that comes to your e-mail inbox. Set it up for the manager's names to the manager's inbox. It strokes their ego when they discover where they are mentioned (even if it is only the todo lists of last week's meetings :-)
Busy helping non technical users of OpenOffice.org - http://plan-b-for-openoffice.org/
English Language
|_ American Spellings
|_ Misspellings
|_ C
|_ CATAGORIZED, replace CATEGORIZED, British CATEGORISED, not shouting "categorised"
If I spelt anything wrong I'm sure you guys have my back ...
See point #3 on the 10 Tips for a Successful Wiki poster from gdc;
Invest in your wiki
Nothing is perfect off the shelf and your wiki will be no different. Most wiki packages are designed to be customized and extended, so take advantage of that by investing in a few days of developer time to modify it to suit your needs. Re-evaluate wiki usage at regular intervals to gather feedback on the most common gripes and dedicate resources to adjusting the wiki accordingly. The few hours or days spent tweaking it will come back tenfold in time saved by users.
That's a valid part of the metric. Easy is good. When you're figuring benefits / costs, "easy" reduces the denominator and makes the total score much higher.
When you just compare the accuracy of weather forecasting methods, "looking out the window" doesn't rate very high. But it's generally so easy that in practice it's the technique of first recourse and you'd be a fool not to use it. The great thing about "easy" is that, if you find you need more detail, more precision, or whatever, you haven't used up the alloted resources. And if you don't you are time / money ahead.
Anyone who could go to their bosses and say "our website is so easy to use that people just come back time and time again, and here are the statistics to prove it" is in great shape.
--MarkusQ
IBM History Flow is software that graphically represents the evolution of a live document or wiki.
http://www.research.ibm.com/visual/projects/history_flow/