Slashdot Mirror
Are Website Performance Metrics Still Relevant?
jackvalko asks: "Recently Keynote Systems began an upgrade of Transaction Perspective, one of their performance measuring products. The data collected is used by the Executive staff at the dotcom I work for as a means to evaluate our customer experience. Now that they are almost done, we've noticed a better than 40% reduction in response time of our site.
While I'm happy that our performance 'looks' good, this major change has given us pause to question the statistical relevance of the data that is being collected and the somewhat arbitrary nature of performance metrics collection in general. And, of course, how we message all of this to upper management.
So I put it to you, Slashdot! Do you find performance metrics relevant to your customer experience? How do you manage expectations and change to upper management? And, most importantly perhaps, are organizations that collect this data still relevant on The Internet as it exists today?"
If it is you then: like all performance metrics, if the results are good then they matter - argue so. If the results are poor, then they are just statistics and you should point towards other, less numerical, measurements to justify your continued employment.
Of course, for these types of measurements to matter you have to be measuring the correct thing. If the website loads faster but the customer still can't find the needed information because the layout sucks, then clearly that is not a good measure of customer satisfaction. You have to correlate what you are measuring (website performance) with what you care about (customer satisfaction). Unless you can show how those are related then you are correct in your worry - the performance stats are useless.
If your upper managers aren't complete idiots, that should be clear enough to them, but YMMV...
What I'm listening to now on Pandora...
As you well know, it's pointless to compare apples to oranges, or in this case, the performance of your web site to other web sites in your industry.
When we collect performance metrics, we use that information to drive the next round of improvements. We look at bottlenecks, common areas of complaints, feature requests, load peaks, cache performance, etc etc with the goal of identifying the biggest bang for the buck.
Customer satisfaction is however an important aspect of your business model and knowing how your software system compares to others will give you a sense of perspective. For example, if your system responds ten times faster than your competitor, there's probably little point in making the system faster from a business perspective.
But since you asked for some specific examples, our most important "question" is, "do our customers find what they're looking for?" Towards that end, we look at how often certain pages get hit (such as help pages, search pages, site maps, the back button, hierarchial links etc) as well as surveys. Our userbase is so small (around 2,000 daily visitors) that we don't really care whether it takes 0.1 seconds to serve a page or 0.001 seconds.
So the real question is... what does your company consider to be a good performing website verses a bad performing one? That will determine how relevant the metrics you collect are.
>> Do you find performance metrics relevant to your customer experience?
Art thou high, Romeo?
Website too slow? I leave. Website payment insecure? I leave. Website crapfloods me with cookies? I leave. Website menus unorganized? I leave. No content? I leave. Too many clicks to get what I want? I leave. These are my performance metrics.
Bye.
This must be a record. Two hours and only five comments? [Insert witty comment here. Subject may be the character of the original poster, the quality of Slashcode, or one of your own choosing.]
I know this is anathema to most big businesses, but you say you work for .com, so maybe this isn't so far out in left field. What is the point in paying Keynote big bucks to tell you what your page load times are? Great, you've improved your performance 40%. Does it matter? No one here can tell you, Keynote can't tell you. Only your customers can tell you what about your site makes them happy and what frustrates them. So ask them!
In the realm of online trading and other time-sensitive transactions, obviously it matters.
Outside that, as long as the reponse "feels" fast enough, and you're getting zero complaints, either it's good enough or no one's using it. Imposing artifical hard & fast numbers (like "no more than 5 second response time") leads to budget overruns and disappointment all around. The overall experience for the user is what's important. Your server logs will tell you if no one's using the app/site/system.
For example, this morning I went on my insurance company's site to get a quote on making a change to my policy. Moving from step to step of the "wizard" was pretty snappy. At the end, when they told me "ok, now we'll submit the quote", it took a little longer, but I also knew that at this point, it was more than data collection - it was actually crunching numbers. At this point, I think most users are accustomed to this.
I have worked on a few systems that had severe performance problems, and you will know when people aren't happy with performance - they speak up loudly and frequently. One of them, we were never able to really fix it, both because of architecture and beauracracy. The other, we found a huge speed boost on the front page, and even though I wasn't happy with response on a couple pages deeper in, users/management found it acceptable.
Customer experience is far more than load speed, therefore measuring Customer Experience by means of site performance makes no sense. Maybe your customers don't care about load speed, but want more dancing baby gifs. Getting inside the head of the customer is the only way to know what they want and how to give it to them. Learn what is important to your customers, then you can accurately define what/how to measure. Look into Opinionlab www.opinionlab.com
Install a 2400 baud modem in his machine before you demo the site. I bet it will take him only a few agonizing minutes to see the difference your performance improvements make.
I looked in my Preferences for the "thinly veiled advertisement" Section, but I can't find it.
Get off my lawn.
When you serve out a website that pushes in excess of 200,000,000 pageviews a day (yes I really do, dynamic even!) then you start to use a lot of metrics to try to gauge how your code changes and network changes affect things. There are lots of companies that specialize in this sort of thing. This runs from the application level all the way down to the network level, sometimes even the transport layer if you buy their marketing.
Certainly from the network (RouteScience, Internap, etc) to the application (Zend, Urchin, Webtrends) there are all kinds of companies willing to provide you metrics and solutions, usually at a high cost.
The trick is to build your own metrics and then find or build the products you need to solve your bottlenecks or improve your user experience.
Just some ideas, not sure if it is the answer you want.
OT: And since I refuse to post in the "how do I build a mail server for 1mm users" thread I'll just say it here. "What the fuck dude? First of all, 1mm users isn't all that large these days. SMTP and POP3 are no brainers. IMAP is a bit harder but not really. Check out perdition and maildir and qmail or postfix or even sendmail if you know it well. This shit has been done over and over and over again. There is nothing even remotely complex in providing mail stores for 1mm users. THERE ARE FUCKING HOW-TO's to do it even. Okay, </rant>"
And I'm out.
-david
# Hack the planet, it's important.