Slashdot Mirror

← Back to Stories (view on slashdot.org)

Are Website Performance Metrics Still Relevant?

Posted by Cliff on from the stuff-to-discuss dept.

jackvalko asks: "Recently Keynote Systems began an upgrade of Transaction Perspective, one of their performance measuring products. The data collected is used by the Executive staff at the dotcom I work for as a means to evaluate our customer experience. Now that they are almost done, we've noticed a better than 40% reduction in response time of our site. While I'm happy that our performance 'looks' good, this major change has given us pause to question the statistical relevance of the data that is being collected and the somewhat arbitrary nature of performance metrics collection in general. And, of course, how we message all of this to upper management. So I put it to you, Slashdot! Do you find performance metrics relevant to your customer experience? How do you manage expectations and change to upper management? And, most importantly perhaps, are organizations that collect this data still relevant on The Internet as it exists today?"

4 of 14 comments (clear)

  1. Performance Metrics by PhiznTRG · · Score: 3, Insightful
    First you have to decide what difference it makes if the results are good or bad - who gets judged and is held responsible.

    If it is you then: like all performance metrics, if the results are good then they matter - argue so. If the results are poor, then they are just statistics and you should point towards other, less numerical, measurements to justify your continued employment.

    Of course, for these types of measurements to matter you have to be measuring the correct thing. If the website loads faster but the customer still can't find the needed information because the layout sucks, then clearly that is not a good measure of customer satisfaction. You have to correlate what you are measuring (website performance) with what you care about (customer satisfaction). Unless you can show how those are related then you are correct in your worry - the performance stats are useless.

    1. Re:Performance Metrics by Oculus+Habent · · Score: 3, Insightful

      I think it is fair to say that for a large share of the market, performance statistics are only significant if they are bad. If you have no complaints - and remember, the number of people who report a problem is a tiny fraction of those who experience it - and your numbers are "average" then you should be fine. If the numbers are bad, you have something to work on.

      For some markets, like online trading, etc, performance can mean something entirely different.

      --
      That what was all this school was for... to teach us how to solve our own problems. -- janeowit
  2. Absolute versus relative by Otter · · Score: 2, Insightful
    Knowing nothing about website metrics per se, but if I understood your point correctly -- note the difference between absolute and relative metrics. Obviously, your site hasn't suddenly improved 40% because you're testing it with something new. The score from the old version can't be directly compared with those from the new version, let alone with numbers from different software. But that doesn't mean you can't draw valuable conclusions by comparing different results obtained with the same software.

    If your upper managers aren't complete idiots, that should be clear enough to them, but YMMV...

    --
    What I'm listening to now on Pandora...
  3. It depends... by Anonymous Coward · · Score: 2, Insightful

    As you well know, it's pointless to compare apples to oranges, or in this case, the performance of your web site to other web sites in your industry.

    When we collect performance metrics, we use that information to drive the next round of improvements. We look at bottlenecks, common areas of complaints, feature requests, load peaks, cache performance, etc etc with the goal of identifying the biggest bang for the buck.

    Customer satisfaction is however an important aspect of your business model and knowing how your software system compares to others will give you a sense of perspective. For example, if your system responds ten times faster than your competitor, there's probably little point in making the system faster from a business perspective.

    But since you asked for some specific examples, our most important "question" is, "do our customers find what they're looking for?" Towards that end, we look at how often certain pages get hit (such as help pages, search pages, site maps, the back button, hierarchial links etc) as well as surveys. Our userbase is so small (around 2,000 daily visitors) that we don't really care whether it takes 0.1 seconds to serve a page or 0.001 seconds.

    So the real question is... what does your company consider to be a good performing website verses a bad performing one? That will determine how relevant the metrics you collect are.