Mobile Magazine's Notebook Tech Support Reviews

antdude writes "Mobile Magazine tested companies' technical support for their notebooks/laptops. Each test had three calls to each of ten major notebook manufacturers (added three additional vendors since last year). Also, called three third-party providers of PC help. On the whole, what they found was a sea of ignorance -- and annoying fixation with pinning down our name, address, and serial numbers. Things haven't gotten any better since our 2004 test -- and most of the vendors we tested have actually gotten worse..."

Comments

Three samples isn't statistically kosher.

[mellon]

This isn't based on a customer survey - it's based on three contrived problems and the phone calls that went with them. Because of the incredibly small sample, you really can't generalize - the results are essentially random. Too bad, because a lot of people will probably just look at the scorecard and never notice the incredibly lame way they did the survey.

Re:Three samples isn't statistically kosher.

[MoralHazard]

The article never claims to be a statistical study. You don't have to approach the issue in a statistical fashion to obtain an informative result.

The validity of an anecdotal study like this hinges on how strong your expectation of consistent service is. If you have good reason to believe that a single experience is probably representative of most visits, you can have a very small sample and still come back with valid conclusions.

Take restaurant reviewing as an example: The Michelin and Zagat's restaurant reviews are generally based on a single visit to each establishment. (Not only that, but different reviews are performed by different people, presumably with different tastes. But that's a side issue, here.) Each reviewer visits a restaurant and writes up a review of his or her experiences at that place, on that day.

But these restaurant guides are relatively accurate, and quite useful. A restaurant business generally provides consistent customer experiences because they TRY to provide consistent experiences. Fast food takes it to the hilt, but any place is doing it, to some degree. Therefore, you don't really need a sizable sample of identical tests to come up with a conclusion.

It's reasonable to believe that customer service operates the same way: companies generally have consistent hiring policies, management policies, and training regimens for their tech-support people. They tend to pay consistent wages and provide consistent benefits. The computer routing systems that connect you to a tech, and the on-the-phone policies that those techs follow, are also highly consistent. Therefore, it seems like the weight of factors creating the "tech support experience" would provide a consistent result.

Statistical analysis would be more useful for something like "what percentage of shipped laptops by this vendor have quality problems", or "what's the average wait time or problem resolution time for a given type of call." You're focusing on the variations and their magnitude, which are going to be pretty small in comparison to the magnitude of the values themselves.

This article is more like reviewing "how aesthetically pleasing is the laptop's case", or "which laptop model has the best feature set for the money". The latter questions, like the article, are focused on parameters that don't submit well to small measurements, and are not likely to vary that much on an absolute scale from customer to customer.

This isn't the most objective, comprehensive, or thorough study I've ever seen, but it's certainly sufficient as an informative review of laptop tech support. And that's all it claims to be.

Re:Three samples isn't statistically kosher.

[mellon]

Your logic is broken. In the one case, they're asking for recommendations, which as you say can be useful. In the case of the article, they try a couple of bogus tests and rate the companies' service on that basis. This just doesn't tell you anything. With a sample of three, even a really good company is only going to get a good rating if all three of the techs who answer are having really good days. This is just a matter of luck. Granted, if all your techs suck, you can be sure of a bad rating, but if only one of your techs sucks, you're going to get a B- with their scoring system, and if two of them suck, you get a D+. Not every tech is going to do a good job every time, so this amounts to a roll of the dice. With a larger sample size, random chance plays less of a part - if you have consistently good techs, you will get a good score, and if you have consistently bad techs you will get a bad score. Generally speaking it helps to have a non-geek asking, because it is for non-geeks that these services exist. So doing a customer survey really is the best way (probably the only cost-effective way) to get a good answer.

It's pretty striking how differently these same companies rated between Consumer Reports, which does statistical surveys using accepted statistical methods across a large number of customers, versus this magazine article, which followed the "let's try this" methodology.

But as I said in my earlier post, most people, like you, will just look at the results and not at the methodology, so if they follow the recommendations here, they may well be steered toward the worst of all the manufacturers, just because that particular manufacturer had a good run of luck. In all fairness, I don't know what Toshiba's customer support is like, so maybe it'll turn out okay, but if so it'll be blind luck that made it work that way, not any actual attempt to inform the magazine's readers. Which is pretty sad.

Re:Three samples isn't statistically kosher.

[benzapp]

Your logic is broken.

This has to be the most pointless and idiotic reply I have ever read in my life. The poster you responded to gave an extremely detailed description of why a small sample size is sufficient. All you have done is repeat the original statement to which he was replying: Oh, its totally random.

ITS NOT FUCKING RANDOM.

If statisticians can statistically determine who the next president of the United States is by polling 1000 people, I think making 3 calls to a tech support line is more than accurate.

There is absolutely NO reason why a reasonably intelligent individual can't solve the problems presented. If they fail on all three tries, that is the entire system of problem solving fails, then there is a problem.

But as I said in my earlier post, most people, like you, will just look at the results and not at the methodology,

The methadology here is quite simple: Companies have standardized THEIR methodology of providing customer service. Randomness should play little role in deciding the outcome. Difficult questions were selected that the average idiot on the phone can't answer right away. The questions are designed to test the interaction of the entire team. This is not Trivial Pursuit or Jeopardy. The entire system in many cases failed, not just an individual telephone flunky.

You claim the logic is broken, but do you have ANY idea how to judge sample size? How many would be enough for you? 50? 100? 10,000?

I think you need to go back to statistics 101.

You get what you pay for

[stoolpigeon]

And frankly I'd rather pay less for a laptop and deal with the service not being so great. I have a hunch this is true with a lot of people.

If a company were to start advertising, 'Hey- our laptops cost more but you get the best service.' I bet they wouldn't sell as well as the company beating their prices.

With the wealth of knowledge available on the web-- I don't usually use support anyway. My family that aren't as tech savvy? They bring their issues to me. They don't use the support either.

My pet on-hold peeve

[QuantumRiff]

If your going to keep my on hold and listening to music, please dear god stop interupting the songs every 15-20 seconds with an automated voice giving me a sales pitch, or thanking me for being a customer, or assuring me a tech is working on the problem. Let me listen to the damn music uninterrupted while I wait.

On the plus side, one tech support line, ( I think it was 3com) had a voice at the start of the hold cue that said, Press 1 for classical music, Press 2 for Jazz, Press 3 for classic rock.... That was pretty nice

Re:My pet on-hold peeve

[RickPartin]

That is a huge pet peeve of mine as well. I can zone out while the hold music is playing and work on other things. But being interrupted every 30 seconds keeps making me think someone has picked up the phone. Grrrrrrrrrr. Yes stupid repeating message I understand how being on hold works. I listen to music until you get around to talking to me. And even if I didn't know this I probably figured out the first 20 times you explained it to me.

I'm not sure if they still do it but Gateway used to have a fake Gateway Radio station for hold music. It was strangely entertaining. Next up is the band blablabla who is currently on tour at....

The Tests Used...

[MS-06FZ]

Call 1: Device driver trouble We disabled our optical drive in Device Manager. Easy fix: Uninstall the relevant device driver and reboot. Windows heals itself.

Call 2: Wi-Fi misconfiguration We turned off TCP/IP routing for our wireless adapter, so we could connect to the router but couldn't browse the web. Easy fix: Check the properties for the relevant adapter to make sure the correct protocols are installed. Or, uninstall the device and reboot.

Call 3: Corrupted operating system We overwrote a critical Windows file (Explorer.exe), a problem that let Windows boot up but made all of our desktop icons and the Start menu disappear. Easy fix: Use System Restore to revert to an earlier configuration. Or, use the operating-system CDs (if provided) to reinstall Windows without reformatting the hard drive.

Yeah, I can't help but feel these tests aren't typical of the problems most people need tech support for... How do these things happen? I mean, apart from being intentional damage caused by someone hoping to test tech support? And in cases where the OS itself does become corrupted beyond usability, is that a common enough problem that it's worth tech support walking people through it? Bear in mind that these people have to answer to hordes of people asking why applications disappear when they click on the little "X".

Re:Employees are not trusted

[Otter]

Instead of me calling up Gateway and saying "Hey my modem is fried, I know what I'm doing with computers, send me a new one" I have to go through an hour of pointless troubleshooting.

The problem is that if they listened to everyone who says "I know what I'm doing with computers!", they'd spend all day shipping out new computers to thousands of people whose cat knocked the power cord out of the outlet.

Horrible Testing

[Qwerpafw]

I'm really amazed by how badly they conducted this test. Some of the reasons why their procedure is horrible have already been touched upon:

Statistically insignificant sample size.

Bad choice of problems.

Arbitrary Grading.

These problems are even more ridiculous when you look at what they did to their apple laptop. As a reference point, Apple has the highest ranked Tech Support by Consumer Reports for both desktops and laptops. In one phone call, they decided this was not the case--apparently conducting surveys of thousands of people is unnecessary.

They also chose different problems for the mac--booting off a non-existent network drive? How is this even remotely a real-world problem? Furthermore, holding down option while rebooting lets you choose the drive your computer will boot from--which is a fix for the problem. If they reset the preference after they booted so that it broke again, that's not Apple's fault.

Misconfiguring wi-fi is also an amazingly horrible test. There's no way to know what settings someone's wireless network and router use, unless you're the LAN administrator. Apple was more than correct to refer them to the manufacturer of the router--could you tell me, right now, what my IP, DNS, Gateway, and hostmask settings should be? What about the SSID and password for my router?

The test was stupidly conducted, and worse yet, only conducted once. Their results were meaningless.

maybe it really isn't about service

[yagu]

I've posted on this before... one of many episodes of trying to get support. In this particular case, I pretty much KNEW what the problem was, which as I'm sure many/most slashdotters also try to determine before resorting to call tech support.

From the word "go", (ironic), it was clear my dance with HP (love their product... a laptop zx5000) was less about them helping me solve my problem and more about them doing anything they possibly could to avoid doing warranty work! And, once they discovered I had a dual boot machine, they immediately jumped to the claim that dual-booting my machine voided the warranty, though not one of the support people I talked to (I talked to four) could point to the words in the warranty whereby dual-booting my machine really did void the warranty.

This was not a unique experience for me... my typical experience is usually along the lines of:

• reboot the machine (as if I'd not tried that multiple times)
• uninstall the drivers around specific bad behavior
• re-install drivers
• complete OS re-install or system re-image.

I don't know what HP and other companies are smoking when they put together "support" staff, but based on empirical and andecdotal evidence they don't "get it". Especially for the slashdot type (not being elitist... just pragmatic) it would be nice to be able to get to a support call where you either get to skip the preamble (see above list) and immediately discuss symptoms and possible causes along with solutions.

So, bottom line, I see the problem being:

• Companies create support centers as a first line of defense by either:
  • creating so much FUD and confusion to the uninitiated consumer they give up in frustration and just go on living with their "problem"
  • arguing with the more savvy consumers on the merits of whether or not warranty service applies
• Companies creating support centers in the belief that support is doable with non-expert staff using only a flow-chart of "troubleshooting techniques" and "countermeasures"
• Companies create support centers ignoring that often consumers are quite technical and have done most of the work up front and are ready to talk shop by the time they call a support line. This lack of consideration to the consumer ends up costing the company in good will, the company in time spent for support that really isn't, and for the consumer in wasted time jumping through unnecessary initial hoops they'd already considered.

Oh, and I don't see this getting better soon, if ever. Sigh.

Yet another Apple slam

[EvilStein]

I worked as an AppleCare support rep. If a customer doesn't feel that the issue is resolved (and clearly these "testers" didn't) then all they have to do is ask to have the issue escalated.
About the broken Wifi "test" - there are *so many* brands of 802.11 base stations out on the market that if you're not getting any network information at all, and the computer thinks everything is ok (Tiger has a nice "Network Diagnostic" utility) then suggesting that you contact the manufacturer of the 802.11 base station certainly isn't a bad thing at all. It's a *third party product* (I'm goign to assume that they did't try with an Airport Base Station, because if they did, Apple would have addressed it.) and Apple's policy was to not even try to support 3rd party products.
The write up was pretty vague, and that's sad.