Slashdot Mirror


Consumer Reports Updates Its MacBook Pro Review (consumerreports.org)

Reader TheFakeTimCook writes: Last month, the new MacBook Pro failed to receive a purchase recommendation from Consumer Reports due to battery life issues that it encountered during testing. Apple subsequently said it was working with Consumer Reports to understand the results, which it said do not match its "extensive lab tests or field data." According to an article from Consumer Reports, Apple has since concluded its work, and says it learned that Consumer Reports was using a "hidden Safari setting" which triggered an "obscure and intermittent bug" that led to inconsistent battery life results. With "normal user settings" enabled, Apple said Consumer Reports "consistently" achieved expected battery life. Apple stated: "We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life." Apple said it has fixed the Safari bug in the latest macOS Sierra beta seeded to developers and public testers this week.

30 of 246 comments (clear)

  1. Meh... by Anonymous Coward · · Score: 5, Insightful

    ... battery life wasn't really the straw that broke the proverbial camel's back anyway.

  2. So they didn't enable cheat mode by AmiMoJo · · Score: 2, Insightful

    By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.

    --
    const int one = 65536; (Silvermoon, Texture.cs)
    SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    1. Re:So they didn't enable cheat mode by TheFakeTimCook · · Score: 4, Informative

      By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.

      Read, then post:

      Disabling the Cache did more than just cause Reloading each time. Apparently, it ALSO triggered an intermittent bug in Safari that caused REPEATED loading of "icons" from the page.

      Apparently, THAT is what burned the battery. Very similar to a "runaway process", like I (and others) originally postulated.

      So, you can safely remove that extra layer of tinfoil, Hater.

    2. Re:So they didn't enable cheat mode by TheFakeTimCook · · Score: 2, Informative

      Apple are saying there is an obscure and intermittent bug that is triggered by that testing practice.

      Presumably that bug can be fixed.

      (As a total aside, in my daily use, I visit about ten websites more or less constantly -- I don't even mean social media. Caching very definitely is extending battery life for me. For those who use social media sites, or any webkit-based app, caching is helping enormously.)

      It already HAS been fixed, and will be released in the next Sierra update (which is due out pretty much any day now).

    3. Re:So they didn't enable cheat mode by Shalian · · Score: 2

      What MacBook Pro model has a spinning hard drive?

    4. Re:So they didn't enable cheat mode by saloomy · · Score: 3, Insightful
      Sounds to me (with the limited info in the summary) that developer mode was constantly reloading page icons?

      Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab

      That would obviously chew up battery life downloading icons over and over again, chewing up CPU cycles to refresh the icon on the window, and chew up wifi power because its easy to see how a stuck loop re-downloading icons could cycle thousands or millions of times.

      Good for Consumer Reports for sticking to their guns, seeing an issue, reporting the issue, and forcing Apple to fix it. It's obscure sure. But many developers probably use Safari Developer Mode to work on their projects, and this will help them.

    5. Re:So they didn't enable cheat mode by Altrag · · Score: 3, Insightful

      From how I read it, CR released the report.. Apple went "wtf that's not right" and then asked CR to help them solve a problem. This sounds like fairly standard debugging practices to me -- discover a problem, figure out how to duplicate it, and then fix it.

      The fact that it was CR that ran into the problem rather than some random guy on the internet posting a rant on Reddit is just luck and maybe makes for a "fun" story for conspiracy nuts, but it doesn't indicate that either party did anything that they shouldn't have done or was being shady or misrepresenting facts or anything like that.

      Sure CR added an update to their original report but didn't try to redact the problem out of existence. It seems to me to have been handled completely appropriately by both parties.

    6. Re:So they didn't enable cheat mode by Just+Some+Guy · · Score: 4, Insightful

      But many developers probably use Safari Developer Mode to work on their projects, and this will help them.

      Yes, but those developers don't get their recommendations from Consumer Reports. That magazine's audience would never have encountered that bug.

      Obligatory car analogy: say they're testing a Ford Focus. They disable its antilock brakes so that a professional driver can get its best-case dry pavement stopping distance. Along the way, the find an OBD-II bug that causes the brakes to take twice as long to stop the car. They report the bad results instead of the normal, expected values. Yes, their test was correct! It found a bug that needs to be fixed. However, the only people who would ever see that bug are the exact ones who'd notice something was wrong and be able to troubleshoot it. You and I aren't ever going to disable our antilock brakes, even if a test engineer might.

      I think that's kind of what happened here. Again, yes, they legit found a bug. My problem with it is that they reported the buggy results instead of the actual ones that a normal non-developer would see. A developer would notice their battery draining in a fourth the expected time and that it only happened when they were debugging in Safari, so they probably wouldn't even be significantly affected by the bug.

      --
      Dewey, what part of this looks like authorities should be involved?
    7. Re:So they didn't enable cheat mode by Obfuscant · · Score: 2

      Was it a stress test? Or a typical usage test?

      Neither. It is a comparison test that removes variables as much as possible so all laptops are on the same playing field. From CR itself:

      ... our battery tests are not designed to be a direct simulation of a consumer's experience. Rather, we look to control as many variables as possible, then perform a test that gives potential users a reasonable expectation of battery life when a computer's processors, screen, memory, and antennas are under a light to moderate workload. This test has served as a good proxy for battery life on the hundreds of laptops in our ratings.

      If your testing relies on going to a bunch of web pages that may or may not have a lot of cached content depending on when the test is done, or the caching is set differently between models by default, then the caching is a variable that could make a model with lower battery capacity show up as being better than a model with more capacity. Battery life is not an absolute measure that you can depend on because users do different things. But battery comparisons are relative -- a model with double the lifetime in CR's tests should last twice as long as the other model in real life, too. That's what "a good proxy" means.

    8. Re:So they didn't enable cheat mode by Solandri · · Score: 2

      My problem with it is that they reported the buggy results instead of the actual ones that a normal non-developer would see

      So by your reasoning, if a car manufacturer accidentally made a bug which caused the engine to cheat on diesel emissions tests, it's actually the EPA's fault for not designing their test to more accurately mimic how people use their cars in real life?

      Everyone seems to be trying to spin this for or against Apple / Consumer Reports. The no-spin version is that CR was using an industry-accepted practice to simulate web browser usage in a repeatable manner necessary for accurate testing. Apple accidentally created a bug which could cause excessive battery drain during these simulations (we know it's a new bug because previous CR tests didn't have any problems). The CR test occasionally triggered the bug. They and Apple worked together to track down the bug. Apple fixed it. CR is re-testing now. End of story. Nothing to see here. Move along.

  3. Re:Should Consumer Reports be trusted? by Chmarr · · Score: 4, Informative

    Read the article. It answers your question.

  4. Inaccurate headline by celeb8 · · Score: 5, Informative

    They did not update their review, they posted that they may.

  5. Don't most people revisit the same site many times by SuperKendall · · Score: 2, Insightful

    By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi.

    Yes, that is the definition of a cache...

    In normal use you don't sit around reloading the same page all day

    You don't? Are you seriously saying you do not visit several sites multiple times in a day?

    Not to mention, lets say some sites you only go to ever so often - say Amazon, I go to a few times a month. A cache is still useful there for many of the page components and CSS files do not change much over time.

    In fact I would say 95% of the sites I visit in a day - news sites, recipes, various blogs, Slashdot, etc. benefit from caching, because they are places with logos and things that don't change much if at all over time. There are just not that many times I'm visiting a new site in a day.

    --
    "There is more worth loving than we have strength to love." - Brian Jay Stanley
  6. How to get Apple to fix a bug by phantomfive · · Score: 5, Interesting

    So that's what you need to do to get Apple to fix your bug. A couple years I found a bug in their version of sqlite3, it stopped accepting international characters (chinese, japanese, specifically). I tried to submit a bug via their bug reporting console, and I got this error message. So I sent an email to the address listed there, explaining the situation, and I got a response,
    "Please report that through our bug console." The console was still broken.

    --
    "First they came for the slanderers and i said nothing."
  7. Look for another Safari setting by bjdevil66 · · Score: 5, Insightful

    .. - one that returns a USB and HDMI ports as well.

    If there's a setting to make the dynamic touchbar a static set of function keys I'm used to using, vs. having to wonder what buttons/functionality is there at any given point in time, that would be helpful as well.

    1. Re:Look for another Safari setting by mangamaster03 · · Score: 2

      I asked about that at the apple store. You can actually order the new laptop with the normal function keys instead of the dynamic touchbar from their website. It's on their website, and $400 cheaper than the first one with the touchbar.

      USB and HDMI ain't happening though...

  8. Re:What do you know. by Anonymous Coward · · Score: 5, Informative

    That isn't what the article says.

    There were no flaws in the tests. The tests were tickling an obscure bug, which Apple has now fixed.

    Consumer Reports are retesting with the patch.

    This is good news for everyone, surely? Methodology better explained, bug found.

  9. Re:What do you know. by sjames · · Score: 5, Informative

    The test was good. It was Safari that had the flaw. Thanks to the Consumer Reports test it was revealed so it can be fixed.

    As for actual user, you do know that Macs are popular among web designers?

  10. OK, I'll Ask the Question by LifesABeach · · Score: 2

    What is this Hidden Safari Setting?

    1. Re:OK, I'll Ask the Question by Mordaximus · · Score: 5, Funny

      I believe it's called "Do Not Load TFA/TFS." Apparently it saves battery life because some can't be bothered to read the entire summary.

  11. Re:Don't most people revisit the same site many ti by squiggleslash · · Score: 2

    You visited Slashdot at least five times today (evidence.) Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.

    --
    You are not alone. This is not normal. None of this is normal.
  12. Re:What do you know. by omnichad · · Score: 5, Insightful

    If you leave caching on, you'd can't test by loading a handful of sites over and over. They would have to have a script of non-repeating real-world web sites that would handle 8-10 hours of battery life.

    By limiting to a handful of sites and disabling a cache, you can make a more consistent and repeatable test. While you're testing real-world (and real web sites), you still want to test in a repeatable and somewhat verifiable way.

  13. Consumer Reports updates? by Chris+Mattern · · Score: 4, Insightful

    Sounds more like Apple's updating CR's review. Not much in there about what CR thinks about all this.

  14. Re: Don't most people revisit the same site many t by squiggleslash · · Score: 2, Insightful

    First of all, yes, the implication was that nobody benefits from caching. And secondly, if you're getting every damned thing from the net and nothing from the cache, your network usage is orders of magnitude higher than "normal". AmiMojo is suggesting it's "cheating" to base power usage metrics in part on cached data. But most of the data that built the webpage you're looking at right now came from your cache. You think the HTML - the only dynamically generated part - made up most of the bytes of data that made up the page? Think again.

    Worse still, you're focusing on something that's relatively meaningless. Do you think your Wifi card draws massive amounts of power when it's receiving data? I actually don't know if the difference in power between Wifi "listening" and Wifi "receiving" is more than the difference between "SSD drive idling" and "SSD drive sending data", and I suspect you don't either.

    But I can say, without too much doubt, that the GPU and CPU suck much, much, more power than both of those put together when they're rendering a webpage. Hell, I just leave Twitter open in a tab before I go to work and my 2011 i7 laptop at home's fans are whirring by the time I get home.

    On the face of it, calling it "cheat mode" when you instruct a laptop to cache webpages in a perfectly normal way is ludicrous hyperbole. The cache makes little or no difference to power usage, and it's normal behavior to have it enabled. It's also irrelevant - as others have pointed out - to why this particular benchmark turned out to be flawed, which was a bug in the no-cache mode, not something to do with caching specifically.

    --
    You are not alone. This is not normal. None of this is normal.
  15. True dat by MAXOMENOS · · Score: 3, Insightful

    Apple: "Consumers don't want a touch screen or ports; they'll settle for a touch bar and dongles."

    Every other laptop maker: "TOUCHSCREENS! PORTS! FRACTION OF THE PRICE!"

    1. Re: True dat by Type44Q · · Score: 4, Funny

      1/10th the courage!

    2. Re:True dat by torkus · · Score: 2

      You haven't actually looked at the available options then...

      --
      You can get rich if you own a politician, but you have to be rich to buy one in the first place.
  16. Re:"Real-world usage" by TheFakeTimCook · · Score: 2

    I didn't assert 'No one develops websites on the Mac', all the websites hosted on my servers are developed on the Mac.

    But the number of people who do this is Much Less Than the total Mac user population.

    Furthermore, few people who develop websites on any platform get their tech advice from Consumer Reports.

    But then, when you can't produce a useful thought, insults work just fine.

    Rather, when you don't include pertinent information, and then simply post what sounds like an inflammatory comment, you invite insults.

  17. Re:MacBook had bug is the real issue by farble1670 · · Score: 2

    Next time Apple test your software better.

    I don't care if software has bugs (because it always does). I care if they get fixed in a timely manner.

  18. Re:What do you know. by TheFakeTimCook · · Score: 2

    The idea that your laptop's battery life will significantly extend or deteriorate based upon whether the data used to render a webpage (or video) is coming from an SSD or Wifi is absurd.

    Yes it is; when everything is operating right.

    But, Apple has found that, disabling the Cache in Safari uncovers an INTERMITTENT bug that causes assets from an ALREADY LOADED web page to be CONTINUOUSLY RELOADED, and THAT is what causes the excessive power-drain.

    DO try to at LEAST read TFS...