Slashdot Mirror
Consumer Reports Updates Its MacBook Pro Review (consumerreports.org)
Reader TheFakeTimCook writes: Last month, the new MacBook Pro failed to receive a purchase recommendation from Consumer Reports due to battery life issues that it encountered during testing. Apple subsequently said it was working with Consumer Reports to understand the results, which it said do not match its "extensive lab tests or field data." According to an article from Consumer Reports, Apple has since concluded its work, and says it learned that Consumer Reports was using a "hidden Safari setting" which triggered an "obscure and intermittent bug" that led to inconsistent battery life results. With "normal user settings" enabled, Apple said Consumer Reports "consistently" achieved expected battery life. Apple stated: "We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life." Apple said it has fixed the Safari bug in the latest macOS Sierra beta seeded to developers and public testers this week.
affectionately termed AppleCore
... battery life wasn't really the straw that broke the proverbial camel's back anyway.
By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
They didnt update their review yet
I'm expecting a lot of Consumer Report Hate right now
Skimp on QA to your peril.
There really were flaws in the tests. It does surprise me a bit that this setting was enabled, but it also doesn't make sense that consumer reports and apple's self reported battery life numbers would be so far off. As manufacturers go, Apple has a pretty good reputation for not hyper inflating their battery numbers.
I'm glad that Consumer reports was willing to look at this again and re-test these machines with settings an actual user would have enabled.
"In America, first you get the sugar, then you get the power, then you get the women..." -H. Simpson
Why was this setting enabled in the first place? It sounds like the setting in question isn't on by default, nor should it be. Why would Consumer Reports have enabled it for their testing? Unfortunately, it will certainly be a bigger deal that Consumer Reports did not recommend the MacBook Pro than it will be that the recommendation has been amended. The damage may be done in this case, but should Consumer Reports be trusted going forward? It raises the question of what other unusual things may be done in the testing of other products, and if that might affect the quality of the recommendations. Perhaps it's an isolated case, but it's confusing as to why this occurred at all.
No cache and redrawing icons can take off 9 hrs? Wow
Yeah, eat my shorts Apple!
> Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage
Developing web sites on a Mac does not reflect real-world usage. Gotcha.
1 out of 5 stars. Newest MBP is missing key.
They did not update their review, they posted that they may.
By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi.
Yes, that is the definition of a cache...
In normal use you don't sit around reloading the same page all day
You don't? Are you seriously saying you do not visit several sites multiple times in a day?
Not to mention, lets say some sites you only go to ever so often - say Amazon, I go to a few times a month. A cache is still useful there for many of the page components and CSS files do not change much over time.
In fact I would say 95% of the sites I visit in a day - news sites, recipes, various blogs, Slashdot, etc. benefit from caching, because they are places with logos and things that don't change much if at all over time. There are just not that many times I'm visiting a new site in a day.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
So that's what you need to do to get Apple to fix your bug. A couple years I found a bug in their version of sqlite3, it stopped accepting international characters (chinese, japanese, specifically). I tried to submit a bug via their bug reporting console, and I got this error message. So I sent an email to the address listed there, explaining the situation, and I got a response,
"Please report that through our bug console." The console was still broken.
"First they came for the slanderers and i said nothing."
.. - one that returns a USB and HDMI ports as well.
If there's a setting to make the dynamic touchbar a static set of function keys I'm used to using, vs. having to wonder what buttons/functionality is there at any given point in time, that would be helpful as well.
Disabling or clearing cache is common in testing/benchmarking to measure/simulate performance loading fresh pages or otherwise doing something the first time, but repeating the test for more accurate results.
CR probably had a list of 12-25 commonly used web sites they used as example sites, and loaded each of those sites many times rather than loading hundreds of different sites. If you want to load 12 web sites ten times, to simulate loading 120 web sites, you need to disable cache to simulate loading 120 different sites.
It sounds like Safari had an issue where loading a site with cache off did *not* work the same as loading a fresh copy of the site for the first time, with the cache on. You'd expect the two scenarios to be more or less equivalent, and they normally are, but not always. Safari was probably doing the same thing in normal operation that IE used to do - first download it to cache, then immediately display the cached version rather than displaying first, then saving to cache). With cache disabled, it would need to use a completely different code path.
MacOS isn't so bug free, the setting of turning off the caching triggered a icon reload error. Next time Apple test your software better.
It has an infinity of function row keys. That is the opposite of missing.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Skid marks are still skid marks!
"You're testing it wrong."
"...you don't sit around reloading the same page all day..."
Maybe you don't but most sites I visit have some sort of auto-refresh on them unless you disable it... especially news sites. So maybe you don't sit there hammering the refresh button... but the web page you are on is. ;) Posting anon to keep mods.
Did that cost Apple?
In normal use you don't sit around reloading the same page all day
You don't? Are you seriously saying you do not visit several sites multiple times in a day?
No, I'm saying I don't sit around reloading the same page all day. I might reload a number of pages, but most of my browsing is to new content. Sure, a lot of the CSS and images and the like will be cached, but that's quite different from the whole page being cached entirely.
Clearly Apple is trying to minimize network traffic, because wifi uses a lot of energy. That's a good thing to do, a perfectly reasonable optimization, but it does distort automated test results. To make tests repeatable and fair Consumer Reports has to set up its own web pages and repeatedly load them, because real web pages change and disappear so tests done in 2016 might not be repeatable in 2017.They deliberately disabled the cache to make sure that traffic was actually generated, to check wifi power consumption as part of the test.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
What is this Hidden Safari Setting?
You visited Slashdot at least five times today (evidence.) Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.
You are not alone. This is not normal. None of this is normal.
Does Firefox have this bug out of the box? I only use Firefox and the battery on my mac only lasts 4 hours. Tested three times now.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
How are morons on here so smart and so fucking stupid. No one is saying people don't benefit from caching. They are only arguing it's a valid PERFORMANCE METRIC to test hours of battery life pulling always new content. So worst case you go to all new Web pages all day you get X hours. That's a reasonable metric. Then you would expect more from that point.
CR reported battery life results in line (or better than) Apple's advertising when using Chrome. Did CR disable the cache when they tested using Chrome?
Don't fucking hide your settings or features.
But they clutter the perty display!
They how the fuck are we supposed to use or debug them!
They're intuitive and they just work!
Bullshit!
If I visit the first page for a site to retrieve all of the graphics-intensive formatting stuff, then as I browse thirty more pages on that same site I do not have to re-download that stuff because it's cached then that could make for a difference.
That depends on a particular user's browsing habits. You might "browse thirty more pages on" Slashdot while reading stories and writing comments but hit only one document on a site when reading each story's featured article. Or you might "browse thirty more pages on" a web search engine while performing queries but hit only one document on a site when reading each result.
Sounds more like Apple's updating CR's review. Not much in there about what CR thinks about all this.
Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.
I can't speak to the CSS because that is normally invisible, but I can speak to "image files that make up every page". They are invisible here. A banner at the top, a couple of small icons at the bottom, but other than that -- what images?
When I visit a /. page I explicitly refresh the page because I want to see everything that is new. If I wanted to see the same things over and over I'd just print the page and tack it up on the wall.
That's the profile I use on the couch and at my bar. Unplugged, surfing the internet, and Pandora or YouTube playing music.
They are only arguing it's a valid PERFORMANCE METRIC to test hours of battery life pulling always new content.
If it's not what anyone would ever do, how is that a valid metric? After about three days of using a laptop most of my browser usage is going to hit the cache in some way.
Especially if you are not using the SAME METRIC to compare other laptops with - which is the point of the article, that a bug in Safari was disabling caching. So MacBook Pros were being comparing with all caching off, to laptops that had all caching enabled - do you think that is a fair comparison?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
First of all, yes, the implication was that nobody benefits from caching. And secondly, if you're getting every damned thing from the net and nothing from the cache, your network usage is orders of magnitude higher than "normal". AmiMojo is suggesting it's "cheating" to base power usage metrics in part on cached data. But most of the data that built the webpage you're looking at right now came from your cache. You think the HTML - the only dynamically generated part - made up most of the bytes of data that made up the page? Think again.
Worse still, you're focusing on something that's relatively meaningless. Do you think your Wifi card draws massive amounts of power when it's receiving data? I actually don't know if the difference in power between Wifi "listening" and Wifi "receiving" is more than the difference between "SSD drive idling" and "SSD drive sending data", and I suspect you don't either.
But I can say, without too much doubt, that the GPU and CPU suck much, much, more power than both of those put together when they're rendering a webpage. Hell, I just leave Twitter open in a tab before I go to work and my 2011 i7 laptop at home's fans are whirring by the time I get home.
On the face of it, calling it "cheat mode" when you instruct a laptop to cache webpages in a perfectly normal way is ludicrous hyperbole. The cache makes little or no difference to power usage, and it's normal behavior to have it enabled. It's also irrelevant - as others have pointed out - to why this particular benchmark turned out to be flawed, which was a bug in the no-cache mode, not something to do with caching specifically.
You are not alone. This is not normal. None of this is normal.
CR will reverse their findings. There isn't a company out there that apple would not bride, bully or coerce to make them see the error of their ways.
Try "View Page Info" (if you're using Firefox), it'll list the media embedded on the page. These vary from the Slashdot logo, the zoo icons, the social media icons, etc.
There's more than you think.
You are not alone. This is not normal. None of this is normal.
Apple: "Consumers don't want a touch screen or ports; they'll settle for a touch bar and dongles."
Every other laptop maker: "TOUCHSCREENS! PORTS! FRACTION OF THE PRICE!"
Finding God in a Dog
Likely not a whole lot. Couple days of a dev's salary to find and fix the problem. Sounds like its being included in an already-planned update to Safari so there shouldn't be much if anything in the way of extra distribution costs.
CR may have charged a consulting fee if it took the people on their side more than a couple emails but that wouldn't add up to significant amounts either -- at least not when scaled against the depths of Apple's pockets.
There's more than you think.
I used about:config to disable image loading (preferences.image.default = 2) and I see only one tiny difference between the previous version of this page and the one I see now: the two little 64x64 icons at the bottom of the page attached to the previous and next stories are gone. That's all.
Oh, and the "Slashdot" logo is gone. That's not very large compared to the page itself, which is useless if I use a cached version.
Well, OK, in your specific case you probably don't load much that's cached, but you must admit, you're not really doing the same things most people are!
You are not alone. This is not normal. None of this is normal.
I've tried using that keyboard several times and it's just doesn't make the cut. If they would have left the thickness they could have had better keyboard travel and better battery life.
Dumbass! The shitty webdevs use CDNs to load jQuery, Bootstrap, FontAwesome, at al for their generic looking websites. Unneeded javascript libraries and Internet fonts are big reasons why the payload for the average web page has grown to over 2MB these days. But, the not so bad news is that these crappy javascript libraries and fonts are often cached because the shitty webbev only has to copy and paste the URL into their source code--this is presented as the "best" solution, so that's what these shitty webdevs do. The CDNs are configured correctly and cache these static resources for a long time.
Now, when you visit shitty site A and it requests jQuery from the CDN. Later you visit shitty site B and it requests jQuery from the same CDN, but it does not need to be fetched because it is in your cache. Hooray!
This is not a setting used by customers
Even Apple is telling us to just use Chrome.
Proud neuron in the Slashdot hivemind since 2002.
INTERMITTENT bug.
all bugs are intermittent, otherwise nothing would ever function
Something like 16 hours? If the bug just stopped bugging for one test, the high result should still be closed to advertised result, not way beyond it.
Likely not a whole lot. Couple days of a dev's salary to find and fix the problem.
Likely a lot, which is why they went out of their way to work with CR and find the root cause. Like them or not, CR has a pretty large user base. And those non-techie type of users are exactly where Apple targets their products.
Battery life is not something apple is good at managing. There last iOS update to my iPhone 6s destroyed my battery life and basically makes my phone unusable. It went from a few days per charge to barely 6 hours with a little use. Resetting all settings and content does nothing and they offer no way to roll back to the last iOS version. I wish they would spend more time on testing and fixes before they release an iOS and less time on removing important features.
CR didn't "Updates Its MacBook Pro Review", they instead made a post explaining Apples claimed reasoning for the tests.
The post says "Once our retesting of the MacBook Pro’s batteries is complete, we will report back with our update and findings."
So they are going to install the update for Safari and re-run the tests, after those tests are ran they will then give a update. For all we know the tests could come back the same. The bug that was fixed may not been the whole source of the battery drain, so until the results of the re-testing is done on the patched Safari, there is no update to the review.
Obviously.
I'd argue that it isn't. The best pro notebook they ever made was the last one that allowed you to upgrade the ram and hard drive, and replace the battery. That's "pro".
Their latest macbook pro might be the best appliance they've ever made, but it isn't "pro".
First of all, yes, the implication was that nobody benefits from caching. And secondly, if you're getting every damned thing from the net and nothing from the cache, your network usage is orders of magnitude higher than "normal". AmiMojo is suggesting it's "cheating" to base power usage metrics in part on cached data. But most of the data that built the webpage you're looking at right now came from your cache. You think the HTML - the only dynamically generated part - made up most of the bytes of data that made up the page? Think again.
Unfortunately, this is true. Web pages done right though (no bloated images, javascript, flash, etc) the dynamic part of a web page should be the largest part.
On the face of it, calling it "cheat mode" when you instruct a laptop to cache webpages in a perfectly normal way is ludicrous hyperbole. The cache makes little or no difference to power usage, and it's normal behavior to have it enabled. It's also irrelevant - as others have pointed out - to why this particular benchmark turned out to be flawed, which was a bug in the no-cache mode, not something to do with caching specifically.
Agreed. When I was in software testing years ago (testing web site performance), we intentionally tested and recorded metrics in two passes - once with caching on and once with caching off. Of course we were testing for the performance of the web site on a given device, and not the performance of the device, but given that CR was changing a setting that most consumers wouldn't change, they could have done something similar.
If they're going to rate a consumer device, they should first test the device with the settings from the manufacturer. If they choose to "optimize" the device to see if they can squeeze more out of it, great, but it should be noted as a non-standard configuration.
Can we all just agree this updated review is paid for?
It's called the Apple Wallet. Once it's opened and the correct items have been transferred recommendations are granted for Apple's broken shit.
I'm not that stupid anymore. Others are making devices that have the plugs I can use today with my $1000 in peripherals.
Never again ... until a standard USB3 port and HDMI ports are back. Don't get me started on the headphone jack and DELETE key.
I do appreciate having a USB-c port, but not at the expense of normal USB2/3 ports.
I've never seen a DisplayPort monitor. Don't own any. Really don't see the point.
For some reason, part of Apple's business model it to force customers to buy converters or new peripherals so the plugs fit.
I'd argue that it isn't. The best pro notebook they ever made was the last one that allowed you to upgrade the ram and hard drive, and replace the battery. That's "pro".
Their latest macbook pro might be the best appliance they've ever made, but it isn't "pro".
The max. RAM is 16 GB. What "Pro" isn't going to configure it that way out of the box? So, there's one of your three arguments nicely refuted. Let's try the other one.
You can replace the battery in the new MBP. Considering it is something that is done maybe ONCE in the lifetime of the product, having it be a little inconvenient is not a reason to say that it is an "appliance". And, BTW, Apple doesn't gouge for battery replacement. In fact, they charge little enough that I simply can't imagine wanting to do it myself to "save money." Ok, so now let's try for that last objection...
Can't upgrade the hard drive. Well, that's something that, now that most laptops are going SSD, we're going to start seeing more and more. But no one will do any hand-wringing about them; because they aren't Apple, and so nobody cares what the other guys do. Again, survey says: "Most people, by and large, never upgrade the storage in their laptops." And the more "Pro" a laptop-user is, the LESS likely they are to be able to store their DATA on an LAPTOP's internal drive. Even at 2 TB, editing video projects is going to make that 2 TB seem crowded in a year or so. And with USB-C/TB3 available, there really isn't a speed penalty for having an external drive enclosure. And, although the TB3 enclosures are still a bit pricey (and hard to find, both which might change now that everyone and his dog is switching to USB-C/TB3), USB-C enclosures/drives are plentiful and available for the $20-30 range (driveless) on Amazon, like this one. So, it sounds like 3 for 3 to me.
You are out of gas...
Apparently not
... our battery tests are not designed to be a direct simulation of a consumer's experience. Rather, we look to control as many variables as possible, then perform a test that gives potential users a reasonable expectation of battery life when a computer's processors, screen, memory, and antennas are under a light to moderate workload. This test has served as a good proxy for battery life on the hundreds of laptops in our ratings.
CR
In other words, CR was lazy and disabled the browser cache instead of developing a test representative of real life users -- their target audience. Sure, they found a bug, but not one that an ordinary user would ever encounter (is this mode even "supported"?).
I assume CR ran their tests against an in-house web server with some sort of synthetic content rather than consumer web sites (such as Facebook, Amazon, gmail etc) -- else they would be unable to have repeatable tests (perhaps Amazon would be loaded at some times and not others or Facebook would change the layout/design of their pages and invalidate all testing that CR had already done). CR should have done some real world monitoring (they could probably dig through their logs to see how their own employees access external web sites as a starting point and work from there) to determine representative patterns and built their web server and client side tests to replicate the same patterns. This pattern almost certainly would include a mix of cached and uncached accesses.
It's interesting that CR uses the "default browser" on each machine for its tests because they feel that's what the typical user would do, but then go and change obscure settings in that browser to make it behave in a non-default way. In the real world, I'll bet more Mac users in CR's audience (at least those that would rely on them for advice on a computer) use a browser other than Safari than use Safari but screw around with "hidden" settings intended for developers.
I'm not a fan of Apple, but CR really is the one with egg on their face here.
In general, I'm becoming increasingly distrusting of CR's results -- they should spend more of their resources testing products properly and less time editorializing on political topics.
Why is there an "insightful" mod and why isn't it "-1"? If I wanted insight, I wouldn't be reading
in the 1970s CR ran a review of bicycles; one of the tests was how far the bicycle rolled without pedaling
In the next issue of Bicycling Magazine (which was, before it was rodalized, a great magazine) they pointed out that how far a bike roles is stupid, cause it is sensitive to tire pressure
Next bike review in CR didn't have that test.....
All hail the new Internet Explorer.
and your full of something.
n/t
Because of the safari bug, the actual benchmarking that CR's test did accurately was measure power consumption of using the developer mode.
So, the report stands without need of correction.
Wonder how much Apple paid to have CR rerun their tests with the "special patch" from Apple? Enough for a better report?
Apple has the cash, I'm guessing they bought the better report.
Apple: so now, please click this cache feature. (while sliding $10,000 across table).
Consumer Reports: ok
Apple: and now, please update your findings (sliding another couple of 10k-packs across the table)
Apple: you of course have no memory of this (pointing at the money) "conversation". (The Force handwave)
Consumer Reports: (nods, eyes glazed over) Yes, master.
Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.
I can't speak to the CSS because that is normally invisible, but I can speak to "image files that make up every page". They are invisible here. A banner at the top, a couple of small icons at the bottom, but other than that -- what images?
When I visit a /. page I explicitly refresh the page because I want to see everything that is new. If I wanted to see the same things over and over I'd just print the page and tack it up on the wall.
Okay. Measure the time it takes to load the Slashdot homepage. Now turn off caching and try again. Report. Especially if any of the icons next to each story changed - which each needs a TCP connection to the web server to download.
Of course news about a fake are Fake News.