Slashdot Mirror
Consumer Reports Updates Its MacBook Pro Review (consumerreports.org)
Reader TheFakeTimCook writes: Last month, the new MacBook Pro failed to receive a purchase recommendation from Consumer Reports due to battery life issues that it encountered during testing. Apple subsequently said it was working with Consumer Reports to understand the results, which it said do not match its "extensive lab tests or field data." According to an article from Consumer Reports, Apple has since concluded its work, and says it learned that Consumer Reports was using a "hidden Safari setting" which triggered an "obscure and intermittent bug" that led to inconsistent battery life results. With "normal user settings" enabled, Apple said Consumer Reports "consistently" achieved expected battery life. Apple stated: "We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life." Apple said it has fixed the Safari bug in the latest macOS Sierra beta seeded to developers and public testers this week.
affectionately termed AppleCore
... battery life wasn't really the straw that broke the proverbial camel's back anyway.
By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
Yeah, eat my shorts Apple!
How many Mac users develop web sites?
Read the article. It answers your question.
Disclaimer, I do not use Apple products or work in Consumer Reports.
Sent from my ASR33 using ASCII
They did not update their review, they posted that they may.
By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi.
Yes, that is the definition of a cache...
In normal use you don't sit around reloading the same page all day
You don't? Are you seriously saying you do not visit several sites multiple times in a day?
Not to mention, lets say some sites you only go to ever so often - say Amazon, I go to a few times a month. A cache is still useful there for many of the page components and CSS files do not change much over time.
In fact I would say 95% of the sites I visit in a day - news sites, recipes, various blogs, Slashdot, etc. benefit from caching, because they are places with logos and things that don't change much if at all over time. There are just not that many times I'm visiting a new site in a day.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
So that's what you need to do to get Apple to fix your bug. A couple years I found a bug in their version of sqlite3, it stopped accepting international characters (chinese, japanese, specifically). I tried to submit a bug via their bug reporting console, and I got this error message. So I sent an email to the address listed there, explaining the situation, and I got a response,
"Please report that through our bug console." The console was still broken.
"First they came for the slanderers and i said nothing."
.. - one that returns a USB and HDMI ports as well.
If there's a setting to make the dynamic touchbar a static set of function keys I'm used to using, vs. having to wonder what buttons/functionality is there at any given point in time, that would be helpful as well.
Why was this setting enabled in the first place?
Probably because they use some kind of automated testing framework, so they don't need to have an actual human sitting there the entire time clicking on things.
"First they came for the slanderers and i said nothing."
Disabling or clearing cache is common in testing/benchmarking to measure/simulate performance loading fresh pages or otherwise doing something the first time, but repeating the test for more accurate results.
CR probably had a list of 12-25 commonly used web sites they used as example sites, and loaded each of those sites many times rather than loading hundreds of different sites. If you want to load 12 web sites ten times, to simulate loading 120 web sites, you need to disable cache to simulate loading 120 different sites.
It sounds like Safari had an issue where loading a site with cache off did *not* work the same as loading a fresh copy of the site for the first time, with the cache on. You'd expect the two scenarios to be more or less equivalent, and they normally are, but not always. Safari was probably doing the same thing in normal operation that IE used to do - first download it to cache, then immediately display the cached version rather than displaying first, then saving to cache). With cache disabled, it would need to use a completely different code path.
Why enable it?
It was enabled because that's their standard testing procedure. They test all laptops with caching turned off. That would mean that turning caching on for the test would be favouritism to Apple.
Apple in their turn have declared that having caching turned off revealed a software bug and have now released a fix.
So, it turns out that Consumer Reports not only tested fairly, but that Apple had something genuinely wrong. The report was therefore entirely justified.
That isn't what the article says.
There were no flaws in the tests. The tests were tickling an obscure bug, which Apple has now fixed.
Consumer Reports are retesting with the patch.
This is good news for everyone, surely? Methodology better explained, bug found.
Or, assuming they control the site (implied by your use of new pagename/asset/etc) send cache control headers so that no caching is done of some of the assets.
The article say they disable cache to facilitate consistent behavior for repeating a test.
I'll admit the phrasing makes it sound all weird and exotic, but all they did was disable cache.
XML is like violence. If it doesn't solve the problem, use more.
and Streamed Content I'd argue is a much bigger % of people's traffic (volume wise (hits wise probably less so... but that is the nature of the beast))
That seems very unlikely for the usage profile of being on batteries. Sure, we all stream when we're at our desks, but I don't know how that translates to out-and-about traffic patterns. Restaurant Wi-Fi is often so spotty that you wouldn't try to listen to live streams. I doubt many people are consuming lots of video content from their cell phone tethers.
Dewey, what part of this looks like authorities should be involved?
The test was good. It was Safari that had the flaw. Thanks to the Consumer Reports test it was revealed so it can be fixed.
As for actual user, you do know that Macs are popular among web designers?
Why was this setting enabled in the first place? It sounds like the setting in question isn't on by default, nor should it be. Why would Consumer Reports have enabled it for their testing?
As much as I would like to blame CR for this, it comes down to practicality.
They "locally" host the webpages they download, in order to factor-out stuff like internet traffic from their tests (and possibly to be nice to the websites that they have "targeted"), and so, there is literally no other way they could force a "page reload" (which they do for ALL laptops being tested) without disabling the browser cache.
And, as Apple said, the real problem was that, when CR did that, it triggered an intermittent bug in Safari that caused repeated re-loading of information from the page, even AFTER it was already loaded, which WAS a "shouldn't happen" condition that sucked-down the battery.
Apple has fixed the problem. CR has done some preliminary re-testing, and feels confident that Apple's explanation and fix warrants a full re-test.
Good for them!
I don't disable the cache, I clear it when I need to. I also don't use just Safari, but every browser needs to be tested separately.
Did that cost Apple?
In normal use you don't sit around reloading the same page all day
You don't? Are you seriously saying you do not visit several sites multiple times in a day?
No, I'm saying I don't sit around reloading the same page all day. I might reload a number of pages, but most of my browsing is to new content. Sure, a lot of the CSS and images and the like will be cached, but that's quite different from the whole page being cached entirely.
Clearly Apple is trying to minimize network traffic, because wifi uses a lot of energy. That's a good thing to do, a perfectly reasonable optimization, but it does distort automated test results. To make tests repeatable and fair Consumer Reports has to set up its own web pages and repeatedly load them, because real web pages change and disappear so tests done in 2016 might not be repeatable in 2017.They deliberately disabled the cache to make sure that traffic was actually generated, to check wifi power consumption as part of the test.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
What is this Hidden Safari Setting?
. to me it is obvious that the first Consumer Reports test was correct.
I would somewhat agree but if Apple was able to fix the bug so that the original configuration they used with the cache off also significantly improved then it isn't a problem with the test as much as a problem with the bug that Consumer Report accidentally found. I'm actually kindof surprised that Apple never tested with the cache off as this would seem like a common thing to test.
You visited Slashdot at least five times today (evidence.) Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.
You are not alone. This is not normal. None of this is normal.
Does Firefox have this bug out of the box? I only use Firefox and the battery on my mac only lasts 4 hours. Tested three times now.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
The test is as good as it can possibly be.
You can't browse random web sites because they change constantly, so no repeatability or comparisons to previously tested machines. Also, internet speed varies. So you set up your own local HTTP server with some fixed, typical sites and browse those with a script.
Unfortunately the browser is doing its best to minimize network traffic, so your static content gets cached quite aggressively. While that is useful in real life, it breaks the test because you are not generating realistic amounts of network traffic any more. So you disable the cache. At the very worst, you will get the minimum battery life a user will see and they may get more. The test is repeatable and comparable.
Apple install small batteries in their laptops, in order to make then thinner and lighter. They compensate by carefully optimizing the OS and their own apps for battery life, helped by having only a limited number of models to support. It works well for some use cases, but not for others. This test breaks it, and so will doing things that force the laptop to use more energy like high screen brightness* or high CPU/GPU load. It's important to communicate this to consumers, in case they think that good web browsing numbers mean they will also get good gaming time.
* Note that Apple tests at "75% brightness", a meaningless number. Consumer reports set the screen to a fixed brightness, measured with a light meter, so they can compare it to other laptops.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
Developing web sites on a Mac does not reflect real-world usage. Gotcha.
Well, yeah. What percentage of a typical user's time on his Mac is spent developing web pages? In general, very little. In many (most?) cases, none at all.
He's getting rather old, but he's a good mouse.
This one.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
MacOS isn't so bug free, the setting of turning off the caching triggered a icon reload error. Next time Apple test your software better.
Spoken like a person who has never written more than 10 lines of code. Ever.
If I visit the first page for a site to retrieve all of the graphics-intensive formatting stuff, then as I browse thirty more pages on that same site I do not have to re-download that stuff because it's cached then that could make for a difference.
That depends on a particular user's browsing habits. You might "browse thirty more pages on" Slashdot while reading stories and writing comments but hit only one document on a site when reading each story's featured article. Or you might "browse thirty more pages on" a web search engine while performing queries but hit only one document on a site when reading each result.
If you leave caching on, you'd can't test by loading a handful of sites over and over. They would have to have a script of non-repeating real-world web sites that would handle 8-10 hours of battery life.
By limiting to a handful of sites and disabling a cache, you can make a more consistent and repeatable test. While you're testing real-world (and real web sites), you still want to test in a repeatable and somewhat verifiable way.
Sounds more like Apple's updating CR's review. Not much in there about what CR thinks about all this.
Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.
I can't speak to the CSS because that is normally invisible, but I can speak to "image files that make up every page". They are invisible here. A banner at the top, a couple of small icons at the bottom, but other than that -- what images?
When I visit a /. page I explicitly refresh the page because I want to see everything that is new. If I wanted to see the same things over and over I'd just print the page and tack it up on the wall.
Key word is 'keys'. And no, there are none. Of course the actual keys don't give much more feel than the touch bar anyway.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
First of all, yes, the implication was that nobody benefits from caching. And secondly, if you're getting every damned thing from the net and nothing from the cache, your network usage is orders of magnitude higher than "normal". AmiMojo is suggesting it's "cheating" to base power usage metrics in part on cached data. But most of the data that built the webpage you're looking at right now came from your cache. You think the HTML - the only dynamically generated part - made up most of the bytes of data that made up the page? Think again.
Worse still, you're focusing on something that's relatively meaningless. Do you think your Wifi card draws massive amounts of power when it's receiving data? I actually don't know if the difference in power between Wifi "listening" and Wifi "receiving" is more than the difference between "SSD drive idling" and "SSD drive sending data", and I suspect you don't either.
But I can say, without too much doubt, that the GPU and CPU suck much, much, more power than both of those put together when they're rendering a webpage. Hell, I just leave Twitter open in a tab before I go to work and my 2011 i7 laptop at home's fans are whirring by the time I get home.
On the face of it, calling it "cheat mode" when you instruct a laptop to cache webpages in a perfectly normal way is ludicrous hyperbole. The cache makes little or no difference to power usage, and it's normal behavior to have it enabled. It's also irrelevant - as others have pointed out - to why this particular benchmark turned out to be flawed, which was a bug in the no-cache mode, not something to do with caching specifically.
You are not alone. This is not normal. None of this is normal.
Too many.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Well, yeah. What percentage of a typical user's time on his Mac is spent developing web pages? In general, very little. In many (most?) cases, none at all.
Conversely, among the web developers who use Macs (which is quite a lot of them), maybe 75 percent of their workday is spent developing web pages. What's your point?
Breakfast served all day!
Try "View Page Info" (if you're using Firefox), it'll list the media embedded on the page. These vary from the Slashdot logo, the zoo icons, the social media icons, etc.
There's more than you think.
You are not alone. This is not normal. None of this is normal.
I visit basically the same dozen or so sites every day, with a few random ones thrown in there from links from twitter or facebook.
There was BOTH a bug in Safari AND a problem with the testing protocol. That is, the test was made by lazy people that don't ACTUALLY want to simulate what a real day is like, they want to reload a few sites over and over and pretend that that's what people do, and it's not. The fact that they're lazy isn't actually the reader's problem.
If they had a more representative test, Apple's bug wouldn't have mattered. Moreover, they should've tested with another browser to see if the results were replicable there. I understand that other browsers are bad with battery, but they should've verified their experimental setup.
But this got more clicks, right? Releasing a test that's OBVIOUSLY broken is more interesting than going back and verifying that you did everything right, or asking Apple what's going on. Consumer Reports is already apparently having trouble staying afloat with places like The WireCutter cropping up.
There are plenty of gripes about the new MacBook Pro, but this particular problem is more on Consumer Reports than Apple. Apple can walk away saying their hardware was perfectly good all along, and CR is going to have to issue a retraction and update to the test.
That is, the test was made by lazy people that don't ACTUALLY want to simulate what a real day is like
If you have real people clicking buttons all day, that's likely to give false readings, too. If you have enough web site samples to be representative of average web sites, then you don't actually need all 600 sites visited over 8-10 hours to be different as long as caching is disabled.
Their testing protocol would have been fine if it weren't for the Safari bug.
Moreover, they should've tested with another browser to see if the results were replicable there
I don't 100% disagree, but they're giving a single battery rating. Not one for every possible use case. Adding 3rd-party software is not completely fair to Apple, because it's not a test of what they've themselves have made and they don't need to accept the blame for other browsers killing the battery.
Apple can walk away saying their hardware was perfectly good all along, and CR is going to have to issue a retraction and update to the test.
CR didn't test the hardware. They tested the hardware/software in combination. The update would not be a retraction, it would be a tangible change to the product.
I didn't assert 'No one develops websites on the Mac', all the websites hosted on my servers are developed on the Mac.
But the number of people who do this is Much Less Than the total Mac user population.
Furthermore, few people who develop websites on any platform get their tech advice from Consumer Reports.
But then, when you can't produce a useful thought, insults work just fine.
Obviously you don't understand that "settings an actual user would have enabled" is actually not what needs tested. I'd rather the tests be done with the first Consumer Reports configuration... because when I'm browsing the Internet I generally don't reload the same page 1000s of times... I browse different pages all the time, sure some % of those pages gets cached as resources, but... all streamed data (Music, Video, Networked Games) can't be cached...
and Streamed Content I'd argue is a much bigger % of people's traffic (volume wise (hits wise probably less so... but that is the nature of the beast))
So having a test that reloads the same page 1000s of times or one that actually uses the network (and even finds a bug in the browser)... to me it is obvious that the first Consumer Reports test was correct.
But what you are ignoring is the fact that it wasn't the page-reloads that CR was INTENDING; but rather, disabling the browser cache actually triggered a BUG in Safari that caused it to CONTINUOUSLY reload certain assets on a page that had already been loaded. THAT is what caused the battery drain.
. to me it is obvious that the first Consumer Reports test was correct.
I would somewhat agree but if Apple was able to fix the bug so that the original configuration they used with the cache off also significantly improved then it isn't a problem with the test as much as a problem with the bug that Consumer Report accidentally found. I'm actually kindof surprised that Apple never tested with the cache off as this would seem like a common thing to test.
INTERMITTENT bug.
- Screen
- CPU/GPU
- (Lots of other things)
- *Wifi card that's receiving data rather than listening for it/*SSD that's not idling - *Wifi card that's listening for data - *SSD that's idling.
* Delete what's not applicable.
And just to make it clear. A low speed Ivy Bridge i7 uses about 17W of power. Your Wifi card probably uses less than 1W, and won't vary that much between power draw while receiving data and power draw while listening and waiting for data. An SSD drive can be anywhere (doing a lot of Googling right now) between 0.3W and 1.5W. Presumably SSD drives use pretty much no power when they're not transferring data.
The idea that your laptop's battery life will significantly extend or deteriorate based upon whether the data used to render a webpage (or video) is coming from an SSD or Wifi is absurd.
You are not alone. This is not normal. None of this is normal.
Apple: "Consumers don't want a touch screen or ports; they'll settle for a touch bar and dongles."
Every other laptop maker: "TOUCHSCREENS! PORTS! FRACTION OF THE PRICE!"
Finding God in a Dog
Likely not a whole lot. Couple days of a dev's salary to find and fix the problem. Sounds like its being included in an already-planned update to Safari so there shouldn't be much if anything in the way of extra distribution costs.
CR may have charged a consulting fee if it took the people on their side more than a couple emails but that wouldn't add up to significant amounts either -- at least not when scaled against the depths of Apple's pockets.
There's more than you think.
I used about:config to disable image loading (preferences.image.default = 2) and I see only one tiny difference between the previous version of this page and the one I see now: the two little 64x64 icons at the bottom of the page attached to the previous and next stories are gone. That's all.
Oh, and the "Slashdot" logo is gone. That's not very large compared to the page itself, which is useless if I use a cached version.
Well, OK, in your specific case you probably don't load much that's cached, but you must admit, you're not really doing the same things most people are!
You are not alone. This is not normal. None of this is normal.
I've tried using that keyboard several times and it's just doesn't make the cut. If they would have left the thickness they could have had better keyboard travel and better battery life.
Point is that the benchmark should be geared toward the typical expected user - Not the niche that is web developers. Seems pretty straight-forward. Make sense?
He's getting rather old, but he's a good mouse.
Apple install small batteries in their laptops, in order to make then thinner and lighter. They compensate by carefully optimizing the OS and their own apps for battery life, helped by having only a limited number of models to support. It works well for some use cases, but not for others. This test breaks it, and so will doing things that force the laptop to use more energy like high screen brightness* or high CPU/GPU load. It's important to communicate this to consumers, in case they think that good web browsing numbers mean they will also get good gaming time.
Bzzt! Wrong, Hater! Thanks for playing. Now GET OUT!
The REAL issue was, if you had bothered to read TFA, you would know that Apple found an INTERMITTENT BUG in Safari, that was, coincidentally, triggered by Disabling Safari's Browser Cache. This BUG caused Safari to sometimes fall into a LOOP, CONTINOUSLY RELOADING assets from the SAME Page, again and again. THAT is what caused the battery drain; NOT just reloading the pages per their test-script.
""What they were testing is not a real life situation, and therefore Consumer Reports tests were invalid for what they were trying to do. I don't know why the decision was made, it was a really stupid one. "
You should read all about that and learn ya something.
"...whenever any Form of Government becomes destructive...it is the Right of the People to alter or to abolish it..."
I didn't assert 'No one develops websites on the Mac', all the websites hosted on my servers are developed on the Mac.
But the number of people who do this is Much Less Than the total Mac user population.
Furthermore, few people who develop websites on any platform get their tech advice from Consumer Reports.
But then, when you can't produce a useful thought, insults work just fine.
Rather, when you don't include pertinent information, and then simply post what sounds like an inflammatory comment, you invite insults.
Caching was off in all laptops they reviewed - a very fair comparison. The bug was that something else weird happens when caching is off. The bug was specific to the new MacBook.
Socialism: a lie told by totalitarians and believed by fools.
So load more sites... I dont vistit a million new sites every day. I visit the same ones over and over again, sure its not 3 or 4 but it is probably 20 or 30 and thus caching is extremely relevant as many assets are reused.
Seems flawed to me.
"In America, first you get the sugar, then you get the power, then you get the women..." -H. Simpson
The test was good. It was Safari that had the flaw. Thanks to the Consumer Reports test it was revealed so it can be fixed.
As for actual user, you do know that Macs are popular among web designers?
This would be of more consolation if the flaw actually got fixed and consumer reports continued running the same test rather than changing a testing regime for a single product.
So their test actually puts more load on the computer (or they adjust by loading less often). It didn't account for the battery life difference and the patch seems to have solved that problem.
The rendering part is what's power-expensive. I'm not sure caching helps much with that in normal browsing, but in simply re-loading the same page over-and-over it might. I understand turning cache off across the board to compare laptops when viewing a small set of static web pages as a test - it allows the test to work fairly. I doubt it makes any difference to actual power consumption in normal browsing.
Socialism: a lie told by totalitarians and believed by fools.
This is not a setting used by customers
Even Apple is telling us to just use Chrome.
Proud neuron in the Slashdot hivemind since 2002.
The test was good. It was Safari that had the flaw. Thanks to the Consumer Reports test it was revealed so it can be fixed. As for actual user, you do know that Macs are popular among web designers?
The test was accurate for the conditions it was tested under. The test was not representative of ordinary use. The former you can blame Apple for. The latter you can blame Consumer Reports for. I mean, you can either assume that Apple hasn't run battery tests on production models or you can assume that you're triggering some kind of abnormal behavior, even though it's not obvious why. It could be faulty hardware, disk corruption, odd settings, flaw in the script, anything really. Wouldn't the sane, normal action be to contact Apple and say we're getting really strange and poor battery life results here, is this normal? And then suspend the review process pending further investigation.
Instead they took the public road and said "nope, not giving it our recommendation battery life is all over the place". They got up on their high horse and said there's nothing wrong with our tests or how we're testing, we're done here. I'm sure they were hoping this would be a huge scoop for them, they certainly got a lot of PR. But at least for me in the end they've come across a bit like crying wolf or at the very least making a mountain out of a mole hill. On the bright say it proves they actually do testing and not just look over the press release and "review" it by bragging about how great it is.
Live today, because you never know what tomorrow brings
Something like 16 hours? If the bug just stopped bugging for one test, the high result should still be closed to advertised result, not way beyond it.
Slight problem with that analysis: Consumer Reports isn't a security or software testing company, they write reviews for, well, consumers. As no Consumer would have run into this scenario, the test is flawed when used as a basis for giving the unit a poor Report.
There can be an issue with Safari (bug) and CR's testing methods at the same time.
The test was good. It was Safari that had the flaw. ...
I disagree, at least in part. The setting which was modified for testing purposes was intended to disable local caching of webpages, correct? Theoretically this was done in order to simulate a more accurate representation of real world performance, by pulling static test pages across the network instead of from the local cache. I don't know about you, but my own browsing habits do not have me frequently reloading static pages; rather, I'm loading various dynamic pages throughout the day, such as Google search results, Slashdot, Ars Technica, a webmail interface... etc. Certain of these pages will sometimes -- but not every time -- require a network fetch. That is to say, now and then the browser determines that a page has not been updated since the last fetch, and so that page can reasonably be pulled from the cache instead of over the network. Very obviously, that's the whole point of the cache. Therefore, an accurate testing scenario would by necessity have to include some mixture of dynamic pages alongside the static pages within the test suite, so that toggling hidden settings not commonly used by the general public would not be required in order to facilitate a "more accurate" simulated testing environment.
That said: It's a peculiar side effect that they happened upon a bug in that hidden setting, and the existence of that bug was indeed Apple's fault. However, the fact that Consumer Reports based their recommendation on tests which have proven to be poor representations of the systems performance, and as of yet they have failed to update their review to account for that... that firmly leaves us in a mixed responsibility scenario, with Consumer Reports sharing some of the fault as well. Apple has performed their due diligence by fixing their bug and patching it for the next beta build; Consumer Reports now needs to likewise complete their due diligence, by updating the article accordingly upon receipt of that fixed build. (Or, you know... I suppose they could fix their tests in a more permanent fashion, by simply adding a few artificially generated dynamic pages to their test server, and not using hidden and/or non-default features during testing in the first place. Creating those dynamic pages wouldn't even remotely be difficult; a basic random number generator and just about any server-side scripting environment would handily do the trick.)
... As for actual user, you do know that Macs are popular among web designers?
The original poster should have said "typical user" rather than "actual user" -- but nonetheless, I would still have to call this a red herring; it's largely immaterial to the testing scenario under discussion. But if it is pertinent at all, then I would suggest that it's probably still quite common for manufacturers to (rightly) cry foul when a supposedly impartial reviewer changes the default settings of a unit under testing, generating adverse test results... just as it's common for reviewers and users alike to cry foul, when manufacturers attempt to pull the same stunt to inflate test results.
Next time Apple test your software better.
I don't care if software has bugs (because it always does). I care if they get fixed in a timely manner.
Likely not a whole lot. Couple days of a dev's salary to find and fix the problem.
Likely a lot, which is why they went out of their way to work with CR and find the root cause. Like them or not, CR has a pretty large user base. And those non-techie type of users are exactly where Apple targets their products.
Battery life is not something apple is good at managing. There last iOS update to my iPhone 6s destroyed my battery life and basically makes my phone unusable. It went from a few days per charge to barely 6 hours with a little use. Resetting all settings and content does nothing and they offer no way to roll back to the last iOS version. I wish they would spend more time on testing and fixes before they release an iOS and less time on removing important features.
CR didn't "Updates Its MacBook Pro Review", they instead made a post explaining Apples claimed reasoning for the tests.
The post says "Once our retesting of the MacBook Pro’s batteries is complete, we will report back with our update and findings."
So they are going to install the update for Safari and re-run the tests, after those tests are ran they will then give a update. For all we know the tests could come back the same. The bug that was fixed may not been the whole source of the battery drain, so until the results of the re-testing is done on the patched Safari, there is no update to the review.
Obviously.
The idea that your laptop's battery life will significantly extend or deteriorate based upon whether the data used to render a webpage (or video) is coming from an SSD or Wifi is absurd.
Yes it is; when everything is operating right.
But, Apple has found that, disabling the Cache in Safari uncovers an INTERMITTENT bug that causes assets from an ALREADY LOADED web page to be CONTINUOUSLY RELOADED, and THAT is what causes the excessive power-drain.
DO try to at LEAST read TFS...
If they had a more representative test, Apple's bug wouldn't have mattered. Moreover, they should've tested with another browser to see if the results were replicable there.
I agree with you that they probably should have tried some other things OR contacted Apple and said, "Hey -- we're not getting battery life anywhere near what you claimed. What's up?"
That would have been NICE of them to do. But I hardly would say the test was invalid.
These aren't scientists on some sort of fact-finding mission, trying to root out the causes of hardware behavior. They're writing consumer reviews, using default software (which in this case is actually software MADE by Apple). Why exactly should their testing protocol require them to check 3rd-party software's performance (which, if it came out more poorly, would probably receive complaints from Apple fans -- "Why didn't they test it with default stuff?").
But this got more clicks, right? Releasing a test that's OBVIOUSLY broken is more interesting than going back and verifying that you did everything right, or asking Apple what's going on.
I love these conspiracy theories. If you were talking about some crap unknown review site, maybe I could understand doing something outlandish for the clicks. But Consumer Reports? If they KNOW a test is broken, they KNOW the error will be found -- and then they KNOW they'll have to retract.
For a company whose business model is trying to help consumers sort out reliability, having too many errors like this would be a DEATH SENTENCE.
The more logical explanation is that they legitimately thought their testing protocol uncovered a significant discrepancy in reported battery life, and they thought it was important for consumers to know about. I agree that it would have been NICE of them to call up Apple and get them to check for bugs first or whatever, but if Consumer Reports spent all their time asking businesses to figure out why their tests don't live up to expectations, they'd waste all their time becoming a QA service for corporations, rather than serving consumers.
There were multiple problems here -- some with Consumer Reports, some with Apple. Why are you so intent on trying to deflect blame from Apple for a buggy piece of software? This is NOT the first time that Apple has released laptops with buggy software that made them run in funny ways. (I've been the victim of it myself.) And, you know what, I'm going to hold Apple to a slightly higher standard on this than other hardware manufacturers, because basically the ENTIRE point of paying for the Apple premium in hardware is because you get good software/hardware integration and support for a VERY limited class of hardware options. There's simply no excuse even for obscure bugs given how locked-down they have their hardware and integrated software. (And if it were the first time something like this happened, I'd give them a pass. But there have been much worse bugs in the past.)
Testing protocol flawed. Apple bug discovered. Both are apparently now fixed.
That would have been NICE of them to do. But I hardly would say the test was invalid.
These aren't scientists on some sort of fact-finding mission, trying to root out the causes of hardware behavior. They're writing consumer reviews, using default software (which in this case is actually software MADE by Apple). Why exactly should their testing protocol require them to check 3rd-party software's performance (which, if it came out more poorly, would probably receive complaints from Apple fans -- "Why didn't they test it with default stuff?").
I've got a quibble here. They're writing consumer reviews, as you say, but they're disabling the cache, something that users don't do. And do they want to give the best user-facing test or not?
Either they want to do the best possible test that gives the most accurate results and give the best advice or they don't.
As for my conspiracy theory, CR has been in a bit of decline for years for the same reason that most older media has been. Their model doesn't work as well as it used to with the abundance of free, excellent reviews online.
I'm not really intent on deflecting blame from Apple, I just don't like Apple being blamed for the wrong things. I think there are more pertinent problems with the MacBooks Pro, and it was obvious from the outset that CR had done something goofy. They were literally the only testing site that found massive problems with the battery life, and they didn't sit down and go, "Gee, I wonder why Ars Technica didn't mention the battery life was awful?" It's an absurd attitude to take.
Show me any significant codebase that's bug free. Just one. It doesn't exist. Suck it, noob.
Are agnostics skeptical of unicorns too?
I'd argue that it isn't. The best pro notebook they ever made was the last one that allowed you to upgrade the ram and hard drive, and replace the battery. That's "pro".
Their latest macbook pro might be the best appliance they've ever made, but it isn't "pro".
The max. RAM is 16 GB. What "Pro" isn't going to configure it that way out of the box? So, there's one of your three arguments nicely refuted. Let's try the other one.
You can replace the battery in the new MBP. Considering it is something that is done maybe ONCE in the lifetime of the product, having it be a little inconvenient is not a reason to say that it is an "appliance". And, BTW, Apple doesn't gouge for battery replacement. In fact, they charge little enough that I simply can't imagine wanting to do it myself to "save money." Ok, so now let's try for that last objection...
Can't upgrade the hard drive. Well, that's something that, now that most laptops are going SSD, we're going to start seeing more and more. But no one will do any hand-wringing about them; because they aren't Apple, and so nobody cares what the other guys do. Again, survey says: "Most people, by and large, never upgrade the storage in their laptops." And the more "Pro" a laptop-user is, the LESS likely they are to be able to store their DATA on an LAPTOP's internal drive. Even at 2 TB, editing video projects is going to make that 2 TB seem crowded in a year or so. And with USB-C/TB3 available, there really isn't a speed penalty for having an external drive enclosure. And, although the TB3 enclosures are still a bit pricey (and hard to find, both which might change now that everyone and his dog is switching to USB-C/TB3), USB-C enclosures/drives are plentiful and available for the $20-30 range (driveless) on Amazon, like this one. So, it sounds like 3 for 3 to me.
You are out of gas...
Apparently not
... our battery tests are not designed to be a direct simulation of a consumer's experience. Rather, we look to control as many variables as possible, then perform a test that gives potential users a reasonable expectation of battery life when a computer's processors, screen, memory, and antennas are under a light to moderate workload. This test has served as a good proxy for battery life on the hundreds of laptops in our ratings.
CR
The manufacturer is quite likely to pick default settings that will favor their benchmarks. If the tester doesn't deviate from that, the test will be well skewed. As for the rest, IF and when Apple is providing the patched safari on new Macs, CR might do a follow-up review. Until then, their current review reflects the bug the end user would actually get.
The altered setting SHOULDN'T have had much impact on battery life and Apple has acknowledged that. It did because of a bug. CR did not introduce the bug. Sorry it made your favorite product look bad. If CR is feeling generous, perhaps they'll give Apple a do-over once they're shipping the fixed software. If they're feeling less generous, they'll give Apple a re-do but ding them for poor QA coverage.
CR's job isn't to show the product in it's best possible light, that's for Apple's marketing department. CR's job is to show you the worst performance you'll likely get. Consider, would you rather buy something and find it even better than rated or find it nowhere near as good as it's rating?
According to CR, that's what will happen.
There wasn't anything wrong with their test, it was the same test everything else went through. Evidently they did discuss the matter with Apple.
Are you honestly shocked when the MPG you get falls short of the manufacturer's claims?
Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.
I can't speak to the CSS because that is normally invisible, but I can speak to "image files that make up every page". They are invisible here. A banner at the top, a couple of small icons at the bottom, but other than that -- what images?
When I visit a /. page I explicitly refresh the page because I want to see everything that is new. If I wanted to see the same things over and over I'd just print the page and tack it up on the wall.
Okay. Measure the time it takes to load the Slashdot homepage. Now turn off caching and try again. Report. Especially if any of the icons next to each story changed - which each needs a TCP connection to the web server to download.
Of course news about a fake are Fake News.