It turns out Consumer Reports’ battery life tests were being impacted by a Safari bug, as we had hypothesized was the case. Apple has resolved the bug in the latest beta version of macOS (10.12.3) and will push the solution out to all users when that flavor of the OS is ready for prime time.
Apple’s statement on the matter reads: “We appreciate the opportunity to work with Consumer Reports over the holidays to understand their battery test results… We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache…. We have also fixed the bug uncovered in this test.”
The reason Consumer Reports disables its browser cache is because the publication is trying to test battery life, which means it wants its benchmark to hit the Wi-Fi radio rather than repeatedly loading web pages from the browser cache. This is the sort of configuration change a publication might make when trying to isolate something as complex as battery life — there’s an intrinsic tension between the need to automate a test to complete it in a reasonable time frame and the need to design a test that reflects real-world usage. Since most people don’t spend hours refreshing a single page, forcing the browser to reload one page repeatedly is a better way to capture web browsing than reloading the same page from the browser cache.
Here’s how CR describes its laptop testing practices:
We download a series of 10 web pages repeatedly, starting with the battery fully charged, and ending when the laptop shuts down. The web pages are stored on a server in our lab and transmitted over a dedicated WiFi network. We conduct our battery tests using the browser that is native to the computer’s operating system—Safari, in the case of the MacBook Pro laptops…
We also turn off the local caching of web pages. In our tests, we want the computer to load each web page as if it were new content from the internet, rather than resurrecting the data from its local drive. This allows us to collect consistent results across the testing of many laptops, and it also puts batteries through a tougher workout.
According to Apple, this last part of our testing is what triggered a bug in the company’s Safari browser. Indeed, when we turned the caching function back on as part of the research we did after publishing our initial findings, the three MacBooks we’d originally tested had consistently high battery life results.
Apple, of course, is hinging its rather testy reply on the idea that this test doesn’t reflect ‘real world conditions,’ and that may be true. But it doesn’t explain why many users have reported terrible battery life before the CR tests. According to Apple, Consumer Reports’ unusual testing scenario “also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab.” No explanation is given for how an obscure icon-reloading bug can cause laptop battery life to vary between 3.75 and 19 hours, and no explanation has been offered for why some users have seen battery life decline from 9-10 hours on previous MacBook Pros to 6-7 hours that many users have reported with the newer models.
Given that Apple already had an obscure bug that was triggering in this specific case, it’s entirely possible that other obscure bugs have caused issues with other battery life scenarios. And while the CR bug may be specific to its testing practices, it’s also a useful demonstration of why publications publish results corporations aren’t going to like in the first place. Previous Apple laptops didn’t have this bug, which means Apple introduced the flaw at some point. When a company isn’t willing to work with you prior to publication, publishing results is often the only way to get a company to investigate a problem.