Happy weekend!
There has been a lot of news related to benchmarking lately, including an admission by Google that they blocked Play Store downloads of benchmarking apps during the Pixel 8 review embargo, as well as fresh chips coming down the pipeline by Qualcomm and MediaTek.
Discussion questions:
- Do smartphone benchmarks matter?
- Are they still a useful reference and do you consider them when shopping for an upgrade?
Reminder: If you haven’t already, be sure to subscribe to !askandroid@lemdro.id!
You must log in or register to comment.
Do smartphone benchmarks matter?
Probably, but not for me.
Are they still a useful reference and do you consider them when shopping for an upgrade?
No.
What do you prioritize instead?
Battery life and screen quality first and foremost. They’re often related to each-other. Battery life because I feel it’s the one aspect that hasn’t improved very much at all over the past 10 years, and if I don’t have enough battery, I literally can’t use my phone. Screen quality because I look at the screen whatever I do with the phone, so if the screen is bad, everything else cannot make it a better phone.
If anything, battery life is a bigger issue now because you can’t just swap batteries like you could 10 years ago. You could have a backup battery fully charged and ready to go, or if you were like me you could just buy a triple capacity aftermarket replacement battery and have an extra chonky phone.
All those options are gone now. When the battery dies, you either pay $80 for someone to replace it or (more commonly) trade it in for a new phone that’s marginally better.
Storage space, support cycle, type of screen, third party OS support, aftermarket accessories, camera quality. Size.
Also kinda part of the SoC, but the frequencies supported since I travel a lot.
I think phones have been fast enough for a while now. There’s more to a SoC than speed. When I came back to Android, I went from the fastest iPhone to a SD480 with only 6GB of RAM and it was…fine for daily use. But the camera was a big letdown on that device so I got something a little bit better a year later.
Do smartphone benchmarks matter?
Benchmarks matter inasmuch as it reflects the user experience, which is to say, benchmark numbers taken alone are meaningless. However, if you tell me a game runs with specific characteristics of user experience such as quality settings and frame rate, that data describe what I can experience playing that game.
Maybe I don’t play that game. Perhaps, I’m not a mobile gamer at all. Even then, benchmarks can provide value by describing what real world performance is attainable provided that benchmarks reflect real user experiences.
I would say that benchmarks matter but only within the context of how benchmark numbers relate to a tangible thing you could experience with the phone. A CPU score? I think the value is questionable until you talk about how an app runs.
Are they still a useful reference and do you consider them when shopping for an upgrade?
I think benchmarks are more useful for enthusiasts to understand relative performance but are usually detached from the user experience that I really care about when making a purchase.
I want to know the benchmarks, but these would not drive my purchasing decision. I want to hear from reviewers who actually used the phone… because I plan to use the phone to do phone things, not to run benchmarks.
These are good points. Storage and wifi speeds are two important ones for real world user experience but they’re overlooked by many reviewers.
Performance testing is still incredibly important. It’s how we determine improvements from generation to generation, and it’s how we discover there might be issues with products.
Someone with a Snapdragon 865 probably didn’t receive a better phone overall by “upgrading” to an 8 gen 1. Someone with an iPhone 13 might not understand an iPhone 15 doesn’t bring battery improvements, and might throttle more in heavy lifting tasks.
The problem is, all our chips now are in a race to score well for synthetic benchmarks. Those scores rarely help us predict performance in real world apps. The design of the 888 and 8 gen 1 seemed tailored to delivering a big number score in a synth bench, at the expense of battery life and sustained performance. Consumers don’t user their phones by randomly testing a batch of little pieces of apps and math.
We would mock a laptop or desktop or gaming system review if all they had to offer was a pair of synth scores.
The issue is, phones aren’t ONE thing. Performance testing for a communicator phone matters more that we look at performance per watt and battery longevity. Performance testing for a gaming phone should lean harder on a collection of different game types and genres, and looking at sustained game play. A productivity phone needs a healthy balance of CPU and GPU performance to tackle laptop grade compute tasks.
We live in a time where phones are the first computers people reach for, and in many regions the only computers someone might own. We increasingly use them for tasks that required laptops and desktops, and it matters how we evaluate the claims of phone and component manufacturers.
The problem is, we’ve bought into the “average consumer” narrative which only describes the most basic level of use from the broadest swath of users in more affluent developed nations. The next billion Internet users won’t have the luxury of owning multiple computers to do individual tasks. They’ll own a phone. Probably only a phone.
We need more folks who will test devices in a variety of tasks, and will grade more than the performance of one or two games, to help consumers make more informed purchasing decisions. That takes a lot more work, which does not pay well on YouTube.
Running geekbench and antutu is insufficient to educate consumers, and aggregate scores will often misrepresent the actual daily performance aspects those consumers are likely to care about most.
I think a good review of the overall perceived performance is more important. Some phones are fairly powerful on paper in their specs, but real world performance can be very different. That might not even show in benchmarks: CPU might score great, GPU might score great, but somehow day to day performance even just scrolling the home screen is horrible. I’ve even seen phones where 3D games render faster than just a plain 2D scrolling in the settings app.
I guess benchmarks can be useful to put some numbers on, but most don’t end up benchmarking the Android APIs just the raw performance. But it’s not gonna catch that the CPU<>GPU bandwidth is insufficient for a basic ScrollView to be smooth as new items gets rendered.
My Galaxy S7 ended up slower than even a an aging Galaxy Nexus or Nexus 5 despite being much better in most specs, even running the display at 720p instead of 1440p, resolution didn’t make a difference whatsoever. My OneTouch Idol 3 was unbearingly slow despite having the exact same chipset as a Motorola model that ran butter smooth.
> Do smartphone benchmarks matter?
Depends.
> Are they still a useful reference and do you consider them when shopping for an upgrade?
Yup!
I would need to be a serious enthusiast to benchmark a phone unless it’s for pure curiosity.
Benchmarks are less important on phones as the hardware doesn’t change through it’s life. If I was writing custom drivers or testing custom images, sure. Regular benchmarks make sense to see if software changes are correctly utilizing the hardware. That falls squarely in enthusiast territory though.
For comparison, when I building a new PC, I turn all the knobs to 11 and overclock it to full capacity. Benchmarks show performance numbers as well as test system stability. I’ll benchmark again after major hardware changes or major driver changes. The system wasn’t built in a lab and components are all selected by me to meet my performance goals.
Now that I think about it, benchmarks on any device makes sense for troubleshooting. If I had a problem with my phone, running benchmarks could be extremely useful in identifying what components are failing. Still, I have rarely, if ever, done that.
And no. I don’t bother with benchmarking new phones. Performance is nice, but functionality is more important to me. I will absolutely minimize stress on the components to ensure longevity of the device.
They only matter if you are big into smartphone gaming and emulation ig
As long as the phone works well I don’t worry too much about the bechmarks. I would rather use my phone for some years and enjoy the ride. I think at some point I just tired of chasing the latest shiny Pokemon and just try to enjoy what I have.
We shall see in a few years if I feel the same.
No.
I use my phone to make calls, text, and read lemmy posts. I could probably get by with an iPhone 3 or a pixel 5 or something equally low powered.
They used to but I think we’re past the point that most matter. Battery life being the exception. Phones are more than powerful enough these days so unless you’re using it to play some serious games they don’t really matter anymore.
They used to, but now all smartphones are sufficiently powerful, so I don’t even worry about it anymore.
- No
- Also, no
