How we test headphones: The SoundGuys testing methodology
Whether you’re an audiophile or someone looking for your first set of headphones, finding information you can trust online is key. With so many conflicting opinions, marketing, and—let’s be frank—lies surrounding personal audio products online: how is anyone supposed to know what’s right or wrong? We believe it takes a dram of experience, and a gallon of testing.
SoundGuys is a collection of journalists dedicated to cutting out as much BS as possible, while still giving you the information you need in order to maximize your happiness with your personal audio products. In our quest to demystify the world of music, we regularly post education features—but we also test the headphones that come our way with a standardized process. In order to make sure nobody misses out on what our reporting means, we want to take a minute to go over each of our tests.
Editor’s note: this article was updated on April 4th, 2022, to expand upon the microphone and battery testing sections.
Objective testing
In order to gather information about how well headphones work, we subject each product to a battery of tests we’ve created for our readers. Objective testing refers to the scientifically verifiable performance of the product using standardized tests in controlled conditions. While it is certainly extremely important, we here at SoundGuys don’t subscribe to the false dichotomy of the objectivist vs. subjectivist debate in audio reporting.
For our headphone testing, we use a Bruel & Kjaer 5128 head simulator hooked up to a B&K 1704C-102 to power the ear simulators, then a high quality audio interface to digitize the signal. The measurements are controlled by our test supercomputer, and we use a program called SoundCheck to record, analyze, and export all the data. This software is used in both R&D labs, and on some manufacturing lines, so we’re able to test and quantify products repeatably, to an advanced degree. It also means that, very occasionally, we might find defective products, and so can replace them before wasting time on a full review.
Playback frequency response
When anyone mentions “frequency response,” they’re talking about how each tone in your music is emphasized. Most headphones don’t reproduce all sounds with equal sensitivity, and your own personal physiology also comes into play.
We measure each headphone on our Bruel & Kjaer 5128 test fixture, which represents the average listener, then take the data from that test and put it into a nice chart that’s a little easier to read than a spreadsheet with 57,000 data points. Here’s what that looks like:
The cyan line represents the product’s normalized frequency response, and the pink line represents our house curve, what we believe is the response shape most people will like the sound of. This chart will give you a general idea what the headphones sound like—and also give you an idea how you might want to equalize them if you’re into experimenting with the sound.
There is no one true ideal that works for everyone. Every single person on Earth hears slightly differently. Not everyone wants a super analytical sounding headphone, and some people can’t abide mass market headphones. It’s just a fact of life. We listen to the headphones, and point to actual songs to contextualize what you’d hear if you bought them.
Isolation and active noise canceling
Because lots of people like to take their headphones out in the world, we also feel that it’s important to test and show how each set of headphones or in-ears blocks sound. Not every noise is blocked out equally, and reading advertisements can’t tell you how well a set of headphones cancels out sound. Using the same scale as the frequency response chart, we can represent how different headphones respond to noise.
To test how well a set of headphones or in-ears keeps sound from reaching your eardrum, we first play a sample of shaped noise over loudspeakers, check the level, and record with our artificial head. Then we put the headphones on the head and record again. After subtracting one curve from the other, we can display the data in a way that makes sense.
In the example above, we can see that the Bose QuietComfort 35 II noise cancellation combats low noises (20-1000Hz) like engine rumbles, car noise, and voices. The pink line represents isolation performance which takes care of higher-pitched sounds (1000Hz and up), think of nails going across a chalkboard. The Bose QC 35 II is great for travel and surprisingly better than the newer Bose Noise Canceling Headphones 700.
Note: the ANC score on each review reflects objective performance, not comparative, so it reflects a product’s gross attenuation. This can make things like versus articles a bit odd: when we compare the Jabra Elite 75t to the Apple AirPods Pro, the ANC scores alone appear to indicate that the Elite 75t has more effective noise canceling than the AirPods Pro, but that’s not the case. Again, the ANC score accounts for the gross attenuation, and combines passive isolation with noise canceling performance.
Microphone
In 2021, we switched over to a standardized test setup that plays back pre-recorded spoken phrases from a calibrated artificial mouth in our test chamber, either with or without simulated background noises, simulated reverberant spaces, or artificial wind. This means that samples from every product can be directly compared, which makes it far easier to make meaningful comparisons between products in terms of the raw speech quality or the product’s ability to reject noise.
Since most wireless headsets incorporate digital signal processing, including speech processing algorithms, conventional test signals don’t give accurate results when quantifying the microphone behavior. For this reason, we use the ideal recordings to derive the microphone’s frequency response.
We also present our recorded speech samples on the product’s review page and encourage visitors to rate the quality of the samples. The microphone score follows how our readers rate the samples. We feel it’s important for the mic performance to be assessed in the appropriate context—in the case of headphones and headsets, this means the mics are generally used for hands-free calls and conference calls, not for live musical performances.
Battery life
Some people like Bluetooth headphones, and that means having to deal with charging a battery. Using testing fixtures with human-like ears, we can tell you how long you can expect your cans to keep making music.
We're trying to meet a normal use case—and an independently verifiable performance metric.
We set up the product running with a constant playback of real music, peaking at 75dB(SPL), a safe and enjoyable listening volume. A recorded waveform tells us exactly how long the headphones or in-ears lasts from that point. Often, the result is different than what manufacturers tell you, but because our test is standardized it means you can compare figures across our reviews.
Amplifier requirements
Wired headphones are increasingly rare nowadays, but sometimes they simply require too much power to be used on a smartphone. In the event that happens, we will let you know. By using a little bit of math and the specifications page of any set of headphones, we can tell you if the power output of your smartphone is enough. If you don’t see a section talking about power, assume that everything is okay and you shouldn’t run into volume problems.
Subjective testing
In the world of audio, objective testing isn’t enough because there are so many factors that can’t be controlled for. Tests regarding certain things are generally impossible to perform given the insane range of head sizes, ear shapes, and noise-induced hearing loss of individuals across the globe. That’s why we take each audio product we review and give it an extended spin in a normal home, with normal people—just like you.
Sometimes we find some faults, other times we don’t, but we always do our due diligence in figuring out what each product is like. Sometimes our experiences will differ from yours, and that’s okay! Let us know in the comments if that happens, and we will periodically update our articles if we’ve missed something important.
Build quality
Not all headphones are made to last, and it’s usually quite apparent from the materials they’re made of. We try to list as much as we can about the construction of the cans, whether they have removable cables, and other concerns about durability.
Comfort
Obviously, no two heads are identical, but we can figure out when headphones aren’t going to be right for you by actually trying them out. We check for heat buildup, how much force is put on your head, and many other considerations. We never review a set of headphones without using them for hours on end.
Value
Sometimes headphones aren’t amazing, but they satisfy a need extremely well. Not only so we identify the main target for every model of headphones we review, but we also weigh its value. That could mean how much you’re getting for how little money spent, or it offers a truly unique feature not seen anywhere else.
Je ne sais quois
If there’s a feature or product that is so different from what we cover, we will do our best to develop and add testing for it. However, that’s not always possible. In the event that happens, we will usually put it into a real world situation, and kick the tires there.
We’re constantly re-evaluating how we do things here at SoundGuys, and are always looking to upgrade our equipment, coverage, and education resources. We don’t chase numbers, we chase the truth. You can read our ethics statement for more.