Blind Audio Testing: The Scientific Truth Behind the Myth

Audiophiles frequently discuss "blind testing".  In some people's opinion, doing a "blind test" is the only way to test whether or not one audio component sounds different than another.  "All amplifiers sound the same. It's been proven in blind tests!" For others, the validity of blind testing can be problematic because of a variety of different variables.

Today, I was listening to the Home Theater Geeks podcast with Stereophile's John Iverson.  (As an aside, this is a truly excellent weekly segment hosted by Scott Wilkinson.  I recommend you check it out if you haven't).  John Iverson made a point in the interview about blind testing that I wanted to highlight because I think he's spot on.  Let me explain.

Blind testing isn't so much about the differences in the equipment as much as it is a test about the listener.  Moreover, just because a listener is able to perceive a difference between Product A and Product B, you can only talk about the listener's preference for a particular product.  If you want to take the variable of the listener out of the equation, then your listener panel needs training.

The importance of the listener in any kind of audio test simply cannot be emphasized enough in my opinion.  As someone who has taken some listening training I can say from experience that this is a wonderfully humbling exercise.  In fact, I will even go so far as to say that using untrained listeners in a "blind" test is a variable that should be explicitly stated in any test.  If you're skeptical of those claims, then I suggest you check out the benefits of using trained listeners as documented here:

• Olive, Sean E., "Differences in Performance and Preference of Trained Versus Untrained Listeners in Loudspeaker Tests: A Case Study,” J. Audio Eng. Soc. Vol. 51, issue 9, pp. 806-825, September 2003. Download for free here, courtesy of Harman International.
• Bech, Soren, “Selection and Training of Subjective for Listening Tests on Sound-Reproducing Equipment,” J. Audio Eng. Soc., vol. 40 no. 7/8 pp. 590-610 (July 1992).
• Toole Floyd E. "Subjective Measurements of Loudspeakers Sound Quality and Listener Performance," J. Audio Eng. Soc., vol. 33, pp. 2-32 (1985 Jan./Feb.).

John Iverson's comments are right in line with the work pioneered by Dr. Floyd Toole and now Dr. Sean Olive of Harman International.   Harman is the parent company of the luxury audiophile brands Mark Levinson, Revel, JBL and Lexicon.  I'll highlight for a moment some previous work posted by Dr. Olive from his blog post here about a "Method for Training Listeners and Selecting Program Material for Listening Tests".  I'd like to point out that when Harman's own Revel speakers subsidiary tests their speakers, they only use trained listeners.

There are two programs that I know of that the average enthusiast can participate in to become a better, trained listener.
The first is Harman's own "How to Listen" software.  It's a free download for Mac and Windows computers.  It comes complete with sound clips that will help train you on becoming a trained and better listener.  The software works by presenting you with an audio sample.

Harman's How to Listen software is a free download for Mac and Windows computers

It will then introduce some variable into the audio sample.  You then need to then specifically identify what that variable is.  You'll be able to track your progress and pass from one level to another once you've "mastered" that particular section.

The software is incredibly flexible.  It will give you the options to input your speaker setup and fine-tune several aspects of the program material.

Philips "How to Listen" is a free online course you can take to improve your listening skills along key metrics.

Training audio engineers was a way Philips felt it could create products of superior sound quality that highlighted every musical detail.  Philips states that graduates of the Golden Ears program can reliably detect subtle differences in sound in timbre, details, spacial impression, bass, and loudness/dynamics.  Users can signup for free and take the Golden Ears challenge and chart their progress accordingly.  While I've been part of the Harman program, I haven't yet tried out the Philips program.

Philips online Golden Ears tutorial site helps you setup your system with a simple, step-by-step walkthrough.

Both are free.  I applaud and tip my hat to both Harman and Philips for making the products free.  Helping people become better listeners is an invaluable contribution to the audio and home theater community.  On behalf of all audiophiles, "Thank you!"

I also want to thank John Iverson for a great interview and great insights.  I also don't take John's comments to an extreme by saying that only trained listeners can evaluate equipment.

The next time someone makes a definitive, vocal claim in some Internet forum that "Blind tests have shown that..." or,  "You can't say that without blind tests..." you can tell them that without trained listeners the only thing you've proven in the blind test is something about the listeners, not the equipment being tested.  Oh, and if they question you, just point them to the peer-reviewed research from the major audio industry publications ;-)