Audio Science Review Discussion

I guess all other reviewers are testing in a stereo setup while ASR prefers, as Amir has explained, a monaural setup. All others base their verdicts mainly on listening tests with music using measurements just to verify and explain details, his verdicts are 80% depending on measurements as he stated.

So it sounds like it is fairly easy to decide whom to trust: If you plan to listen solely to mono recordings or do scientific experiments requiring technical qualities, trust Amir. If you want to listen to music in stereo or surround, base you decision on other reviewer´s recommendations.

If this would be true to measurements which are showing audible and relevant sonic qualities, there would be no reason to object to that. In reality, it seems that condemning verdicts are in many cases based on either inaudible or irrelevant aspects. Or - even worse - based on a completely wrong interpretation of what a certain measurement would translate to in a realistic listening scenario (as it was the case with contiously increasing directivity index towards higher frequencies versus approximately constant directivity).

Having a technical estimation why a component (particularly loudspeakers) is not ´crap´ does not necessarily mean it would be needing your taste and blending with your environment acoustically. There are many products around which are very good at what they are suppose to deliver under recommended circumstances (e.g. studio-grade room acoustics) but fail under different conditions (like a certain living room).

Maybe you should have read at least the headline of the survey before you evolve your wrong conclusions. Everybody can read that this is a discrimination test on different equalization scenarios. It is trivial that in a mono environment it will be easier to distinguish audible differences between several EQ curves as in stereo and moreover in surround different listening angles and placements in the room will lead to something like an average curve. I do not need a survey to know it is the easiest way to discrimate differences.

On the other hand it means the results are completely useless when it comes to judging if a system is tonally balanced or sounding well with music in stereo.

Yes, and believe it or not, but the goal in cases i deal with is listening to music or judging recordings, and not hearing artifacts or differences.

The research had two outcomes: which EQ works well and testing method for differences in tonality. The latter showed once again that you want to test with a single speaker as otherwise listeners are much less critical when it comes to frequency response errors in speakers.

Keep in mind that I have been a professional colleague with the researchers in that paper. I have been to Harman and taken the same test in mono. I have spoken countless times with Dr. Toole whose work in this area goes back decades. He will tell you in no uncertain terms that you want to test speakers in mono. Here he is saying the same out of many such posts on ASR:

“As has been well publicized, listeners are far more critical in their assessments of sound quality when listening to a single loudspeaker. This fact has generated a fairly constant stream of flak from those thinking that stereo is a more rigorous test and that listening in mono was all but irrelevant. However, decades of double-blind tests show that listeners hear problems in loudspeakers more readily when listening in mono. The superior sound quality was less clearly reflected in scores in stereo comparison tests, and even less in multichannel evaluations. Monophonic components exist in stereo and multichannel programs, so designing loudspeakers to meet the most stringent (mono) test was considered worthwhile. But,the question remains; why? Were the audible defects more clearly revealed because the spatial complexity and inherent amplitude/phase (linear) distortions of stereo were absent? Is this why headphone listening has such an almost magical clarity? One sound to each ear, not two. It does seem reasonable.”

As I wrote, all of Harman research and development listening tests are done in mono. Try doing mono testing and you too will quickly realize how much easier it is to find response errors in speakers.

Nope. I explained all of this in my video. Here is Dr. Toole writing in his book:
Sound Reproduction
Loudspeakers and Rooms

"And more thoughts to take away: A loudspeaker that sounds good in a
monophonic comparison is likely to sound good in a stereo comparison, but the
reverse is not necessarily true. Evaluate your loudspeakers in monophonic comparisons
(to find out what you really have). Demonstrate your loudspeakers in
stereo or, presumably, multichannel (to impress everybody)."

The science here couldn’t be more clear and authoritative. Whether it makes sense to you is not material.

11 Likes

It seems to me, you’re arguing for the sake of being controversial.

You can‘t seriously want to have a loudspeaker, or any other upstream system component for that matter, introducing more artifacts or differences into the signal than technically and economically avoidable.

Room acoustics add even more distortions to the original signal and often swamp loudspeakers’ or electronics‘ contributions, but can never, ever remedy any flaws caused by upstream components.

So, when doing such optimizations professionally, you’d absolutely want to see, minimize and document any single contributor’s influence on the end result, instead of just listening without traceable verification.

:person_facepalming:

14 Likes

Its not that hard.

You test/compare in mono as its easier to differentiate tonality.

Listen in stereo, knowing you have picked the best for you, and enjoy!

Its a method based on proper science!

7 Likes

Your use of the term “science” in the above statement is misleading. Your conclusion directly refers to the (obviously informal) advice Toole gives to customers looking for good loudspeakers or to people trying to evaluate the quality of speakers they already have. By definition, such advice isn’t science. Scientific research is always descriptive, not prescriptive. Assumptions about what is “likely to sound good”, as Toole himself puts it, obviously don’t fall into the category of science either (even if they’re science-based).

Toole’s position is a lot more controversial than you’re trying to make people believe. While every serious researcher in this particular field knows and acknowledges that mono testing can be extremely useful, only very few consider it the golden standard for evaluating how loudspeakers interact with a given listening room. The vast majority of scientific listening tests aiming at such evaluations are carried out in stereo.

Yes, true. But how does this help anyone? Who wants speakers that are “likely to sound good” in stereo? “Likely” is the same as “not necessarily”. As Toole himself admits (without realizing it), you can’t really know from mono testing alone…

4 Likes

We might possibly have a misunderstanding here, since to my understanding, mono testing has less to do with how a loudspeaker interacts with the given room‘s acoustics, but more about its own tonal aberrations.

10 Likes

Just because your “goal” is “listening to music” or “judging (the quality of) a recording” doesn’t mean that you can’t benefit from test methods that help you to hear artifacts etc. better/more reliably. That’s not mutually exclusive…

Exactly my point. That’s why mono testing is (almost) never used for this specific purpose. Quite obviously, what you hear is never the speaker(s) alone. It’s always the result of the interaction between the speaker(s) and the room’s acoustical properties (and your brain/pair of ears, of course). This also includes any tonal imbalances you might perceive. If you want to know what the result of this speaker-room interaction actually sounds like in stereo, you need to do stereo listening tests, especially (but not only) in rooms with no or non-professional acoustic treatment. That being said, serious scientific research is never based on such (necessarily subjective) listening tests alone. For obvious reasons, researchers also need measurements to be able to evaluate how two stereo speakers interact with the room they’re in…

3 Likes

What you quoted from me is the conclusion of much research and my own experience. The “science” part was the controlled testing I quoted and others I covered in my video. As to Dr. Toole, that was a “thought” he offered at the end of extensive explanation of his controlled testing in Chapter 18 where it was very clear that listeners are more critical of speaker flaws in single speaker testing than mono.

What “majority of scientific listening tests?” Take away Dr. Toole and his teams research from NRC to Harman and there is essentially no research into listener preference when it comes to speakers. Much less it showing that stereo testing is superior to mono or equal to it. So please don’t assume such research exists let alone making a case for stereo.

There is a lot of research into room acoustics but it is not directed at preference. A lot of that research such as audibility of reflections and such, is conducted with single speaker and not with any kind of stereo presentation.

It helps you by teaching you how to properly evaluate speakers. If you are auditioning and comparing speakers, do it in mono. Members at ASR have done this and have been astonished at the improvements that makes in their assessment ability.

What you don’t know from mono testing is not solved by stereo testing.

Regardless, this line of discussion came about saying it was ridiculous that I was listening to speakers in mono for my reviews. I am showing you research which hugely validates that method as opposed to stereo which you all use and so do reviewers not schooled in the audio science.

The goal of every reviewer should be to provide as reliable of information they can provide. Here, I am filling the holes left from measurements with specific listening test strategy backed by research. If someone wants to dispute that, much less ridicule, they better show up with their own research and not “everybody knows” type of arguments. Some things in science are not intuitive and this is one of those.

Not true. If a sound is panned left or right, that is all you hear. If you have a center channel for movies, that is all you hear. As such, you want a speaker to produce the right tonality by itself. Chance of the other speaker helping it in this regard is very low as you are relying on complex interactions between two speakers and rooms.

How would me including a random room with two speakers aid you in realizing that interaction in your room? Answer is that it can’t. All that does is confuse the listener and produce random nonsense as you see in plenty of speaker reviews.

The answer to what you seek is in the measurements that are correlated with listening preference. We use smooth directivity to predict a speaker being well behaved and performant in varied listening spaces. Such spatial qualities by the way is also apparent in single speaker testing and I routinely comment on that. You don’t need two speakers to do this.

12 Likes

Yup. It would be nice for you to include references to your long posts :wink:

3 Likes

Here in Germany alone, there are currently at least four research teams whose field of expertise is the very thing you say doesn’t exist. I personally know more than a dozen other such teams in England, Switzerland, France, Spain and Hungary. There is a ton of literature on this subject matter, not only in English, but in multiple other languages as well… Please do your homework.

6 Likes

Why don’t you show us that you have done your homework by providing links to such papers? This is the second opportunity you have had to do so but still arguing without.

And what I said doesn’t exist is published research. Not that people aren’t doing R&D. Without publication, you have no idea that such work exists and its level of rigor.

6 Likes

Anyone who asks for proof that such research “exists” shoudn’t be taken seriously. This is googable. It literally takes seconds to find out there are hundreds of such publications…

7 Likes

The funny thing is that Dr. Toole in his answer obviously understands why judging speakers for music reproduction in mono which are later used for music reproduction in stereo is nonsense. He is referring to the point that “All direct sounds arriving from ±30 deg provide HRTF characterization for the wrong incident angle generating an unavoidable timbral error”. That’s correct at least to phantom sources being reproduced by a stereo system and localized in a ±10deg window around center. The closer you get to 30 deg from the center, the more likely sources created by any type of intensity stereophony are getting closer to ´the real thing´ in terms of timbre. In the recording community this is a well-accepted thing known for decades. Many recording engineers have their own methods to counter the fact that mono sources panned close to center using intensity stereophony change their timbre and perceived elevation. The easiest and wide-spread cure to that is using time-based stereophonic mic arrangements and panning in the first place. Funnily this is not a very common thing with popular music being recorded close-miked in an anechoic studio.

Judging a speaker in mono which later will be used for stereo will lead to inevitable misjudgment in most of the important aspects of reproduction quality, especially tonal balance, localization, depth-of-field and ambience.

Response errors like frequency-depending localization, width of phantom sources, blurred or alternating localization, compressed depth-of-field, too broad or too narrow imaging, overlaying reverb pattern of the listening room or stereo imaging appearing to be too close to the listener or too far away?

[moderated]

That’s simply untrue. A loudspeaker which creates a well-balance tonal perception coming in from 0 deg angle to the listener will lead to an unbalanced one in a ±30deg constellation (i.e. stereo). A loudspeaker creating a more or less stable and ideally sized localization in a mono environment will lead to unstable and overly broad phantom sources in stereo. A loudspeaker giving a fairly distanced and ´natural´ perception of depth in mono will lead to far-away, distant and undynamic spatial impression in stereo. I could continue this list endlessly…

No, the research and the comments by Dr. Toole say rather the opposite - you seem not to understand it.

So you hear in an anechoic environment?. That’s is of course one more step further away from common listening conditions and making you results even more invalid to everyone else.

Furthermore, die example shows that you seem to have no idea about time-related stereophony which is the most common method in recordings of acoustic concerts to create both ambience and ´natural´ tonal balance from all angles.

The right tonality is depending on the angle the sound from the speaker is coming in. If this is 0deg (mono), judgement of tonal balance is fundamentally differing from ±30deg (stereo), so verdicts are incompatible.

If a stereo setup in a random room cannot give any useful judgement on sound quality, how can a mono setup in a random room? It is acoustically even further from what will be the listening situation later.

There is a certain truth in the claim that verdict in one room cannot be transferred to others without reflecting the differences. That is why it is important to have basic understand of speaker directive, rooms RT, listening distance and possible interaction between room and speakers in the bass region before doing a listening test and before providing advices to others.

There is no such thing as ´smooth directivity´. The question whether [moderated] Amir can draw a straight line into a graph or not has nothing to do with the speaker´s interaction with the room.

If you want to judge a speaker´s interaction with the room you need to know directivity index in the relevant frequency band being either low, medium or high (or zero) as well as judging if we talk about fairly balanced or constant direcitivity, continuously increasing direcitivity, being any steps up in DI or alternating directivity as well as keeping the room´s RT60, ideally tonal data on early reflexions plus listening distance and wall distances in mind.

If ´smooth directivity´ is translating to anything that is either constant or increasing or sharply increasing, being high or low index, it does say nothing about sound. It can lead to a balanced reproduction or anything utterly dull, dark, muddy, murmuring, voluminous, midrange-heavy, present, aggressive or damped.

That’s what people will get if they follow your advice on ´perfect directivity´ being unlucky enough to choose the wrong speaker for their environment.

I have called this method of drawing a line into the graph and calling it smooth in order to predict sound quality in a room ´hifi-astrology´. [moderated]

Ad hominem attacks and not providing any data to your lengthy elaborations don’t make your point any more convincing …

19 Likes

Why? Do you intend to listen to music with four speakers in mono? :rofl:

Mono testing isn’t ridiculous at all. People like @Audiogeek, @Marian Weigel, @hwz and others have made that clear more than once. However, what is ridiculous is Amir’s denial of the fact that most published research involving scientific listening tests is based on stereo testing or his claim that such research doesn’t even exist. Stereo testing is the rule, not the exception (in science, audio gear development, music production etc.).

Mono testing has absolutely nothing to do with unscientific beliefs.

4 Likes

Once again I must remind everyone to keep the discussion civil and to critize ideas not people.

Moreover, please provide reasoned counterargument, and avoid endless contrarianism.

Endless Contrarianism

It’s ok to disagree with others, but if you find yourself engaged with the same battles over and over again, it’s time to move on. If you are developing a reputation around an issue or position, then you’re not building constructive discussion. Move on.

12 Likes

Drawing random lines into graphs basing either praising or comdemning verdicts on the question whether a straight line is visually achieved or not, has nothing to do with science or reproduction quality as well.

The mono test discussion was completely separate from that topic.

There has never been any disagreement on the necesity of mono tests. They are very useful in research and early stages of development when the main goal is to find out which phenomema are audible and which not, which are annoying and which not, nailing down audible flaws and artifacts. Same is true to bandwidth-limited listening tests (like subwoofer or tweeter alone) or listening in anechoic chambers or alike.

If you want to hear differences of narrow-banded peaks or dips, low levels of distortion or resonances, that perfectly makes sense.

They are not useful for judging a speaker’s overall tonal balance in a stereo setup, understanding its interaction with the room, imaging, ambience, localization and orher important aspects that define the quality of a system in a room.

Certainly not. But could you please elaborate on the question which particular aspects of reproduction quality can be reliably judged in a mono test allowing a more or less precise prediction how it will sound in a particular room in stereo? I guess this list is pretty brief, except from obvious things like distortion or narrow-banded amplitude response filters (which it was used for in the survey quoted several times).

That’s a job which i would expect even a manufacturer of budget loudspeakers to do for me when choosing drivers and developing the crossover.

Because cars have four wheels. Duh!

4 Likes

No, since that wasn’t what I was referencing to .
It was aimed at your statement, that you’d not use measurements in validating what you arrived at by only earballing your work in your professional endeavors, as you make us believe - something seeming like a semiprofessional practice at best, in my book.

5 Likes

Could you be more specific? Perhaps link to some of their papers? There seem to be few references given in this entire conversation. I’d be happy to see what further research the Toole/Olive work has engendered.

But, to be more constructive, let me begin. If we go to ResearchGate.net, and search for “Floyd E. Toole”, we find

https://www.researchgate.net/profile/Floyd-Toole

Selecting one of his 29 publications, in this case his 2008 book, " Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms", and looking for citations of it:

https://www.researchgate.net/publication/285228105_Sound_Reproduction_The_Acoustics_and_Psychoacoustics_of_Loudspeakers_and_Rooms

This is standard research stuff, folks. Looking for citations of previous work. And so far, looking through the publications listed, I see little rejection of Toole’s work and assessments.

13 Likes