The Trouble With A/B Testing

0 Members and 1 Guest are viewing this topic. Read 1018 times.

James Tanner

  • Facilitator
  • Posts: 20477
  • The Demo is Everything!
    • http://www.bryston.com

BrysTony

Re: The Trouble With A/B Testing
« Reply #1 on: 3 Mar 2014, 04:06 pm »
Good article that makes a lot of sense.  I agree on the coffee analogy too!

Tony

PMAT

Re: The Trouble With A/B Testing
« Reply #2 on: 3 Mar 2014, 04:23 pm »
Great read. Thanks. The responses to the main article were interesting. One person talked about open space in music and I am really attracted to that kind of sound in music.

Wind Chaser

Re: The Trouble With A/B Testing
« Reply #3 on: 3 Mar 2014, 04:33 pm »
I have argued this point ad nauseum for decades.  :deadhorse:

werd

Re: The Trouble With A/B Testing
« Reply #4 on: 3 Mar 2014, 05:28 pm »
This is the most misunderstood and manipulated topic in audio. No one is good outright at A-B testing. Like everything else it requires practice. How or why anybody would try and get good A-B testing is beyond me but it doesn't come automatically. The ability to pick out changes randomly requires time  to train an ear to become good at random listening back to back.

Taking people into an environment and testing using A-B is either done unknowing of this or just plain out deceitful.

This has nothing to do with audio imo. In system building one becomes knowledgeable of their system. Literally practice listening to the quality and sound of gear. . This is why a hobbyist can make changes in cabling, amps or what ever and hear it. It's not a massive delusion.

That was my rant for the day.

gdayton

  • Industry Participant
  • Posts: 201
Re: The Trouble With A/B Testing
« Reply #5 on: 3 Mar 2014, 08:31 pm »
Yes, A/B testing does require an extraordinary amount of practice to become good at it. But, sometimes there is a reverence for it that I believe is misplaced. I think it should be used when designing audio equipment, but I don't think it should be held up as the gold standard superior to any other test. Device-under-test aware evaluations are quite useful too. The best designs often come from companies who know just how to use both.

avahifi

  • Industry Contributor
  • Posts: 4689
    • http://www.avahifi.com
Re: The Trouble With A/B Testing
« Reply #6 on: 3 Mar 2014, 10:42 pm »
I have heard all the anti ABX testing excuses before.

So, we designed our ABX box to allow a test session to be as long as one wants.  Listen and make notes for an hour or a day and then push a button and listen again and make more notes.  Do this as often and as long as you like.  When you are done, exist the test mode and the AVA ABX box will tell you what you had been listening to for each previous step.  No excuses.

We will send anyone the owners manual if interested.  Just e-mail me at avahifi@comcast.net

Frank Van Alstine

werd

Re: The Trouble With A/B Testing
« Reply #7 on: 4 Mar 2014, 03:45 pm »
I'll take 2.

Mudslide

Re: The Trouble With A/B Testing
« Reply #8 on: 4 Mar 2014, 05:16 pm »
I have heard all the anti ABX testing excuses before.

So, we designed our ABX box to allow a test session to be as long as one wants.  Listen and make notes for an hour or a day and then push a button and listen again and make more notes.  Do this as often and as long as you like.  When you are done, exist the test mode and the AVA ABX box will tell you what you had been listening to for each previous step.  No excuses.

We will send anyone the owners manual if interested.  Just e-mail me at avahifi@comcast.net

Frank Van Alstine

Thanks for the sanity input, Frank.

Farmelo mistakes two approaches to A/B'ing as all there is.  That is far from the truth or goals of the science of A/B/X testing/reporting.  Also, what he proposes in 'living with' an audio format or presentation is fraught with more decision making dangers than any A/B session.

I too like Neil Young, but I'm not sure I'd like his voice in a high def format without some studio ... enhancement.   :lol: