[Mister Pig]

I do a bit of writing for Affordable Audio, and have my spent a bit of time thinking about how to properly set up the reviewing process. It is my opinion that there are two main ways to review a component.

One is to install the new component into an existing reference system, and report the results, either good or bad. I can understand the benefits of proceeding in this manner, but it's not my method of choice.

Another choice is to find a complimentary mix of components that result in a system that the review component works well in. In this case, understanding the mix of components in the review system is as important as the description given of the unit under evaluation. Of course, this method is limited by the stable of stereo gear that the reviewer has access to, and his abilities to properly set the system up. This is the my preferred method of writing a review, although it may lead to articles with a consistently positive slant. The key to understanding the review is to carefully look at the system that has been assembled for the piece, and think about how the pieces will interact.

A review should never be the sole basis for a buying decision. I think many of us who have been in the hobby awhile understand this. Although the review will give a perspective buyer an idea if a component should be on their short list of components to audition.

These are my thoughts on the subject, yours may vary. Please feel free to express your thoughts and opinions on this subject.

Regards
John Hoffman

[meby]

I agree with that approach.  It is quite possible for a component to be an outstanding component but not a good match with the existing components in a reference system.  I think that is one of the reasons why so many good components are for sale on Audiogon and other places as they are not in a system that complements them well.  That and audiophile nervosa.

[pbrstreetgang]

I do believe in system synergy, but a review component should be able to be place in an existing system that sounds good and stand on its merit. If it cant then it is nearly useless. I usually do this in my two and a couple of friends systems, sure minor changes can and should be made, but no building around a review component. If it cant play nice in a well integrated system it will likely not be able to do so in the vast majority of peoples systems. Things like single driver speakers and other niche products should be evaluated in such niches but be judged harshly on their inadequacies. (for you single driver guys I like the also but you have to admit they are not jack of all trades, and the beaminess and other aspects lend themselves to personal preferences and limited music and equipment choices).

[ZLS]

    I believe that there are certain reasonable parameters that should be respected.  A 2 Watt amp that is matched with inefficient speakers is just not going to work, and it is of no value to anyone-manufacturer, reader,reviewer to do such a review.  OTOH a manufacturer that sets the condition that his/her component can only be judged when used with a severely limited number of other components (if not even other specifically named components) a review of such is of no use to me unless the system as a whole is being reviewed.  IMHO it is the editors responsibility to see that a piece of equipment that is obtained for review is placed with a reviewer that already posses equipment that would allow the subject component a reasonable chance to be heard in it's best light.  
    The same holds true for the known biases of an individual reviewer.  
    I as a reader find review's valuable when I can learn something about the piece of equipment under review.  I am more interested in what you heard from the component rather than whether you liked it or not.  

[nrenter]

I would like to see multi-party system reviews. In short, this is where several people would spend time with a system and independently report their thoughts / opinions. I think this would be far more interesting than simply component reviews, as there are too many variables that can affect the perceived performance of a component. I’m still trying to craft the “rules” on exactly how this process should be executed. Perhaps this is the thread to brainstorm this process.

1.   The system being reviewed must be comprised of components that have not been added / subtracted / modified in an agreed-upon time frame (3 - 6 months). This includes room treatments. However, physical placement of components may be changed.

2.   A minimum of 3 independent reviewers must evaluate a system before the combined review can be published.

3.   Each reviewer must have unrestricted access (no volume restrictions, no music / genre restrictions, etc.) to the system for a minimum of 90 minutes and not longer than 180 minutes).

4.   Each reviewer must have a minimum of 3 reference recordings that the reviewer must use at each review performed by the reviewer for each media type to be evaluated (vinyl, redbook digital, SACD, DVD-A, etc.).

5.   The reviewer must comment on each attribute (TBD) outlined in the standard review template. These attributed include, but are not limited to, imaging and soundstage, specific qualities of presentation of reference recordings, and overall enjoyment of the system / listening environment.

6.   Each reviewer (and owner of the system being reviewed) must list the components that comprise their own system, along with a diagram of their setup, and photos of their room / components / setup.

To have your system reviewed, you would have to post a request to one of the Regional Circles and request a peer review. Volunteers would contact the system owner and coordinate a time for a review. All reviews would be submitted to a central “publisher” for evaluation of the quality of writing and for potential clarifications / questions / etc.

I’m sure there are plenty of members of local clubs that would like to participate in this “co-sourcing” model of system reviews.

I’d be interested to get the community’s thoughts on this approach, and how these “rules” should be refined.

[Mister Pig]

I do believe in system synergy, but a review component should be able to be place in an existing system that sounds good and stand on its merit. If it cant then it is nearly useless. I usually do this in my two and a couple of friends systems, sure minor changes can and should be made, but no building around a review component. If it cant play nice in a well integrated system it will likely not be able to do so in the vast majority of peoples systems. Things like single driver speakers and other niche products should be evaluated in such niches but be judged harshly on their inadequacies. (for you single driver guys I like the also but you have to admit they are not jack of all trades, and the beaminess and other aspects lend themselves to personal preferences and limited music and equipment choices).

This is an excellent argument for using a single reference system. One question though. What about the situation where an inexpensive component is being used with a system that is several tiers above it in price and or performance? I have seen where many of these entry level pieces are given "giant killer" status, yet the people who buy them aren't going to use these pieces in a system comparable to the review one. Now this awesome inexpensive component comes back down to earth when its stuck into a system consisting of like priced units. In this scenario the higher performing pieces have a performance "trickle down" and elevate the sonics of a modest component. I know the argument can be made that the budget piece performs far above its class cause it can stay with its more expensive brethren, and that's a possibility. But on the other hand, maybe the greatness of the reference units rubs off a bit on the modestly priced unit.

Regards
John Hoffman

[Early B.]

IMO, a reviewer should state up front what his listening preferences are, and what kind of music he likes. For instance, if a reviewer is mainly into classical music, I'm less interested in his perspective because the component may sound very different with rock music. The sublime nuances of a viola is not the same as a hardcore Hendrix lick. Likewise, if a reviewer doesn't prefer a lot of bass, his observations about the quality of the bass is important for the reader to know.

Also, audio reviews tend to be too long. I'll bet most readers typically flip the page to the conclusion anyway, so keep it brief.

Finally, all audio reviewers should sing from the same hymn book. Even though audio terms have already been carefully defined, some reviewers use language that leaves too much to the imagination and can be confusing to some readers.


  

[Early B.]

I do believe in system synergy, but a review component should be able to be place in an existing system that sounds good and stand on its merit. If it cant then it is nearly useless. I usually do this in my two and a couple of friends systems, sure minor changes can and should be made, but no building around a review component. If it cant play nice in a well integrated system it will likely not be able to do so in the vast majority of peoples systems. Things like single driver speakers and other niche products should be evaluated in such niches but be judged harshly on their inadequacies. (for you single driver guys I like the also but you have to admit they are not jack of all trades, and the beaminess and other aspects lend themselves to personal preferences and limited music and equipment choices).

This is an excellent argument for using a single reference system. One question though. What about the situation where an inexpensive component is being used with a system that is several tiers above it in price and or performance? I have seen where many of these entry level pieces are given "giant killer" status, yet the people who buy them aren't going to use these pieces in a system comparable to the review one. Now this awesome inexpensive component comes back down to earth when its stuck into a system consisting of like priced units. In this scenario the higher performing pieces have a performance "trickle down" and elevate the sonics of a modest component. I know the argument can be made that the budget piece performs far above its class cause it can stay with its more expensive brethren, and that's a possibility. But on the other hand, maybe the greatness of the reference units rubs off a bit on the modestly priced unit.

Regards
John Hoffman

Well, in this case, we're talking about Affordable Audio, so reviewers should use comparable gear that is "affordable."

[Mister Pig]

Well, in this case, we're talking about Affordable Audio, so reviewers should use comparable gear that is "affordable."

That's another interesting question. What is "affordable"? One sight I post on, people would say that anything more that $500 a pair of speakers is outrageous. Their idea of affordable is a 70's ear Pioneer/Sansui/Kenwood receiver that cost them a $150. Their definition of affordable and mine are different.

The next question is this. Can an "affordably" priced component qualify as a reference piece of electronics? I am not sure that I have a firm opinion on this point.

I do know this, my reference system is affordable from an overall perspective, but certain components are far more expensive than what many would consider affordable. The DAC I use sells for 2.5K, and the Bolder modified Squeezebox I have on hand sells for well over 1K with modified power supply. The cabling I use is probably one of my most expensive components. Yet I have a JVC XL-Z1050TN that makes a fine transport and came off of Audiogon for less than 2 bills. The drivers in one of my speaker systems sell for $199 a pair. The Monarchy SM 70 PRO amps that I use for solid state needs have been considered a perennial favorite in the cheapskate crowd. Yet the Electra-Print 300DRD amps I have are not considered cheap by any stretch of the imagination. So in some respects my reference system is affordable, and I don't have uber-expensive gear. Yet to many people with real world budgets, this system would be a significant financial investment. Yet I think it gets to the heart of "Affordable Audio" and can still be a reference quality system.

So what would be your guys thoughts on an "Affordable" reference system?

Regards
John Hoffman

[pbrstreetgang]

This is an excellent argument for using a single reference system. One question though. What about the situation where an inexpensive component is being used with a system that is several tiers above it in price and or performance?

I think its exacly what Im trying to evaluate, Can this component be a link in the chain with different tiers of components, replacing a component in an already competent system without destroying the whole? Also this is why I feel the need to evaluate it in different systems and different price ranges.


 I have seen where many of these entry level pieces are given "giant killer" status, yet the people who buy them aren't going to use these pieces in a system comparable to the review one.

I believe this "Giant Killer" audio speak needs to be nixed as sound is subjective, A components competency isnt subjective, instead what should be asked is- Is this component competent enough to build around or replace a component in a good sounding system? Are there any sonic limitations or glaring warts or annoyances that the user will have to work around? Does this component detract from the musical pleasure? Also there is no way to help someone that doesnt care about the system as a whole and just buys one decent component, they are not the audience you are trying to reach and would likely be happy with any number of box store discount junk. They likely wouldnt be reading professional reviews for their purchase decision and if they did it should inspire them to budget based on the system as a whole, not one individual review component. you do want to reach those starting out though by letting them know this is a great start and they should seek others components in their budgeted price ranges to build a proper playback system


 Now this awesome inexpensive component comes back down to earth when its stuck into a system consisting of like priced units.


I agree and think I covered my take on it above, same thing as dropping a LS2 into a Geo metro. It likely wont happen and if it is done then the user should be well aware that many other changes should be made. Lipstick on a pig so to speak.


In this scenario the higher performing pieces have a performance "trickle down" and elevate the sonics of a modest component. I know the argument can be made that the budget piece performs far above its class cause it can stay with its more expensive brethren, and that's a possibility. But on the other hand, maybe the greatness of the reference units rubs off a bit on the modestly priced unit.

Agreed and how it should be IMHO. You dont have to have all class "A" rated components to have a great sounding pleasurable system but you do have to have a certain level of competency to have one. If a component can provide good music and pleasure in a top notch system it can certainly do it with other components over all budgets. If it cannot provide good music in a system were all other components are not the issue then it is unlikely to do it at all- or without extreme matching and mixing headaches.

Regards
John Hoffman

[pbrstreetgang]

Well, in this case, we're talking about Affordable Audio, so reviewers should use comparable gear that is "affordable."

That's another interesting question. What is "affordable"? One sight I post on, people would say that anything more that $500 a pair of speakers is outrageous. Their idea of affordable is a 70's ear Pioneer/Sansui/Kenwood receiver that cost them a $150. Their definition of affordable and mine are different.

The next question is this. Can an "affordably" priced component qualify as a reference piece of electronics? I am not sure that I have a firm opinion on this point.

I do know this, my reference system is affordable from an overall perspective, but certain components are far more expensive than what many would consider affordable. The DAC I use sells for 2.5K, and the Bolder modified Squeezebox I have on hand sells for well over 1K with modified power supply. The cabling I use is probably one of my most expensive components. Yet I have a JVC XL-Z1050TN that makes a fine transport and came off of Audiogon for less than 2 bills. The drivers in one of my speaker systems sell for $199 a pair. The Monarchy SM 70 PRO amps that I use for solid state needs have been considered a perennial favorite in the cheapskate crowd. Yet the Electra-Print 300DRD amps I have are not considered cheap by any stretch of the imagination. So in some respects my reference system is affordable, and I don't have uber-expensive gear. Yet to many people with real world budgets, this system would be a significant financial investment. Yet I think it gets to the heart of "Affordable Audio" and can still be a reference quality system.

So what would be your guys thoughts on an "Affordable" reference system?

Regards
John Hoffman

I like and use vintage items that are restored in various ways. Nothing wrong with that and it fits perfectly into the things we say here. An 8K CD player isnt affordable, a 500 player that performs on par with the average 2K player is. It all depends on budget and up to the consumer to determine what is affordable. That works itself out and isnt up to a reviewer to determine or even think about trying to determine. On the vintage front it usually takes a while and mistakes for reality to set in for these guys. Power Amps, some pre amps, and turntables can be reference level once restored and I have heard many great systems powered by old components such as these. its the all vintage attitude that is the sticking point as almost all newer commercial speakers trounce the vintages except for old Quads, Tannoys, Klipsche, some infinities and dalquists. You have to have perspective and not be stuck too hard headedly in one camp or the other.

[Mister Pig]

I would like to see multi-party system reviews. In short, this is where several people would spend time with a system and independently report their thoughts / opinions. I think this would be far more interesting than simply component reviews, as there are too many variables that can affect the perceived performance of a component. I’m still trying to craft the “rules” on exactly how this process should be executed. Perhaps this is the thread to brainstorm this process.

1.   The system being reviewed must be comprised of components that have not been added / subtracted / modified in an agreed-upon time frame (3 - 6 months). This includes room treatments. However, physical placement of components may be changed.

2.   A minimum of 3 independent reviewers must evaluate a system before the combined review can be published.

3.   Each reviewer must have unrestricted access (no volume restrictions, no music / genre restrictions, etc.) to the system for a minimum of 90 minutes and not longer than 180 minutes).

4.   Each reviewer must have a minimum of 3 reference recordings that the reviewer must use at each review performed by the reviewer for each media type to be evaluated (vinyl, redbook digital, SACD, DVD-A, etc.).

5.   The reviewer must comment on each attribute (TBD) outlined in the standard review template. These attributed include, but are not limited to, imaging and soundstage, specific qualities of presentation of reference recordings, and overall enjoyment of the system / listening environment.

6.   Each reviewer (and owner of the system being reviewed) must list the components that comprise their own system, along with a diagram of their setup, and photos of their room / components / setup.

To have your system reviewed, you would have to post a request to one of the Regional Circles and request a peer review. Volunteers would contact the system owner and coordinate a time for a review. All reviews would be submitted to a central “publisher” for evaluation of the quality of writing and for potential clarifications / questions / etc.

I’m sure there are plenty of members of local clubs that would like to participate in this “co-sourcing” model of system reviews.

I’d be interested to get the community’s thoughts on this approach, and how these “rules” should be refined.

Interesting proposition. Not sure it could be incorporated into Affordable Audio, but maybe a variant of this idea could. Personally I think a description and or diagram  of reviewers system and room is a good idea.

Regards
John Hoffman

[Mister Pig]

This is an excellent argument for using a single reference system. One question though. What about the situation where an inexpensive component is being used with a system that is several tiers above it in price and or performance?

I think its exacly what Im trying to evaluate, Can this component be a link in the chain with different tiers of components, replacing a component in an already competent system without destroying the whole? Also this is why I feel the need to evaluate it in different systems and different price ranges.


 I have seen where many of these entry level pieces are given "giant killer" status, yet the people who buy them aren't going to use these pieces in a system comparable to the review one.

I believe this "Giant Killer" audio speak needs to be nixed as sound is subjective, A components competency isnt subjective, instead what should be asked is- Is this component competent enough to build around or replace a component in a good sounding system? Are there any sonic limitations or glaring warts or annoyances that the user will have to work around? Does this component detract from the musical pleasure? Also there is no way to help someone that doesnt care about the system as a whole and just buys one decent component, they are not the audience you are trying to reach and would likely be happy with any number of box store discount junk. They likely wouldnt be reading professional reviews for their purchase decision and if they did it should inspire them to budget based on the system as a whole, not one individual review component. you do want to reach those starting out though by letting them know this is a great start and they should seek others components in their budgeted price ranges to build a proper playback system


 Now this awesome inexpensive component comes back down to earth when its stuck into a system consisting of like priced units.


I agree and think I covered my take on it above, same thing as dropping a LS2 into a Geo metro. It likely wont happen and if it is done then the user should be well aware that many other changes should be made. Lipstick on a pig so to speak.


In this scenario the higher performing pieces have a performance "trickle down" and elevate the sonics of a modest component. I know the argument can be made that the budget piece performs far above its class cause it can stay with its more expensive brethren, and that's a possibility. But on the other hand, maybe the greatness of the reference units rubs off a bit on the modestly priced unit.

Agreed and how it should be IMHO. You dont have to have all class "A" rated components to have a great sounding pleasurable system but you do have to have a certain level of competency to have one. If a component can provide good music and pleasure in a top notch system it can certainly do it with other components over all budgets. If it cannot provide good music in a system were all other components are not the issue then it is unlikely to do it at all- or without extreme matching and mixing headaches.

Regards
John Hoffman

Excellent points. Your post does make a well founded argument for the static reference system position.

Regards
Mister Pig

[Affordable$$Audio]

First, I very much appreciate the thoughtful replies.  It is obvious that all of the replies have taken this concept and truly have focused on John's question.

NRenter:  I would like to comment on your idea of multiple reviews.  The idea is a great one, but there are practical limitations.  One is shipping cost. Manufacturers overall, only want to pay for a shipment to one reviewer.  In a few cases, I've had to ship to another publication's reviewer (luckily, not on my dime).  A$$A, operates on a shoestring budget, the cost of the e-zine to ship to a staffer or two even just a couple of hundred miles is significant.  Now multiply that by 5 reviews each month, and I'd have to win Powerball to survive.

I can tell you that a manufacturer has agreed to letting myself, then Sean Fowler the opportunity to review their product.  So your partial wish has come true.

I also want to make it clear that I run a very non-centralized operation.  Unlike most publications, I DO NOT decide who reviews what product.  Each writer is free to contact any manufacturer about a product they are curious about.  If I'm contacted by a manufacturer, I usually send an email out to those writers that have an interest in a that type of product.

Finally, John  said it best when he stated that any review should never be considered the reason for buying a component, but a reference for taking the time to listen for themselves.

Thanks to all of you for reading our monthly efforts.

[nrenter]

I would like to comment on your idea of multiple reviews.  The idea is a great one, but there are practical limitations.  One is shipping cost.

You are thinking about moving a piece of equipment among a group of reviewers. I propose moving a group of local reviewers through a static local system. For example, I have no doubt I could get a half-dozen local enthusiasts with the ability to articulate their critique in written form to spend 60 to 180 minutes listening to and reviewing my system. Yes, I live in a major metro area (Dallas), but I'm sure the same could be said in St. Louis, Denver, Minneapolis, Orlando, etc. Especially if they are given a template to follow (things to listen for, things to look for, etc.). Help them help you.

One other benefit is that this process does not require any "favors" from manufacturers (in terms of coordination, shipping, or otherwise), as these system owners have already purchased the hardware, and have willingly committed to that hardware for a length of time (say, 3 to 6 months).

All reviews would be submitted to you for review before publish. Since all evaluations are independent, you, as the editor / publisher, would be able to tell if you've collected enough "quality" reviews to form an opinion (even though you've never heard it) of a system as a whole that is sufficient for publication. If the reviews are lacking, simply request more reviews on a system.

I think you'd find that you'll cultivate "open source" reviewers who will review local systems simply because of their passion.

As for the fear of offending a manufacturer with a bad review....we are reviewing *systems*, not components. And why would I want 3 to 6 people over to publically review my system if it sounded awful? I wouldn't.

[Affordable$$Audio]

Nrenter:
That would be fun, but manufacturers aren't keen on having their equipment passed loosely among a non-affiliated group of people.  I've had to sign agreements not letting the equipment go beyond the doors of my home.  They get pounded by requests every day, and they have to filter down to just a few that they feel they can trust.
Now, if the manufacturer was local to the review "group" that would make things quite palpable, and I could see that sort of system working.  However, it's very hard to find reviewers willing to write a good piece.  The time it takes put down quality writing is immense, and daunting.  Most audio fanatics don't want or fear writing anything beyond a short email.  I've had several would-be writers fall by the wayside once they start trying to put down a sample review.
But, if you belong to a local audio club, I'd look at having the group arrange a visit to a manufacturer, and go from there.

[Mister Pig]

I would like to comment on your idea of multiple reviews.  The idea is a great one, but there are practical limitations.  One is shipping cost.

You are thinking about moving a piece of equipment among a group of reviewers. I propose moving a group of local reviewers through a static local system. For example, I have no doubt I could get a half-dozen local enthusiasts with the ability to articulate their critique in written form to spend 60 to 180 minutes listening to and reviewing my system. Yes, I live in a major metro area (Dallas), but I'm sure the same could be said in St. Louis, Denver, Minneapolis, Orlando, etc. Especially if they are given a template to follow (things to listen for, things to look for, etc.). Help them help you.

One other benefit is that this process does not require any "favors" from manufacturers (in terms of coordination, shipping, or otherwise), as these system owners have already purchased the hardware, and have willingly committed to that hardware for a length of time (say, 3 to 6 months).

All reviews would be submitted to you for review before publish. Since all evaluations are independent, you, as the editor / publisher, would be able to tell if you've collected enough "quality" reviews to form an opinion (even though you've never heard it) of a system as a whole that is sufficient for publication. If the reviews are lacking, simply request more reviews on a system.

I think you'd find that you'll cultivate "open source" reviewers who will review local systems simply because of their passion.

As for the fear of offending a manufacturer with a bad review....we are reviewing *systems*, not components. And why would I want 3 to 6 people over to publically review my system if it sounded awful? I wouldn't.

It is an interesting concept. This is an activity that often occurs at audio clubs in large metro areas. They just dont publish their results, or it doesn't get past their own newsletter.

Now it would be interesting to get a group of reviewers together from A$$A and do this sort of thing. However, we are spread out geographically. Now if something like this could be arranged for an evening at an event like RMAF....well that might work.

Regards
John Hoffman

[nrenter]

Thanks, John. Based on the previous responses, I was begining to think that I wasn't making myself clear and I needed to type slower. 

The keys to making this successful would be 1) well defined ground rules, and 2) a good template of questions to be completed by the reviewers.

What questions would you ask me to understand my system's strengths and weaknesses, to learn about my listening environment, and to understand the overall enjoyment of my listening experience without actually listening to my system? Those questions need to be in the "template".

[Mister Pig]

Thanks, John. Based on the previous responses, I was begining to think that I wasn't making myself clear and I needed to type slower. 

The keys to making this successful would be 1) well defined ground rules, and 2) a good template of questions to be completed by the reviewers.

What questions would you ask me to understand my system's strengths and weaknesses, to learn about my listening environment, and to understand the overall enjoyment of my listening experience without actually listening to my system? Those questions need to be in the "template".

Having a template of evaluation questions, or areas to look at would make for an interesting group review. It would add some structure to a somewhat subjective evaluation process. Like I mentioned earlier, maybe this could take place at a major event. Many audio shows have some interesting informal gatherings among individuals after the show is done for the day. An event like RMAF or VSAC could be the venue for this to happen.

Regards
John Hoffman

[Early B.]

Back to the question of what is "affordable." Well, we know it's tough to put a price range on it. We also know that some things obviously won't make the affordable list, such as a $5,000 amp because most of us don't have the financial means to purchase it.

IMO, once you hit the $1,000 threshold (new or used) for a single component (pair of speakers, amp, CDP, integrated amp, etc.), then any items priced beyond this point quickly begin to move further away from what most of us would consider "affordable." 

I suggest running a poll in the next issue of the e-zine and ask readers what they think "affordable" is. It would also be interesting to ask readers what their audio system is worth on the used market. And we can ask them what's the most money they ever paid for a piece of audio gear. Readers can be linked to this forum to respond to the questions.