[TONEPUB]

All the so called rules are great except for one thing.

No matter what any of us do, there is a certain percentage of
you that will disagree with us no matter what we say anyway.

Both I and my technical editor get our hearing checked every
year to make sure we aren't losing our minds, but if I like
speaker A and you like speaker B, what does it really matter in the end?

My God, we are reviewing hifi.  Not finding a cure for cancer or
telling you what medications to take for depression.

And I can tell you why guys like Michael Fremer get grumpy.
Though I don't always agree with the way he handles himself
personally, he spends every day of his life listening to music
and auditioning gear.

As much fun as these forums are, there is a negative side to
it where we bump into the readers that are always curmudgey,
always question everything you do, and always put you on the
defensive. No matter what any of us say in the press, it's always
wrong or suspect.  And I have to tell you from my side of the
fence, it really gets boring listening to the people in the audience
that are always confrontational.

If there were some real absolutes in audio it would be a lot easier.

Music is not created equal. All rooms are not created equal. 
Many people perceive sound differently and others have different
preferences for the type of sound they like.

And way more often than not, I've found that many of the people
Ive bumped into on the forums think their system is the "best"
and anyone who disagree with them is stupid or whatever.

Again, this gets really tiresome to hear.

This is not a car race.  There is no best anything.  There are components
out there that do a better job of accurately reproducing the sound of real
acoustic instruments than others.

But it all goes out the window if you don't hear exactly the same way
I do.  If I have a 2db peak at 7000 hz (this is for example only) and
you have a 2db dip at 7000 hz, a speaker that I find a touch bright
is going to sound a touch dull to you. And so on.

All we can do is try and guide you down a path.  Hopefully, reading
one of our reviews will help you make a purchase decision that will
make sense for you.  McIntosh buying me dinner at CES isn't going to
change that one way or another.  More often than not, if I'm having lunch
with someone from the industry, I pick up the tab just for the shock
value alone, because they always expect me to be a sponge.

Again, someone spending 30 bucks on a meal isn't going to get me
to say something that sounds lousy sounds great.  Until you unexpectedly
see a new Porsche GT3 parked in my driveway, I wouldn't worry about
that.

Even if the manufacturers gave us reviewers our gear, it wouldn't matter
that much because the end users are the ultimate judge.  We can only write
so many reviews that are totally inaccurate before you stop reading, so
dishonesty doesn't benefit us and it doesn't help you.

Because some of us in the upper level of this game have access to just
about anything, we end up getting the stuff we like anyway and that's what
you should do as well.  I'm just trying to help you define it.

[TONEPUB]

Anyone who reviews something is going to get scrutiny.   I'd suggest that some basic ethical guidelines be written and understood and followed by all the members of a publication be first practice.   There has to be written guidelines about acceptable, and unacceptable behavior.    The guidelines should be reviewed on a regular basis and everyone before becoming a member of a publication should know and accept those standards. 

Often it seems to me that publications take someone on to review with little or no formal training.   I don't think reviewers need a prolonged class but an overview of the ethical standards and practices seems to me CRITICAL before I'd let anyone review anything for my publication (which isn't forthcoming).    I'd also have strict rules about personal correspondence.   Once a guy becomes a reviewer for a publication they become a representative of that organization.    Mr. Fremer does his associates a great deal of harm by responding to an antagonist in  a vulgar and unprofessional manner.   I've seen the guy act like an A-hole as CES and I'd never employ him or anyone like him.   

A step in the right direction would be a public statement of ethical guidelines that everyone in the organization understands and agrees to follow.   That ethical statement should also be involved in the FIRST correspondence with any potential advertiser or company submitting a product for review.    That written ethical standard should also be front & center in any publication, as it helps to verify the credibility of the publication in the eyes of the public. 

What the ethical standards are and the details are left open to each publication.   All of this seems to me common sense yet I've never seen such a system used. 

This is a little bit crude, but here's how I've handled it at TONE...

First off, four of my seven reviewers are lifelong friends of mine that
I've worked with on other jobs and/or on other magazines in addition
to their skills and abilities, I know I can trust them with someone's 30
thousand dollar power amplifier.

Second, the other three reviewers on the staff got the same speech:

I told them point blank that if I ever found out that gear was mishandled
(i.e. sold, traded or otherwise weaseled) they could expect a personal
visit from me, where I would promptly beat the shit out of them.

this is a direct quote, ask Marc Phillips if I'm making this up...

Anyone on my staff makes side deals or pulls anything unethical they
are canned immediately.  So far it hasn't been a problem.

[RAW]

This is just to funny.
All with in the last 3 weeks we have been solicited by 3 reviewers asking for a pair of RA8 speakers plus the other 2 wanted complete Apex III all were asking for fully loaded crossovers, wire ect.

Sure we will build out a few pair of hand crafted $5000.00 plus speakers and ship them out for long term use.

O sorry for review.
 

I agree with Kevin as we have talked about this many times,

[TONEPUB]

This is just to funny.
All with in the last 3 weeks we have been solicited by 3 reviewers asking for a pair of RA8 speakers plus the other 2 wanted complete Apex III all were asking for fully loaded crossovers, wire ect.

Sure we will build out a few pair of hand crafted $5000.00 plus speakers and ship them out for long term use.

O sorry for review.
 

I agree with Kevin as we have talked about this many times,

But companies like Martin Logan, Avalon, Wilson and others do it all the time.  Gamut is shipping us a pair
of 130 thousand dollar G-9's this summer and Lars from Gamut is flying out from Denmark to make sure they are set up ok...

What a lot of reviewers don't understand, especially some of the new ones is that if you are a company that has a decent reputation, there is a high probability that a goofy review can really HARM your business.  We have a policy that if a manufacturer feels like we have written a review that is so completely off base that it would damage their business, we will give them the option to pull the review.  In three years, no one has taken us up on it.

But it goes back to the earlier comments about setup.  I know Jacob George from Rethm.  We gave their speaker a great review and I even bought the review pair.  But the guy at TAS that reviewed them put the speakers in an 11 x 12 room and hooked up the 101db lowthers to a 500wpc solid state Bryston amp and wrote that they sounded "thin and uninvolving" 

But TAS doesn't give you the option to pull the review and as a consequence Rethm lost a lot of sales in the US.

I had mine in a 16 x 24 foot room with a pair of Bottlehead 2A3 amps and they were awesome.  But we were only two issues old then and did not have the reader base we do now.

So, if you have a really solid business and more orders than you can fill, you really don't need any reviews.  And the last thing you want is a reviewer not setting up your product correctly and writing a snarly review.

It's always a case by case thing and whatever makes the best sense for your business is the way to go.

[gerald porzio]

What's the point of having a technical editor &  EE on Tone's straff if nothing gets tested? What do they do? Plug in the coffee pot?

[JohnR]

porzio, don't you have anything better to do than troll around on this site? You've already been asked to not post in one circle, very close on another, and now you seem to be shooting for a third.

[gerald porzio]

The real danger in not testing is allowing defective, if not poorly designed components, to receive glowing, if not rave, reviews from the Golden Eared Gurus. Two examples come immediatly to mind - the $350,000 WAVAC amp w/ a 10 db base peak & the Zanden 5000 MK IV Signature D/A Converter, which even the mfg. conceded was defective. This calls the reviewer's credibility, credentials & hearing into question. At least Stereophile, by supplying measurements, gives readers & prospective buyers a comprehensive view of the good, the bad & the ugly. Who knows how much defective & poorly designed gear escapes reviewers who work w/ no scrutiny. This perhaps accounts for the tons of gear on Agon.

[bushbison]

I’m going to “chime in” here at the risk of offending many.  And I haven’t had my fill of morning espresso yet, so please bare with my free-flowing rant!!!
 
I am one of those people who DO have a forthcoming Audio Review ‘E-Zine in the mix that has been a LONG time in coming.

I have desired to do so, and am honest in this regard, to further my enjoyment of music and share it with those who may be interested in ANOTHER / alternative point of view.  I am NOT a physicist, an electrical engineer or have any formal education in acoustics and psychoacoustics….

I DO have about 20 + years of my relatively short life wrapped up in some aspect of the music industry through performing, recording, engineering and producing music.  As a multi-instrumentalist so I DO have a good/great ear for many instruments and have a certain auditory “vantage point” that is unique, especially again compared to many from more scientific backgrounds.
But that is just my VALUE. 

MOST people who purchase audio equipment are not from ANY of these fields (either on the scientific end or “music industry” {?}), so if we provide an alternative to the very learned and experienced publications that exist, and people find meaning in what we have to offer, who loses?  Don’t forget, the theory of gateway drugs!  I do believe more people would “enter” the hobby …and come on everyone, let’s keep this in prospective; it is JUST a hobby…. With a publication like ours (realistically slated for the next month or so), and then as they gather information, gravitate towards those more high-end and serious publications….. so be it!  But people have to start SOMEWHERE ..and I have a VERY good “feeling”, that our philosophy may engage other audiences who would not usually end up on an AWESOME site like this, who may then come on board; the electronics & recording industries are evolving as we speak, whether any of us like it or not; so we can progress or get left behind…. But if I may, if we ignore well-meaning new publications like ours, who honestly are not hugely profit driven, but may have a better “ear to the ground” rather than sticking with the “old boys network”, we can actually HELP both industries and audiophilia in general; again, who loses?  If we suck, people won’t read, and we collapse; simple as that.

In our first publication we endeavor to “come out” (?!) fully about our “mission,” backgrounds, specific focus and potential audience.  And we will not accept advertising from our target component reviews, thus at least attempt to set a somewhat ethical line in the sand.

One more thing: I learn DAILY from EVERYONE with this hobby; from AWESOME discussions via email and phone with manufactures on this site and beyond, to home and business visits of some of these wonderful companies, to reading my supposed “competitions’ wonderful publications (a few VERY respectfully represented on this site), & the online discussions …. Etc., etc….. we have SO much to offer each other, and putting up fences and limitations on all of this totally defeats the purpose; do we not all just LOVE music?  I do.  And I’ll be honest; when talking to the more friendly and insightful people IN the industry they have that in common as well, and we usually will get WELL off topic of equipment, and talk about MUSIC!

I.M.H.O. (…hmmm…. Sounds like a good name for a site ……  )
Jason Gower

[miklorsmith]

There are a couple of conversational threads within this forum thread.  One is related to the original topic regarding ethics, the other is the  "if I was a reviewer . . ." and "all you guys should really . . ."  If we all did what you all demand be done there wouldn't be any reviewers and no reviews.  "You all" have 1000 ideas about what we are supposed to do and I can't imagine any review would satisfy all.

The Sphile-type testing stuff is a good addition to a review no doubt.  It's also really expensive, beyond the reach of un- or modestly paid writers. 

Here's the thing, it's a volunteer gig for me.  I do it for fun and the entertainment of whoever happens to enjoy what I write.  Of course, I am relieved when my perceptions are supported by others' opinions and concerned when they are not.  I am not striving for "correctness", rather honesty.  A wise man once told me if you always tell the truth you never have to remember what you said.  That's a pragmatic approach for sure but one that lets me sleep soundly.   

If there were a definitive approach to more accurately define the listening experience I'd be all over it.  So far, all the artificial constructs I've seen have only a passing relationship, or worse, to what I hear.  I've fallen hard for stuff that measured "like it's broken".  I've scratched my head over stuff that measures into "ridiculously low" distortion figures.  How does one reconcile such things?  My answer is easy, I hear what I hear and damn the torpedoes.  Fire away.

If people choose to buy gear based solely on what I write I can't blame them, I've done the same with mixed results.  But, anyone willing to part with hard-earned dollars for audio gear has to understand the risks involved.  They probably don't hear what I hear as Jeff has pointed out.  I mostly see audiophiles tending the opposite direction, a positive review is worth little more than a yawn or possibly one data point in assessing merit from afar.  At the end of the day there's no substitute for hearing it yourself and depending on the circumstance even That isn't always enough.

[Kevin Haskins]

Actually, we see really close correlation between speaker measurements and listener preference.   

A friend of mine runs a DIY speaker contest at the Pacific Northwest Audio Society every year or two.    He does a good job with it getting a panel of judges, all lifelong audiophiles and some that are in the industry.    All the speakers go behind an acoustically transparent cloth so all the listening test is done blind.    After the listening is done the speakers are carried over to another location for some quick and dirty FR measurements.    Everyone in the contest gets a printed view of their frequency response (on-axis). 

The loudspeakers that are chosen as the "best" by that panel are without exception the ones that measure the best in terms of on/off-axis frequency response.     

Also.... look at the research Harmon has done.   They have very conclusive testing showing that panels of listeners will pick speakers that measure well.    Is on/off-axis measurements all that is needed?   No... but it is safe to say that we can measure with a high degree of certainty the speakers that will be picked in the top positions based upon frequency response measurements alone.    I think other things are important too but they sometimes require more extreme testing at higher SPLs to present themselves.    There is also a very real loudspeaker--amp interaction that is difficult to predict with 100% certainty.   

So the measurements mean something and it correlates MUCH more to listener preference than does the subjective review.   I know that isn't a popular thing to say but it is the truth as I understand it.    It is based upon research (not mine) and experience.   

So... I'd have a hard time taking any review seriously that didn't include some sort of measurement.   Not because I don't value the subjective opinion, but it would show me that the review publication doesn't value a metric that is HIGHLY accurate in predicting subjective preference.     



 

[TONEPUB]

In response to porzios surly comment, we have chosen to stay away from things like the 350 thousand dollar wavac and the zanden for that exact reason.

Measurements aren't the end all because this is all subjective in the end.

People prefer the sound of one thing over another, no matter how it measures.

If everyone had ruler flat hearing from 20-20k we'd have a baseline to go
on, but it's not that simple.  If everyone had a room that had ruler flat
response with no nodes, dips or peaks, we'd have something to go on, but we
dont.  And if all recordings were made to the same high standard....

What I see this being about is just trying to expose you to more gear.

I guarantee that every one of you in the audience has somewhat of a
different system, you chose those components for a reason and to you
they sound fantastic. 

Many manufacturers have told me that they don't expect us to sell their
gear for them, they just want us to tell people about their products.

And even beyond sound, there are other reasons that people buy
hifi gear.  Some buy it for looks, some buy it for perceived status,
some buy it for ease of use, some buy it strictly for sound quality
not caring at all what it looks like etc, etc.

And the reason I have an EEE on the staff is to proofread our copy
and make sure we don't say things about circuit design, or related
things that aren't true or don't make sense.  I've read some pretty
squirrelly copy in some of the other magazines from time to time
where they explain electronics in a way that doesn't make any
sense at all.

Because any component can't be tested in every circumstance, it
is very difficult to just pass off something I don't like as "bad"

Is it ethical for me to review a component in one room, with one
group of associated components with ten albums and proclaim it
bad?

That's why I agree with Nils, on the rare occasion we get something
in that just doesn't measure up, we just send it back.

[Kevin Haskins]

I'm not trying to nit-pick you but we have the same issues designing speakers.   The room and system has an effect and there isn't any way to predict with 100% certainty for a given user what they will like.   It is a statistical thing but it is still meaningful. 

It isn't like this method only works for 1 in 10 people.   It works MOST of the time.   The measurement predicts with high certainty how people will react to the loudspeaker (better or worse).

In terms of other factors, Harmon also performed sighted test where users could see the loudspeaker they where judging.   They tested for cosmetic appeal and there was a HIGH correlation between how a listener perceived the product looked and the sound.   In other words, if it looked like it should sound good, it did.    They also did the same thing with brand identity.   Once again, there was a high correlation between brand identity and perceived sound quality.

Moral of the story; not only should you design a speaker that SOUNDS good, you better design one that LOOKs like it sounds good and company reputation influences the reality for most people.   

[gerald porzio]

Since most, if not all, technical matter in a review is merely a rehash of data from the mfgs.' web site rather than an independent, original evaluation of circut design, certain mags & Ezines would be better served having a proofrreader on staff rather than an EEE who measures nothing.

[Kevin Haskins]

Since most, if not all, technical matter in a review is merely a rehash of data from the mfgs.' web site rather than an independent, original evaluation of circut design, certain mags & Ezines would be better served having a proofrreader on staff rather than an EEE who measures nothing.

You know... most of the readership wouldn't understand the difference.   You are talking about a periodical that would be more appropriately aimed at engineers.    Give them a break.... they have a tough job to do and they don't necessarily have the funds to evaluate everything to the kind of standard you are talking about.    Depending on their readership, it may be totally outside of the bounds of what they want to read.   

If you don't like it, start your own E-zine and you can apply whatever standard you choose and pick whatever target audience you want.   

We are wandering off track though.   The ethical standards were the root question in the post.    For that, you don't need to have one bit of measurement gear.   It just requires a policy that is used, written, understood and enforced.   

[miklorsmith]

Kevin - I agree with you to a point about speaker measurements.  I use them in-room and any speaker review I do will have some in-room measurement info.  In fact, I will be embarking on just such a thing in the near future.  I use the information to dial in my subwoofer EQ so for me it's something I absolutely need to have.  Because I have the equipment and ability to use it, it will appear in whatever I write on speakers.  Electronics?  Very different animal . . .

It's amazing what a 2 db hump here or there in the bass range can do to the entire spectrum, all the way through the treble.

I've heard lots of great looking speakers that I couldn't stand.  That is A metric but not definitive to me.

[gerald porzio]

Ethical standards????? In a process irretrievably damaged from the get go? It begins w/ editors groveling before & cow towing to mfgs. to be the 1st  to review a product. Then they dare not give the component in question an unfavorable review lest they never see anymore gear from said mfg. Factor in the lack of standards for reviewers coupled w/ accomodation prices (potestations aside) & cables on permanent loan. No wonder one so often reads, "I bought the review unit". To shore up the review one, reads that his wife (poodle & parakeet) partook of this audio epiphany & remain enraptured.

[TONEPUB]

Since most, if not all, technical matter in a review is merely a rehash of data from the mfgs.' web site rather than an independent, original evaluation of circut design, certain mags & Ezines would be better served having a proofrreader on staff rather than an EEE who measures nothing.

No it isn't.  You are completely wrong and have no idea what you are talking about.

And please tell me about all this terrible gear that we are lying about.

I've been polite for the whole term of this thread and I am sick of your constant
thread crapping when you have no idea what we do, don't know any of us personally
and haven't heard any of our systems.

We do NOT rehash what's on manufacturers websites.  We spend anywhere from a
few hundred hours to a thousand hours listening to a component before we start typing
up a review.  Most of the time, we try a component in at least three or four different
rooms/systems and experiment with quite a few different component/cable combinations
to get a feel for a piece of gear.  More and more, I am sending gear to one or two additional
reviewers to get their take on somethings sound.  We also do this at our expense and it
can get pretty expensive when a pair of speakers or a big power amplifier is involved.

We also spend the better part of the day taking the photographs here in our studio
for the layout in the magazine.  We rarely use mfr supplied photos, except when
we have the opportunity to show a product in a different finish than the one we review.

We take this very seriously and spend a lot of time doing this.

I'm quite tired of having a grumpy old man constantly put down what I spend 8-12 hours
a day putting my soul into.

A proofreader that knows nothing about electronics wouldn't be very useful at catching
technical mistakes, would they?

We've written almost 200 reviews since the magazine launched.  If we were doing shady
things, we wouldn't have over 100 thousand readers in 92 countries and we wouldn't have
the best companies in the high end sending us their gear to review.  And we wouldn't have
people from Rolling Stone, SPIN, Jazz Times and The New Yorker working on our staff
either.

TONE is free for goodness sake.  If you don't like it, don't read it.  We aren't holding
a gun to your head to make you read it.  

[gerald porzio]

The reason these people are writing for you is that they can't make a living writing for only Rolling Stone, SPIN, Jazz Times & The New Yorker.

[TONEPUB]

Yeah right, Ben Fong Torres couldn't make a living at Rolling Stone
so he writes for me.

You really are an idiot.

[gerald porzio]

What about the other 3? Working for you might just beat being a stringer in Iraq or Afganistan.