The typical non-audiophile actually spends MORE on the typical system owned because they rarely listen to it. Lots of people have purchased POS systems from best buy or circuit city and for the most part they play music 3 times a year for parties. So at that rate... $1000 invested and used for maybe 20 hours over the course of a year = $50/hour.
My system cost me about $10k. I used it approximately 500 hours last year = $20/hour. Less than half what a non-audiohile spends. I'm not joking here. People drop $500 or $1000 on stuff all the time that gets very little use. Or how about clothes that get worn once or worse still have the tags on them.
I don't begrudge people their hobbies at all. If they enjoy them, then I'm happy for them. However, I agree that our hobby does get more than it's share of disrespect. Mostly due to what I'm willing to admit is a charlatan element to the hobby. That's unfortunate, but the positive side of the hobby makes it easy to defend.
Music is not degrading to your character or your soul (or at least my music isn't). Music relaxes. Music is uplifting. Music has been shown to have many positive health benefits. Music can be enjoyed with your family. Ask a golfers wife which hobby she'd choose for her husband.
Once you buy a high end system, the investment is over (in theory

). Most other hobbies require much more on-going expenses. Some people use the same high end system for 20 years (say until their Bryston warranty runs out). Show me a golfer, tennis player, car enthusiast, damn near anything that's still using 20 year old gear.