Yeah, why does this audiophile thing even need to have anything to do with whether someone or how much someone loves music? To me, audiophile is a general attitude of improving quality and fidelity of sound output. Hell, there could probably be some audiophiles out there solely interested in high quality reproduction of recorded speeches, etc.
The point is, when I sit down in my sweet spot after work with a glass of scotch (the combination of which is something hardly anyone except audiophilic music lovers do anymore), I certainly am listening to and enjoying music, but that's not what makes me an audiophile. What makes me an audiophile is the analytical mindset that comes along for the ride and constantly bugs me with the question..."what if I did...?" Fill in the blank. It's a tinkerer's mindset, always reaching for the next potential improvement, regardless of where I am or how good things currently sound. An audiophile is never really at rest, at least in the mind.
It's an attitude toward improvement, whether or not the audiophile has the financial or other means to act on that interest. For some, that leads them to DIY. For others, it leads them to reaching up into rarified ultra exotic components. For others, it leads to endless swapping of components and trying different system design philosophies. For poor college students, it leads to scouring pawn shops and garage sales for hidden treasures on the cheap, and super deals if possible on Audiocircle and Audiogon classifieds and such. For a few, it leads to wistfully looking at living rooms full of the wife's knick-knacks and wondering what it would look like with equipment racks, speakers, and acoustic treatments instead.
To me, the music is irrelevant to whether someone is an audiophile or not. Of course almost any audiophile would say they're a music lover or enthusiast, but it has nothing to do with that unquiet urge for improvement within.