I don't listen at reference levels, either. Just too loud for me. I'll crank it up on occasion, but up to maybe -5 dB, usually more like -15 dB.
One of my goals for '03 is a more elaborate system of room treatments for my HT room. The system is parametrically eq'd in the bass end and has some room treatment, but I'd like to work on integrating some tube traps and some DIY absorbers and a bit more diffusion. I think the sound would hang together a bit better then even as the volume increases.
Even then I don't think I'll ever do much listening at 0 dB!