Hyperacusis is a real phenomenon that often accompanies hearing loss and tinnitus. In this case the compression function of the audiovestibular system ain't working right.
Phonphobia is a dislike of loud sounds, and misophonia is a negative emotional response to some kind of sound (nails on a chalkboard, etc.). Listener fatigue is probably some special kind of misophonia.
If listener fatigue is real, my guess is that it must originate further up the central nervous system rather than at the middle ear, the cochlea, the auditory nerve, or the brainstem. The reason I say this is that it we can record the electrical output from these parts in response to sound, and as far as I know the responses don't decay.
This is kind of funny; I think I suffer from almost everything you mention here (except maybe phonophobia).
Having experienced listening fatigue, I don't think it's really the same as misophonia. Similar at times, perhaps, but not the same. Misophonia usually occurs for me for specific noises (people chewing, rustling of plastic packaging) and results in emotional (often visceral) reactions that I'm not always aware of at the time. For example, I'll be sitting on the couch, minding my own business (i.e., spacing out) and suddenly find myself getting irritable only to later realize it's because my girlfriend is rustling a bag of chips or something.
Listening fatigue, for me, is more related to hyperacusis or actual distortion. I don't experience it too often anymore, but it's usually centered around glaring mids, like trumpets or female vocals, and usually in a too-loud bar or similar place. It's more of a feeling of "Oh, God! Make it stop!" rather than wanting to put my fist through somebody's face, which is what misophonia is like.
*Or maybe I should say that hyperacusis can lead to listening fatigue? That's probably more accurate than to say that they're related.
I used to experience listening fatigue semi-regularly when I was younger and used cheap headphones and cheap CD players. Usually it would happen if I fell asleep while listening to music or if I had been listening for more than a couple of hours at a time. From my experience, I'm going to agree with what others have said that it has to do with distortion and my brain interpolating what I'm hearing into something that makes sense. I think of it as "mental dithering" and after a while my "processor" just can't handle any more and has to cool off.
*I would like to clarify that as a general rule I don't want to put my fist through my girlfriend's face. Though sometimes she does think it's okay to eat Airheads in bed. *rustlerustle* *glomglom*
*cringe*