From what I hear, and it sounds like you have first-hand experience, misophonia can be a real drag. ugh.
I think of hyperacusis as an abnormally low pain threshold for sound. So, the sound only has to be there for a moment or two; if a sound is going to be painful it's going to be painful almost immediately. The treatment for hyperacusis and tinnitus, by the way, generally should NOT involve earplugs. Too much quiet has a good chance of making it worse.
Misophonia is mostly only a drag when I'm tired or irritable or stressed. Otherwise I can mostly actively tune it out. Even so, I don't really think of it as a drag so much as some people's chewing just sounds extra gross to me.

I'm sure other people suffer from it more acutely than I do. It's kind of an interesting phenomenon and I'd like to know more about it as I've just recently learned about it. From my own experience, I wouldn't be surprised if it was actually related to some sort of emotional recall. I can distinctly remember early incidences of each of my trigger noises.
I should admit that I'd never heard of hyperacusis until last night, but reading a little about it sounds like what I've experienced in the past (just today, actually). If it's what I've experienced it
is a low pain threshold for certain (loud) noises, but for some reason they seem to cause resonances and distortion that nobody else hears. I'd explain it like this: imagine the pitch of, say, an ambulance siren at a high volume; normally it starts out being clear, then as it gets louder it becomes slightly uncomfortable and distorted, and then can suddenly sound like a speaker severely clipping, which can be pretty painful.
I used to have a Dayton DTA-100a and the way it would distort when playing Miles Davis was very similar sounding.
Also, I'd argue that listening fatigue is probably mostly just fatigue, as somebody else already suggested. Will your ears get fatigued after listening to something at a high volume? Sure. Will your brain and ears want to give up after listening to something really intensely for hours on end? Most likely. Will your ears feel fatigued when you're otherwise feeling wiped? Makes sense.
I think my point is that there are probably many factors that should be taken into consideration when trying to study listening fatigue. Rather than trying to answer "what is the
one reason for listening fatigue?" instead take a more holistic approach, taking into consideration sound reproduction, environment and room acoustics, as well as the listener and all of his or her moods and conditions. Sounds daunting, but I'd bet it'd give a better representation of what's actually happening.