I agree with others that there may be many reasons, but one is constant change in our minds and emotions. Are you always in a receptive state for anything? For better or worse, I'm not. I find the same thing even at live concerts. Even the best ensemble won't captivate me on some days as much as others. Is it the playing, or is it me?
A recent thought is that perhaps we respond differently to imperfections in our audio systems differently at different times. Removing as many as possible could make our responses less variable. I have a highly treated room, but I was sometimes bothered by a high-frequency echo. When I killed that with an absorber, I found I enjoyed the music not just more, but more consistently.