Flint Dave makes an excellent point. Much of what we hear is limited to the recording itself. As studio typically have crummy monitors and the room setup is laughable its a wonder that even 1 out of 100 recordings have any audiophile grade quality to them. Often I have to chose between musical content and quality of recording when selecting wht to listen to.
System building/synergy is something we each struggle with.
The types of distortion from various components in my mind breaks down into two kinds. One kind is obvious and irratating (clipping, boomy bass, etc.). The other is more of omission (poor imaging, lack of dynamics, etc.). Omissive distortions are more easily accepted as the brain fills in the blanks. OTOH this extra brain effort leads to listener fatigue.
As Dave mentions, the recording is the most important component. My next priority goes to speakers (the only transducer in a digitally based system). Then the room. Some would say, "garbage in, garbage out," and so would make the source a higher priority. But a decent source (guilty primarily of omissive distortions) has been easy to find the last couple of years. The same is true of amps and pre-amps. Amazing what a good amp/pre-amp can do, but equally amazing of how easy it is to live with a medicore one too.