Even though this is a chicken and the egg question, and even though I prefer to keep all of my components at the same level of quality, I agree with werd. I think we probably don't spend enough money and time on our source components compared to the rest of our system.
One reason might be that it is normal for people to get hung up on speakers and amplifiers instead of source components. With speakers it's easy to see why we get hung up on them. They are the part of the system that we see creating sound. They impose themselves upon us! No matter how logical we think about the entire chain of components, I think we subliminally look at the speaker for being responsible for most of the sound. I guess it's just in our nature to do that.
It seems quite normal then for us to obsess over speaker solutions that are far, far from being system friendly. We dismiss the importance of the rest of the audio chain during this obsession. Too many speaker solutions solve problems in an idealistic bubble and we fall hard for that kind of thing. But perfect speakers don't come with an amplifier to drive them, nor a room to put them in. That's always somebody else's problem. Where was the amplifier designer on the speaker team? Which part of the anechoic chamber resembles my living room? Were there test tones or music recordings used to voice the speakers? There's no time to consider that now, because I'm shopping for the perfect amplifier to drive my perfect speaker . . . .
Let's face another fact. Most of us are obsessed with power. If a little bit of power is good, then a lot of power is better. It has to be. We need lots of power on tap to handle our perfect speaker. Power is not cheap and large amounts of power makes for a complicated circuit. The price that you pay for big power is a severe loss in sound quality. The perfect amplifier is now the week link that is placed after the source. This makes it even less interesting to explore the differences in source components.
So it's kind of a funny situation. You could say that the amplifier and speaker are more important to get right so that you can clearly hear the differences in source components and source material. But on the other hand, once you have a really good monitoring system, you are open to the fact that you are not getting everything out of your recordings without a good source component. And also open to the fact that a better source component will extract more of what is locked in the grooves/pits of those recordings that you already own.