Michaelv,
You've probably heard the expression... "less is more." I agree with _scotty_ about the importance of the source material (cd) and source player. Find the best cd you own or can borrow, one you just love listening to because the recording is so wonderful that there is total absence of any grittyness, distortion, siblance, digital artifacts, etc. No matter how reasonably loud you play it, it does not compress and everything about it sounds great. *IMPORTANT* If you don't have a recording that sounds like what I just described, then you definitely have a weak link or two. I'd would start evaluating your system from the front end... the CDP. But, in truth, any weak link can have a very noticeable effect on the overall sonic presentation. However, chances are you do have one or more of those special recordings which will allow you to hear just how good or not so good things are in your system/room. Your system may not be as bad as you think, but regardless, it can always be improved.
I have a very simple system that is easy to evaluate. Each piece of gear has been modified and the sound is scary good on many CDs, but others (more than a few) sound like trashy crap! It's been hard for me to accept that many cds are poorly recorded in today's high tech world. They are just fine for the car where road noise is always a factor, jambox, portable cd player, or for the average midfi home system. The average listener will likely fit into one of these categories. But any high resolution system will leave you shaking your head when you hear what is really on that disc. I use to believe that Redbook standards should somehow prevent this from happening. Maybe I dreamed it, but it ain't so! I apologize for digressing.
BTW, before I upgraded my midfi system, someone advised that I remove my modest SS preamp and try a DacT CT2 10K stepped attenuator since any 2 volt CDP has enough output to drive the system with 90db speakers like mine were at the time. So I bought a used DacT and assembled a simple passive volume controller just to experiment. That single experience really openned my eyes to how much difference one change can make for the better. I would not have believed it if I hadn't heard it. That's just an example... not necessarily a recommendation. Any weak link can make a big difference.
One final thought. As others have said, you can assemble a very nice sounding system without investing lots of $$$. Hope this helps in some way.
-- Roy