If your CD player is not truly high end, and you don't want to spend too much, the best upgrade in my experience is to replace the crystal oscillator with a low jitter master clock. These are cheap to buy, a few hundred dollars in Aussie, and improve treble and bass extension, staging, imaging solidity and resolution of midrange detail (vocals will be easier to follow). No downside in my view. The only examples I have had experience with are LC Audio XO clocks (now up to version 3) which are available in Aussie via Soundlabsgroup, or some such. The only thing you must do before ordering is find out what frequency your crystal oscillator works at. There are sites on the net that offer this information, including the aforementioned.
While you're at it, check the op amps in the analogue output stage, and replace these with higher grade faster alternatives. This is another even less expensive expensive upgrade. After doing these two things, your $1000 player will sound like a $4000 player or thereabouts, and you will have spent about $400 (Australian). Then mess with caps!