A better way to compare is to listen to your new Vision DAC for at least a couple of weeks and only then put the old CD player back on to compare.
That's an excellent suggestion. Sometimes it's really easy to spot whether something is different, a bit harder to accurately decide whether it is an overall improvement. Sometimes, it's not so easy to explain the difference, which leads to psychoacoustic grammatical congestion.
If you really like the DAC, then you will miss it when you take it out of your audio chain, specially once you're used to it. You can make up the words to explain it later.

PGC, a common malady of our times... anyway, when I got my ultradac and did the instantaneous a/b test against the dacmagic, I would have "honestly" stated that it hardly made a difference. But as the song progressed, something in the mood was very different, and much better!