A headphone amp connects directly to your source and does NOT go through your receiver so there is NO additional connection and even less distortion. You connect the headphone amp directly to your CD or DVD player via interconnects.
Is the sound better? Yes, for a good headphone amp, it will be better.
Is it worth it? Only you can decide that.
My experience is that a good tube headphone amp (Mapletree) + the right tubes (still rolling) + an aftermarket headphone cable (moon audio black dragon for 701 with dual connections) make a substantial difference in MY listening enjoyment.