It probably would degrade it, but not by very much - balanced connections are better at rejecting interference than unbalanced ones, and converting XLR to Phono effectively reduces the signal volume by 6dB. There's no active "conversion" involved, just one wire connected to another.
Btw, a standard XLR to phono cable is wired phase-to-signal, ground-to-ground, but I reckon it's better to use a piece of "balanced" cable and wire it phase-to-signal, antiphase-to-phono-ground, and leave the cable shielding wired to the XLR ground - that way you retain most of XLR's ability to reduce ground loops....