So basically, there is NO advantage (or disadvantage) using for example a RCA to XLR connection as there is no balanced output from the RCA end?
Well, there's some advantage and some disadvantage. The RCA to balanced input is a pseudo-balanced situation where the CMR (common mode rejection) ratio is seriously compromised compared to a proper balanced interface and ofttimes you won't have sufficient audio levels as both the (+)ve and (-)ve signals are expected to contain the signal with an associated maximum line level. One of your inputs will be at ground potential.
Conventionally the two signals are fed into a differential amplifier at the receiving end and the CMR advantage is fully realized since the two signal carrying conductors have identical impedances to ground (as in a proper balanced circuit). This will give the maximum noise rejection. Your grounded input will degrade the CMR since there is an impedance imbalance.
Yes, you'll still realize a small CMR advantage compared to an unbalanced to unbalanced RCA connection, but it's quickly lost in practice. The level out of the differential amplifier is at a 6dB (half the voltage) disadvantage to a proper symmetrical signal. So to obtain the same level you must increase the level from the source with resulting raised noise floor. Your advantage is lost.
Using RCA to RCA would be best.