The SB3 is not designed to be used with a balanced output. Not to mention there is no place to mount the connector on the unit.
Actually, I was thinking a "pigtail" of sorts.

In any case, is the new RCA connector that you use in the mod a 75-ohm connector?
I suppose I could hook the coax output out to a Perpetual Technologies upsampler, then output via AES/EBU. Although I have found the Perpetual unit to actually impart a signature of it's own.

I'm really hoping that a modded SB3 will finally be
the solution for me. I've been meticulously ripping my library, and rewriting them onto expensive gold CD-R's to get better sound! I know--into the realm of "he's nuts," but I can't help it...
BTW, it was the owner of Theta Digital that turned me onto the Squeezebox after I told him about my Roku experience earlier this year. Not sure he was aware of the availability of third-party tweaks.