Do the balanced preamp outs mute when headphone cables are inserted?
No, all the outputs remain hot, regardless of whether you use 4-pin balanced or 1/4" TRS. If you need to mute the rear preamp outputs when using headphones, you'd need to turn the amp(s) off.
<edit> When the BHA-1 was first released, it did not have the rear panel XLR outs, but Bryston realised that with some relatively simple mods the facility could be added, thus allowing for rear XLR connectivity and use as a fairly minimalist preamp. Thus, it lacks such niceties as output muting when using 'phones and other common preamp features. Still, it's fundamentally a headphone amp, and a very good one at that!
Regards,
Syncytial.