What is the major drawback of D amps not being good hi-fi ?. Pardon my ignorance on this.
Generally, output impedance, aka source impedance. Most class D amp have output filters that raise the output impedance enough to interact with the frequency response of typical loudspeakers, and cause FR anamolies.
Tubes have the same issue, which is one reason why many tube-heads have become Class D enthusiasts, and why many of the fetishized Class D amps are, objectively speaking, low fidelity devices.
The Hypex modules are an exception to the rule, maintaining flat FR into most speaker loads with noise performance only bettered by the most refined Class AB designs. The latest Icepower designs are also better, but still not quite there.
I suspect the main reason firms such as Bryston don't widely adopt such things is because they don't want to be dependent on someone else's IP for their mainstream products. Bryston's standard product is of a highly evolved, high performance, and reliable design. The efficiency gains from Class D (and lower heat output) compared to well-optimized Class AB are also arguably less important in home audio than they are in, say, cellphones/car audio/whole house audio.