Not that I know of, but usually what people do to improve the sound is either:
1. Get an outboard DAC that has USB input (this usually yields the best results for 2 channel audio)
2. Get an outboard DAC with optical toslink input (not usually the best, but your mileage may vary)
3. Get an AVR or AVP, use toslink, and you're all set for for 5.1
Macs come out of the box ready to provide high quality USB audio output. Windows you have to futz with more. Some USB DACs (some have toslink too) 2-channel DACs are:
Scott Nixon
AudioSector (my personal recommendation for a DAC less than $1K)
Benchmark with USB option
Wavelength Brick, Cosecant, Crimson (the cosecant and crimson support high rez USB)
Red Wine Isabellina
Bel Canto
PS Audio Digital Link III (usb, toslink, spdif coax)
MHDT Laboratory Paridisea+
Empirical Audio
There are plenty, plenty of outboard DACs that have TOSLINK and SPDIF coax, but no USB.
All you need to do optical audio output from your Mac is a mini-toslink adapter.
If you want to do 5.1 this page may be helpful:
http://www.wesg.ca/2008/01/using-true-51-surround-sound-with-a-macbook/But you'll need to find a DAC with 5.1 processing capabilities, which I don't know too much about because I'm more interested in best music reproduction, not DVDs. Presumably any AVP and AVR would do the trick here.