What you are asking about really has nothing at all to do with the software used, but everything to do with the soundcard.
SPDIF is inherently a 2-channel medium for PCM data. There is simply no way to get more than 2 channels of uncompressed PCM data through an spdif link, regardless of what program is doing it.
The '5.1' features on DVD etc are accomplished via lossy compression into either AC3 or DTS. The NVidia Soundstorm chips are supposed to be capable of 'realtime' encoding of 5 channels into an AC3 stream, but this is a bad idea - compressed 5.1 is noticably inferior to the original in the best of cases, and the realtime encoder is even worse than that.
The only worthwhile way to go 'all digital' would be to get a soundcard with multiple spdif output, and then use one of the approaches already out there. Unfortunately, these aren't cheap. The Lynx AES-16 offers either 4 or 8 stereo outputs (although you'd need an AES->spdif xformer for each) and runs ~700 or so I think. The Emu 1820 has 2 spdif outputs plus a bunch of analog outs, and runs $400 I think.
The Emu has no linux support, but 4Front has beta drivers available for the AES-16.