yes, since the DAC inverts phase you should (theoretically) swap the + and - of the speaker cables at one end (either the amp or speaker side).
But in practical terms, even though I know and can hear phase differences in a direct A-B test, throwing the rest of your sources out of phase by doing this may not really be worthwhile. Especially since most CD's have a "guess which phase I'm in" mentality.
Enjoy,
Bob