The 804's are a little on the sensitive side to begin with - at about 90-92 dB.
As a contrasting example, when i hook up my B&W bookshelves (CM1 S2 i think) to my 4BSST2/BP20 combo - i can crank the volume knob damn near to 12 o'clock on those little speakers - with the gain set at 23, and with balanced cables. and they sound deadly. i believe this is in part due to the low sensitivity rating for the CM1 S2 speakers - at around 84dB.
So, I'd say your 804's are definitely part of the equation to your volume dilemma.