Your diagram shows you also have the subs double connected when you don't necessarily need to. That may be causing the problem. If running them off the dsp, don't connect them to the subwoofer outputs from the receiver. It shouldn't create a problem, but could be making it wonky. Run them just off the dsp connection, set the receiver to no sub and full range output on the front main preouts and see what it does. 🤞
Next option would be to diy a speaker level to line level converter with couple resistors and run your setup that way. It'd be using front speaker outs, speaker wire to binding posts on a small box, resistors to drop the level, then rca outs to your dsp. A little funky and a lot of connections in the signal path, but equally effective in a pinch. I did that to run my subs off my tube integrated, and did it again to run my headphones through there as well.
Final option would be to get a full Pre/Pro and separate multichannel amplifier, and that would be a beneficial but $$ upgrade.