I also find it's quite possible to overdo it with the NoRez. Over time Ive come to a method that works really well. I put NoRez on the top panel and the back panel and I use a lighter material on the side walls. Specifically recycled denim insulation. It's cheap, readily available and easy to work with (just spray glue one side).
And I have also found it's necessary to experiment with how much stuffing to use. If I've lined the walls of the speaker thoroughly with NoRez and denim insulations, often times additional acoustic stuffing is not needed. Or, very little. You have to try it though. Start with none, then add a bit, listen, add a bit more, listen, etc... eventually it will sound muffled. Reduce the stuffing by 30% to 50% and you will generally hit the sweet spot.