As tempting as it is, I wouldn't starve low/medium efficiency speakers, even if you only listen at moderate levels, with 20 wpc.
I tried 7 wpc on 90 dB/w/m full rangers and it was very good, but 40 wpc tightened bass, enhanced dynamics, and improved resolution (detail and imaging).
IMO for hi-fidelity listening, the amp needs to have a commanding grip on the speakers.