Well, how do headphones work then? I'm pretty sure good speakers of any size are fine at low volume.
Headphones, though indeed are small speakers, take the room and largely the power and efficiency pieces out of the equation that home speakers deal with (in general). I know you know this.
We're not saying that it isn't possible for all (properly designed) speaker types and speaker sizes to sound good at low volume. It all comes down to listening space, power requirements, and listening distance......not accounting for things like DSP and other types of room/sound equalization which can profoundly affect all the above.
We all have different listening spaces and listening distances and volume level preferences, and thus we'll have varying power requirements and speaker requirements. Choosing the right speaker per the above, can be more complex than many realize.