The true test of a system is, can it play loud and still remain pleasant to listen to.
"System" includes the acoustics of the listening room and may be the largest part of the problem of harshness increasing with volume levels, assuming the amp is not stressed. Reflections off a sheetrock wall are not a perfect match of the original sound. They are colored by the resonances of the reflecting surfaces. The closer the resonant vibration of the reflecting surface matches the frequency of the music the more coloration is added to the reflected sound - just like speaker cabinets. It is the "one-note-bass" phenomenon, but it can happen at any audible frequency depdning on the reflecting material and the vibrating span.
The perception of distortion increases as the volume increases because the distorted reflections come from every direction while the clean direct sound comes from only one direction. Corners and especially tri-corners dstort sound even more than flat walls because multiple reflections are created at the same time, out of phase with each other. Corners also act as horns to amplify the distorted reflections as they exit the corner.
Because sound attenuates exponentially, putting the ears and speakers as far away from reflective surfaces reduces distortion at a given volume setting. But perceived distortion rises exponentially with volume in an untreated room acoustic as everyone has observed, so turning it down really does make it sound better, not just quieter.
But music needs SPL to deliever the intended feeling, and there is a range of volume levels that work well for every piece and every acoustic, and it can vary from recording to recording, even track to track with the mood of the recording and the listener, making a remote volume control a critically important tool. A display of volume level is also important, with sufficiently high resolution to show small changes. A mastering engineer should adjust the levels of a CD so that the listener doesn't have to adjust inside of a CD, but that level of detail is not typical, especially on tracks intended for pop audience or radio, where max SPL means everything, and subtlety means nothing. If a listener likes a tune he may get more juice by cranking it up, so the right level is different for each song and listener. Tweaking the volume by 1dB can make a big difference to the overall enjoyment. A well treated acoustic allows that same level of enjoyment over a broader range of volume settings because it is quieter, to allow hearing well at lower levels, and because it controls reflections to allow lower distortion at higher levels. Full range speakers with excellent coupling at bass frequencies (large drivers) will allow satisfying listening at lower volume settings.
Whereas some materials like glass, metal, gyp-board, etc are not complimentary in their colorations, some materials like spruce and maple do actually sound good, which is why they are used to make vibrating surfaces in musical instruments. A listening room made of materials like this may have equal or even greater THD than a sheetrock and glass room, but character of the distortion is more pleasant or more "musical." Some ideas fwiw...
Rich