I would submit to you that significant research has been done in this field several very talented individuals (not just Toole). Their conclusion has been that the room contributes to the sound and that it's generally best to work with the effects rather than against them (by turning your listening environment into an anechoic chamber). This has been my experience too... when using the right loudspeakers.
http://www.amazon.com/Sound-Reproduction-Acoustics-Psychoacoustics-Loudspeakers/dp/0240520092
If you don't believe their research you can always resort to using headphones. 
Of course, treatment and placement and calibration can all go very far, but there's certainly more to the room than what a measurement looks like. My 10' x 10' studio measures wonderfully well, everything is at an acceptable decay time, its not overtly "dead" by any means - and sounds really great. But sometimes, it doesn't sound nearly as good as a decent sized control room, even if my studio measures better.
The room always contributes to the sound, but in a larger room, you simply have better modal extension in the low end, much more time for low frequency waves to fully develop, more room for the high end to naturally die out without sounding bright and needing to be absorbed, etc. A larger room just gives you a better slate to start from, and gives you much more to work towards. In small rooms you only have certain options available -> in a large room you have tons of different concepts you can use and treatment methods that just aren't possible in typical residential sized rooms. This is what they were getting at - not that a large room doesn't have problems.
