When we were shopping for homes in Durham, NC, we looked at a couple of "passive solar" places. It seemed they had all been built by the same designer sometime in the 70s. It seemed like a great idea, but the fad had never caught on.
There were some features of these homes that could easily have been unappealing. I loved the way that almost a whole side of the house would be covered with windows. But the downside was that the other side of the house tended to be devoid of natural light. These days, use of things like solar tubes could alleviate some of these concerns.
Then there was the choice of flooring material. Desirable materials are those that will stay cool when you want the house to be cool, and will retain heat when you want the house to be be warm. That usually means stone or brick or tile. Now, I like many natural stone floors. But at least one place we looked at had brick floors, which was much less appealing. And let's face it, even the finest slate is unlikely to be an audiophile's first choice for flooring.