[VinceT]

I have cubed shaped room, roughly 25x25x20. The room is a little larger but there is a central fireplace about 1/3 into the room so acoustically I am not counting the space behind the fireplace only in the main living area where the system is. It is also sunken living room with open sides, beautiful mid century modern, but acoustically very challenging.

Pulling the speakers out about half way in to the room nets the best results so far., but there is still some room mode issues. Also I found the speakers right up against the wall each side of the fireplace really improves consistency with the acoustics and tames the room also sounds pretty good. Room treatments are really not an option due to the architecture and design of the room, plys WAF has put her foot down to any major room treatment ideas. I could do a couple panels behind the speakers and get away with it.

I noticed many mixing studios build the monitors into the walls to limit room modes. Has anyone done what with success in a challenging room and what are the pros and cons to such an endeavor?  I do have bookshelves that I could do this with.

[youngho]

Soffit- or flush-mounting speakers is not about room modes but rather about reflections off the front wall (the wall behind the speaker). The room modes themselves are not affected, and if anything, the length axial modes (front-to-back) would be maximally excited by flush-mounting the speakers. Most home loudspeakers are not designed for flush-mounting, so some degree of equalization or bass tilt would be required. Downsides would include cavity resonances if seamless mounting were not achieved, also likely increased noise transmission outside the room.

[richidoo]

Front wall reflection smears mid bass tone, making a chesty warm muddy sound.
Moving speakers closer to front wall shifts the blurring freq band higher so it may be less noticable or more easily damped. But speakers voiced for 2 feet out will have too much bass against the wall. Often a compromise position can be fojnd which minimises the bass blur without boosting bass volume too much. I
In-wall speakers eliminate the front wall reflection altogether, to clean up the bass without excess bass level.
The cause of the tone distortion is SBIR (speaker boundary interference response.) There is a FR cancellation dip at the listening position, but to either side of the cancellation freq there is also phase shifting which smears the direct sound and causes the blur. Acoustic absorption does not fix it, ime. I prefer flat untreated front wall, or flat diffuser.

[Tyson]

Sure, you can do in-wall speakers, but they need to be designed from scratch to go in-wall.  They can sound quite good from a tonal and frequency response standpoint.  But the thing they lose the most is imaging. 

[VinceT]

  But the thing they lose the most is imaging.

That is a tough trade off for me

[Mike-48]

As @youngho mentioned, the major issue with the front wall is SBIR -- speaker-boundary interference response, which causes cancellation at a frequency whose quarter wavelength is the distance from woofer to boundary. For that reason, it's good to keep the distances different from the various boundaries (side, front, ceiling, floor). The first step is to make sure the distance from woofer to front wall is different from the distance from woofer to side wall.