Dave,
I wish you had included specific dimensions because that's a big factor. What's appropriate for a room 35x25 feet is not necessarily appropriate for a room only 10x12 feet.
> Been reading up about diffusion, and how absorption can be bad in a small room because it robs it of kinetic energy <
I don't know about the kinetic energy angle.

The main issue is having enough ambience that the room sounds natural, but not so much that you can hear repetitive echoes or speech clarity is compromised.
> whereas diffusion will correct soundstage and imaging problems <
I'd say it's the opposite, at least when considering treatment for the first reflection points. In a typical small room where the side walls are less than ten feet from your ears, absorption is probably a better choice than diffusion. In a very large room diffusion can be good, but
only if you want to retain all the ambience. Usually this is not the case, especially in a listening room versus a room where recordings are made.
Also, diffusion is often used on the rear wall to make the wall seem farther away and make the room sound larger than it is. So the rear wall is a more likely candidate for diffusion than the RFZ places on the side walls and ceiling. It's mainly the RFZ places that affect imaging and "sound stage."
> My question is whether there are any drawbacks to diffusion? <
If it's "good" diffusion, like RPG's QRD models, then being too close can sometimes give a strange sound. Those diffusors have a series of chambers that resonate at different frequency ranges. So when you're too close you can hear each individual resonance, which is not unlike setting one band on an EQ all the way to full boost. When you get far enough away all the resonances combine, so that problem goes away. With cheap/lousy diffuser this is less of a problem, but those don't diffuse well anyway and so are a waste of money IMO.
> Assuming the corners were trapped with fiberglass, and bass response is not a problem <
That's a mighty big assumption! How big did you say the room is?

--Ethan