I think some space is a good thing. I like to increase cabinet flange recess .020-.030" from actual measured dimension. Finish of any type can easily build to .002-.005 making a snug fit raw too tight. I've filed driver flanges in some cases because it's easier than increasing diameter of flange recess. If finishing in lighter color, I often run a black felt tip pen on vertical wall of recess to create a shadow in that area. I don't like seeing a light ring around driver and just darkening it makes it nearly invisible.