I successfully run a Sumiko Blue Point Special at 2.5mv output. I don't think you will have trouble, though you are starting to get to the end of the rope. And you are right, the noise floor will start to rise, although the AVA phono board is very quiet. You will just have to try it and see, but I do think you will be alright.
Wayner