I think I may have figured it out. In the Audio System's page in Audirvana, there is a feature called 'maximum memory allocated to tracks'- I had the slider on "maximum", so I moved it to the lowest amount and I haven't had the problem anymore. More time will tell.