I have almost the same setup...a Mac Mini with 512MB RAM. If I had it to do over again I would get at least 2GB of RAM. The mini seems RAM starved for just about any task that I do on it, iTunes included. I get the rotating beach ball a lot, which I assume means that it's waiting for a resource.
BTW, I'm also going via USB from the Mac to a Bel Canto DAC3, and I love it!