Naw, keep the swap, remove unnecessary services so things don't swap. i will have to check my configuration to even see if i have a swap file enabled, but it's easy to remove the swap, just edit the /etc/fstab file and comment out the swap line and reboot. I assume you are running MPD on the rPI?
Keep the raspberry pi headless, without a gui, xmbc and all the unnecessary services that you don't need and that's a good start. If you can paste the "systemctl status" here, I can probably tell you what to kill off.
BTW, some dacs don't handle PCM to DSD transitions or just off to PCM or off to DSD, but if you read through the MPD changelog, the developers there have been addressing those issues.
Jim