analog97,
To, me, it seems that you are grounding the tonearm when you touch it. If you connect a wire to the motor of the TT so that you have a ground from where TT power plug is connected. When you are holding this wire, touch the tonearm and see if it still buzz? If not, then it could be that you are experiencing a ground loop. Now, ground the chassis of the tt to this wire, then touch the tonearm again and see if the buzz disappears.
In my system, in between tracks or when playing or when idle, it is hum/buzz free. My speakers are 100 dB efficient. If there's a noise (hum oe buzz), I am pretty sure I will hear it. BTW, I am only using 1.5 watt amplifier. So yes, I could claim that my vinyl playback is quiet.
regards,
Abe