Thanks guys.
I still have to talk to my dealer about it. It's a lease, so I'm a bit hesitant about pulling the doors apart, especially since I have zero car DIY experience. I feel like it should not be this bad, I mean my Audis never had any issue like this, even when blasting bass heavy music. It even happens at very moderate SPLs. It's really noticeable when the music is relatively sparse except for bass, like in jazz and acoustic music. With 'louder' styles, the other stuff covers up the buzzing.
Anyhoo, maybe I'll check YouTube for some tutorials on installing Dynamat, see what's involved.