I'm definitely not an expert on audio drivers, or the USB protocol. But from what I briefly read on the protocol, it uses a technique similar to TCP; data is separated into chunks; each chunk carries its portion in the payload, and there are some header packets attached to each payload that help make sure the data isn't mangled on transport.
So the 'what' to measure would probably be the payload data. Attach 'some' software (if it were a network interface, that sw would be Wireshark or similar) to the USB interface on the computer to watch the packets. Record each payload, aggregate, and store in a file. Then offline, take that file and reproduce the original PCM file. Then do a bit comparison to the original PCM. Do that with the different versions of the build, etc.
Thing is, that sort of thing should be done by Amarra, and automated, and incorporated into the build script; so that the build fails if anything funny is going on with the data during playback, and then that version of the sw isn't yet released.
Also, since you mentioned that the effects are a bit elusive, I'd actually put my money on some sort of timing (not data) interaction between Amarra and the async USB driver. As far as what that interaction is, I'm not sure; as I don't have the low-level driver knowledge required to figure that one out. But I hope the folks at Amarra have that expertise.
It is reassuring to hear that you and the lead Amarra developer are in direct contact.
EDIT: It turns out that Wireshark can also monitor USB interfaces: http://wiki.wireshark.org/CaptureSetup/USB
. Might not be OS X compatible though. But it's a start; shows that there's probably some decent software out there to monitor this interface on OS X.