In an ideal world, you would have a good buffer size and zero latency.
In a perfect world, you wouldn't need a buffer because data transfer is instantaneous and perfect and immune to everything.
In the real world... that doesn't happen. Think of your buffer like a holding stage where the music data goes before it hits the actual digital to analog circuitry. The circuitry pulls data from the buffer as needed, so if your source feeding it is not perfect or it has skips or something momentarily affects your computer, that's ok because the buffer is the reservoir that separates the circuit from the data.
Now the thing is, if there's a big buffer then there's going to be a delay before any changes you make to the data will come out the circuit side (you have to fill that pool with the changed data for a while before it starts to come out). For those who work in live sound or the studio where you have a lot of processors and loop effects, delay is bad because it affects how fast you react to things, and if the delay is particularly bad then your processes and effects may not sync up properly.
For hifi use, you really don't care about the delay (unless you like to spazz track changes all the time), so leaning towards a higher buffer is perfectly fine.
Also, if you're able to use the NuPrime ASIO drivers, I'd recommend that over WASAPI.