It won't affect sonic performance in terms of fidelity. It would improve latency problems (music skipping) if you are running on ancient hardware. Defragging helps if you are adding and deleting lots of files over time, but I reckon that most people using a dedicated harddrive for music will simply add over time, so fragmentation will be minimal. The OS will read large chunks of data from the disk during playback and cache them as it doles out the music stream over the network, so unless your music is skipping, the caching mechanism of the OS will compensate for any fragmentation you may have. The network will be the bottleneck, not the drive.
It would be analogous to changing your car's oil every 1000 miles. Yes, it has benefits for the engine, but it is overkill and not necessary.