Ferrofluid tends to smooth the driver's impedance peak at resonance, that's one of its advertised advantages and you can see it in the smoothness of the resulting impedance curves.
But if the ferrofluid is either lost for some reason or dries out over time, the driver's impedance peak at resonance will increase in magnitude considerably. This is a problem because the crossover will have been designed based on the original smoother impedance curve. As a result, the crossover may well fail to adequately attenuate the driver's output in the region of that impedance peak, degrading the frequency response and/or power handling (which already took a hit when we lost the heat-transfer benefit of the ferrofluid).
Of course the ferrofluid is supposed to stay in the magnetic gap, but in my opinion it is possible for the particle motion within the ferrofluid to occasionally give a particle enough energy to escape the magnetic field. And if that happens once, it can happen again, and again. Think of it being like a "rogue wave" in the ocean, where the prevailing weather conditions don't add up to a 70-foot wave, but sometimes waves just happen to combine in-phase and it happens.
And of course the ferrofluid is not supposed to dry out, but what happens if it gets very hot? Does that change its physical properties enough that it is now more prone to drying out and/or being ejected from the magnetic gap?
I do a fair amount of work in the prosound world, and am unaware of any prosound drivers that use ferrofluid. In prosound heat management is a big issue, so you'd think it would be used quite a bit, unless it had some downside.
I don't use drivers with ferrofluid for these reasons.