Well, if there was smoke being emitted you obviously had more serious problems.

This fuse situation is obviously a trade-off. Some folks feel the fuse characteristics are imparting negative audible consequences in normal operation and are comfortable removing it. Much depends upon the power amplifier used and how well it protects both the speaker and itself if an internal malfunction occurs. Transients generated upstream of the power amp might make their way to the transducers also.
As long as a user understands the trade-offs of removing the fuse then I say go ahead and do it. But let's not kid ourselves and assume the fuse is a 100% foolproof protection device that will safeguard the transducer under any adverse conditions. That's simply not the case. Neither can we say the fuse doesn't affect the audible performance of the speakers.
In this particular case, it's not clear if the OP understands those trade-offs, but they're his speakers and he can do whatever he wants with them.
Dave.