If it's slipping out I'm thinking the contact is already not that good.
Yes, and I think Danny acknowledged that in an earlier post.
If these plugs are loose to the point of coming apart, any additional devices designed simply to hold them in ... springs, rubber bands and even crimping ... are missing the point because they do not improve the contact area to a high level [the essential here], they only hold the connectors together, with diminished contact area.
You could possibly use something like silver conductive paste here but really you've got two options 1) get hold of some larger diam. male pins [or smaller sockets] that will give you a good contact surface area or 2) do away with connectors and connect your external speaker wires directly to the circuit boards in the speaker box.