Very heavy recessed bass drivers can be removed safely by removing all but the bottom (6 o'clock) screw. The weight of the magnet holds the driver in place. Then hold the bottom of the driver against the baffle while unscrewing the bottom screw. When you're ready let the bottom edge slide out slowly and grab it with two hands.
This thread caught my eye because I built a pair of pine bookshelf boxes last month. The driver had a slight friction fit in the rabbet, I should have backed off then but I wanted to "hear the music." After a month the wood swelled and locked the drivers in place. I had to cut off the back wall of the box and pound them out with a rubber mallet.

Thankfully I had to shorten the box anyway due to design error, but from now on I will always route the holes 1/16" too big. Hopefully you won't have to resort to that Sonny!

I use a gasket material from Homedepot which is 3/8" wide soft silicone rubber profile with adhesive on one side, and a empty air balloon on the other side. Works good.