You certainly can color the sound a lot of ways with the box or baffle as many people have tried and some even sell speakers with variable box tuning. But, as Danny said, this is not really pure sound. Now, if the driver has a peak in the response and it sounds that way, then putting it in a box that sonically removes the peak results in a more linear sound. Personally, I would go for the driver with the least coloration and put it on a dead damped baffle that also adds no coloration. Since this game is very subjective (more than computers, video, cameras, cars, etc.). than whats "sounds good" to one person, may not be preferrable to another. What I want from my stereo is it to disappear. If the recording is great, and if all the playback equipment is great then I get to hear the original mic feed. This is what I want. I want to get as close to the original as possible. However, there was an article in an old audiophile mag (I will remember the name later and put it here) where they had a guy stand between the speakers and sing. They also had a CD of him singing the same song and they played it through that speaker system that included a solid state amp. The sound from the CD was very close to him live. Then they played the CD through a single ended triode amp and everyone crapped their pants. It was so "LIVE and PALPABLE". So live and palpable that it sounded better than the guy standing there singing. I used to have original Quads and I had a tube amp and would play Joni Mitchell and Joan Baez and it would make me cry. Then I was at a friends house down the street (who also owned Quads) for a party and he had a piano there and a woman was playing and singing. I was standing maybe 10 feet from her. I closed my eyes and listened to her and I thought to myself "If I heard that voice on my stereo I would think she was veiled". So, a tube mic a few inches from the recording artist, then played through a tube amp and then through over imaging Quads played louder than real.......yep....sure sounded great.....was it real...who cares.
What I am suggesting you do is to experiment. You really have no idea what your baffles are doing to the sound until you make them better/different. You cannot just suppose that your baffle is fine because you put your hand on it and it seems to not be moving much. Same with the magnets on the woofs and mids. This is an area where very very few have gone. Some want to just do what is simple and cheap. Others want to do a little more....then a little more.....and then they die......and some want to find out how good their speakers can be......so they leave no stone unturned. Which are you? If making a rigid baffle, bracing it and also damping and bracing the magnets were hard to do or expensive, then I can see where you might not want to go down that road, or cannot go there. But we are talking a few bucks here. There are speakers that cost multi, multi thousands of dollars that are not done very well. We can do very well for a few bucks.
If someone tells me some intelligent information that I can use to make my system better and I can afford it, then I do it. If they were wrong, I sure did not lose much.
Mostly what people fear most is to lose face. They have an opinion and when they put it out for all to see, then they keep defending it no matter what. Righteousness and fixed attitudes are the killer of joy, creativity and truth. The killer of love. "Fear is the lock, and laughter, the key to your heart".