Nope, not crazy. Actually, very good on you for taking your time to make sure everything comes out "right".
The pebble/flat finish will probably make the gloss color sides "pop" more (the middle of the speaker won't be fighting for attention itself. Tough call with the textured finish already in place. Any chance of shooting some flat/semi gloss over the existing texture to get a feel for how it might come out without a lot of extra time invested?
In any case, no matter what you choose, if you aren't 100% happy with the black as-is, definitely go for something different while you still have the opportunity. It will take a little longer to get the speakers put together, but compare that time to the time you will spend looking at the otherwise finished product knowing that you are unhappy with it and could have done something different about it.
Just my $.02.
(and because I haven't posted in this thread yet) the orange is just sick! I had a car almost the same color a few years ago. . . the color really grew on me, and I still wish I could have got the one I'm currently running in the same color, but it was discontinued for that model year. . .