My experience has always been, dollar for dollar, coax sounds better than optical every time. It used to be that you could spend _way_ more for an optical cable than coax. I found that it always took a more expensive optical cable to sound as good as the piece of coax I was testing.
As many have pointed out here, YMMV. That said, I was amazed the first time I got to compare glass vs. plastic optical cables. Echoing Wayner's take, I thought it's all digital, right? How is a cable going to impact the 0/1 transfer? Well, it does.
As a couple of people have pointed out, this would indicate implementation is indeed a factor. I've found going w/ coax has been much more forgiving (lacking a better term) when I start swapping components out. Meaning, the sound via coax was much more consistent, where the optical cable would give me anything from "This sounds fantastic!" to "Holy cr@p! This sounds horrible." For me, it is pretty easy to hear the difference between coax and optical, regardless of how close the two are to one another in performance.
Disclosure: My perspective may be distorted. I've always had easy access to broadcast grade, truly balance 75 Ohm coax. So, I may not have given optical cables a fair enough shake.