Seems a little odd if you didn't stain or apply dye, might be fine sawdust. Mineral spirits won't hurt anything just don't get it real wet, keep it pretty dry on the cloth. I don't use veneer much so I don't have a great feel for how veneer reacts to how I would treat solid wood. I'm assuming mineral spirits won't interact negatively with the veneer adhesive (don't remember how your was attached).
If needed, you might be able to take a real dilute shellac (like a half-pound cut..3 parts ethanol + one part sealcoat) and wipe it on with a lint free cloth or blue shop towel. I'd think you could coat the side of a speaker quick enough to avoid over-laps. Often..finishing comes down to experiments to work out techniques. A light coat of shellac may seal things up well enough.
