The most useful tool is a table saw. A table saw lets you cut pieces to reliable sizes that are easily glued together. You can fake this with a circular saw with a sawboard, a radial arm saw, a router with a homemade jig, or by paying the guys at Home Depot to do cuts for you, but none of these is quite as accurate or makes life as easy as a table saw.
You also need a good selection of clamps, for holding the pieces together when gluing. Clamps are relatively inexpensive. There are some assembly techniques that let you get away without using clamps, e.g. screws, certain types of complicated joints, etc., but having a set of clamps is almost essential in practice. (Also, screws and brads can migrate outwards and mar a paint job or make veneering difficult.)
A router is also very useful, since it lets you cut holes, flush mount drivers, and do roundovers. There are ways of faking each of these things without a router, but a router is very versatile. To cut holes, you need some kind of jig for your router. You can make a jig at home out of a piece of scrap wood or acrylic, or you can buy a Jasper jig. I just use a homemade jig myself. You can get by with a regular router (I do); you don't necessarily need a plunge router. The advantages of a plunge router are easier flush-mounting of drivers and usually a larger shank, which lets you do wider roundovers. It's hard to find a conventional, non-plunge router that accepts 1/2 inch shank router bits, and it's impossible to find 3/4 inch and larger roundover bits for smaller shanks. (To save the expense of upgrading to a plunge router and getting new router bits, I've always just done large roundovers using pre-cut mouldings I buy at the hardware store.)
You will need a drill to drill screw pilot holes for driver mounting, as well as for drilling small holes (with a circle cutter) like port holes in bookshelf speakers.
Apart from those tools, you really don't need anything else to build speaker cabinets.
