Don't despair, Ben, whatever you have, there is some way to make it better. Let us help!
If I'm understanding right, the edges that will butt against faces are not completely square? Your router can be used like a jointer to square them up. Sharp cutters and blades are a real help, too.
I've built many a cabinet with skillsaw and router. While time consuming, it doesn't necessarily mean imprecision.