One of his most popular albums is an early one, Six and Twelve String Guitar, and it's very good. In fact, I don't think I've ever heard a clunker album from Kottke so I wouldn't be afraid to reccommend grabbing the first one you see to see if you like him.
His vocal style is an acquired taste. He's no Caruso, but then neither is Tom Waits.
I haven't heard any of his live albums, but if they are anything like his shows (especially if the include the bizarre stories and rants of his) then I would grab those first. If you ever get a chance to see his show, do so without fail.