I fixed it or rather it fixed itself.
I decided to (once again) see if I couldn't get the tracking weight spot on and I loosened the screw which holds on the counterweight.
The arm went, "Sproing!!!", it flipped up in the air and twisted itself way the hell to the right. It was a WTF moment for sure.
Once I got the infernal contraption back under control I tightened up the counterweight's set screw and there is now proper tension on the fishing line and the azimuth stays true.
I have no idea what the problem was but it came that way from the factory. I bought it brand-y new, unused in the box a few years back.