The vast majority of in integrated, in receiver, or in any electronics audio device is only a small 'integrated' board. Usually as cheap as possible, even in better brands.
A waste of time to try to 'upgrade' it IMO.
Yes get a small headphone amp.
Cannot say which ones are great at your price point, as i use a Rudistor RPX33mk2 ($1,200 used)
for a little more than your price point a used tubed DAC/headphone like the TDAC I have (with 'tubyness" knob!! ) I got for $550. used last year. With that you would have both a new tubed DAC, AND a good headphone amp.