There are multiple issues here.
One is that the inner grooves' linear speed is least, so the high frequencies have to be "squished" (the wiggles are shorter, or compressed) for the same frequency. That's inherent, and why some superdiscs were cut at 45 rpm, and/or have less music and end well out from the spindle.
Another is that a pivoted arm, as mentioned above, is exactly tangent to the groove in only two places. Different protractors may place the two at different spots; arguments have raged over this one, but it seems to make sense that the inner tangent point should be a bit closer to the center than 2/3 of the way in. Arm length changes the steepness of the arc, and therefore the amount of the error away from the tangent points. Again, as pointed out above, if the error sounds severe, your cartridge may be misaligned. Worth checking to be sure your cantilever is not bent, or if the cartridge suspension is tired, that the cantilever is not off center. And for that matter, making sure the cartridge is not canted to one side.
And again, as mentioned above, skating forces vary across the record, and with level; so adjusting by ear is a good way to go.
If you are tracking a bit lightly the problems above can be made worse. Many cartridges are a tad optimistic about tracking, and do best at the top of their recommended tracking force range.
I recently went from an Eminent Technology II arm to a stock Scout and do not find the inner grooves objectionable. Yes, a difference, but less than other differences.
edit: saw a post below that reminded me; is your turntable perfectly level?