For far too long I have suffered trying to run a headless Mac Mini under Mavericks with absolutely p*** poor latency and slow performance. It was nearly unusable. I finally stumbled on on this solution et voila....problem solved. This is ridiculous to have to implement this kludge -- Apple should fix this problem immediately -- but since they haven't, this fix works. If you are having the same problem I was, you may benefit from the following:
http://www.tristanbettany.com/2013/06/17/slow-screen-sharing-in-osx-vnc-performance-issues/ Cheers.....and happy listening (unimpeded by a sluggish Mac Mini),
-dB