What format does Apple TV Stream out?
Standard PCM (or is it LPCM, whatever). You can set it to downmix 5.1 streams to stereo.
Can it be upgraded to higher quality?
You'd have to define "higher quality" here. However, I'm
not sure the ATV is bit perfect. It may resample to 48kHz. The older AirPort Express router offers a digital audio output and was confirmed by Stereophile to be bit perfect.
I thought he would run the HDMI to the TV and the Optical digital out to a DAC with volume/remote and outputs to an amp and sub.
Or combine the functions after your first "and" into one box. I think both HDMI and optical inputs are active at once, but am not sure of it. I've never used both.
What low end gear would you recommend?
As far as low-end gear, at least for background listening
this dirt cheap HDMI/stereo audio + component video DAC works very well. (I've not tried critical listening.)
I use two AppleTV's at home: a newer ATV3 in the main multichannel system, hooked up via HDMI, and the ATV2 it replaced in my bedroom, hooked up to the aforementioned DAC over HDMI, with audio signals going to an older integrated amp (Meridian 551) and the component video going to an old computer monitor, in case we ever want to watch Netflix or whatever in bed (haven't to date). Both work flawlessly, and the one in the main system I'd be willing to put up against any digital audio transport (that can play the same media!) at any price.