The only iffy part on that is that as a physical switch, you'd want to make sure the switching contacts are rated high enough to carry the speaker load. I've got some heavy duty switchcraft ones which could probably do it, but the body extends sideways and wouldn't fit.
I think you'd also have to do some creative wrangling to get the resistors in line with a standard switched jack, unless you're comfortable running full power into the headphones. Personally, I have a Hifiman HE-6 which is literally one of the hardest headphones in the world to drive, so I in fact run mine directly off the speaker terminals on the back.