Here are some of the things to consider in designing a headphone amp.
1. Low amp output impedance, same reasons as to why this is desirable for driving speakers, remain with headphones. Near zero as possible is what some aim for.
2. Able to drive both high and low impedance cans adequately. While there are a a handful of headphones out there that are in the 300-400 Ohm range, the majority are now less than 70 Ohms. Driving each type presents different challenges, which explains perhaps why some headphone amps now come with a hi/lo switch which toggles the output buffer.
3. Wattage and current. While your typical headphone amp is fine at .25W output, some of the best headphones (such as my AKG K340s) are very power hungry. 1 - 2W is ideal. However, this would be far too much for low impedance headphones, which typically need high current but not high voltage to sound best. Some form of gain control is needed.
In retrospect, adding a headphone option to an RM-5 may turn out to be costly if addressed via adding a transformer (such as a Sowther). Once labor is factored in, probably cheaper buying a headphone amp.