Circuit design. Headphones (well except for the K1000's) require different parameters to sound their best than an amp designed to run a low impedance speaker can provide. A good headamp will also be able to run a wide variety of low and high impedance cans. Some use a switch, or a series of dip switches, some use dual outputs, etc. Popular cans on the market now run the gamut of impedances, from the Denon's at 25ohms, the Grado's at 32ohms to the older Beyer's at 600ohms.
I'll try and find some articles to link you to that explain it better than I can.
Peace,
Lee
Ok call me stoopid but what makes a headphone amp a headphone amp? Is it a mechanical connection thing or a circuit or both.