To me, the primary benefit of a headphone amp is the ability to provide power amplification to a good line-level music signal, and in the process provide volume control as well as possibly a good EQ profile or few.
Line level signals are where its at: properly designed a line level signal coming out of a DAC or DAP represents the ideal best-case scenario in terms of audio performance from that piece of hardware. It will offer the best of everything: frequency response, noise, distortion, separation, etc. The device's built-in headphone amp may not be well designed (noisy, distorted, etc), not powerful enough (e.g. Teclast T51), or just a poor match for certain headphones. In this case, a dedicated headphone amp will be able to provide the kind of high quality amplification and headphone driving capabilities that enthusiasts look for.
A properly executed headphone amp will not only preserve the fidelity of the music signal, but also potentially enhance its enjoyment by providing high quality modifiers in terms of boost/cut EQs, crossfeeds, and etc.
Jack