To suggest that the HM-801 sounds worse than a mass market DAP because of frequency response curves or crosstalk is ludicrous, that's like judging $10K monoblocks based on THD measurements. This one is 0.005% better, therefore it sounds better cause der measurement said so! I maintain that anyone that says the 801 sounds worse than a clip has not listened to it with headphones that cost more than $200.
I've done plenty of comparisons with my 801 and my Cowon S9 using the same FLAC files, with the Cowon both set for no EQ/enhancements for apples to apples, and set with my preferred settings to give it the best possible advantage. Here's what I've found. Sony MDR-7509: no real difference. AKG K601: no real difference. Denon NC-800: very slight improvement with the 801. Audio-Technica ES10: smoother, more refined sound from the 801 with less grain, particularly in the treble region. Denon D2000: considerable improvement with the 801, better bass control, less recessed mids, less treble harshness. Beyer DT880-32: night/day in favor of the 801. Honestly if you try the 880 on an HM-801 and a Clip or Cowon and can't hear a difference, you should give this hobby up and just go listen to some Apple earbuds.
I should make it clear that under no circumstances with any headphone did I prefer the sound of the Cowon to the 801. With my cheapest headphones there just wasn't I difference I could hear.
To actually answer the OPs question rather than just use this thread as an excuse to troll, no, you don't need 24/96 to get the most out of the Hifiman. I would say its performance is somewhere around the level of a $300 or $400 amp and equivalent DAC, and it's impressive that they were able to stuff that level of performance in a battery powered box, but it's not a competitive with an $800 amp and $800 DAC, which is where you might want to start thinking about high-res.