It's mostly a hardware issue. Mango is running on a single core, single thread CPU and it puts pretty much all of its effort into expanding music files and feeding the DAC. There's a bit of latency as it shifts to handling user input. The touch screen seems more responsive because that latency happens while you wait for the screen to activate.
I have no real latency on the screen (measured in real time not from my interpretation of it). I can click the button 10 times before it actually skips to the next track, so it's not just latency but actual response.