As one who goes out of his way to collect gear that accurately reproduces the full frequency spectrum, from lab testing I have to say:
But isn't it part of the experience?
I'm a working musician. I experience things outside of the 20-20 range every day. Part of the appeal of Pipe Organs is their ability to represent different instruments/voices in a clear sound field along with everything else. When Bach designed organs, he refer ed to some having thunderous bass. We know today, that those bass pipes went down below 10Hz all the time. It's part of the experience of the Pipe Organ. Some of the Mixture stops have pipes that scream above 20kHz, and people tune them by tuning the harmonics they produce that we can hear... talk about voodoo magic...
The point is, humans may not be able to register something above 20,000 cycles per second, but if we walked around with a filter on our ears, I guarantee people would state something was missing. I say all that to say this: It DOES matter. Watch an RTA (that can read up above 40,000Hz), and see just what happens up there. Do blind tests. I bet you will be able to tell a difference.
An argument for it Does NOT matter:
The hard part is going to be finding equipment that doesn't cut frequencies, or just roll off anything other than 20-20... which I hate.
Then, think about your source material. CD's go to 22kHz or so. How is a 24/96 recording going to sound better than a CD, if it was made from the CD master. If it was made from an analogue master, perhaps. If it was made from a digital recording? It's likely already 24/96 and the publisher just put it in a format you can hear.
And then, was the source recorded on mics that can actually hear up past 40kHz? You got to have very sensitive recording equipment to do that.
In the end, pay attention to what your SOURCE is. If you are lissening to CD's, MP3's, FLAC... better check to see that there is actually sound info in those registers before you buy a headset that can do 1000000000Hz.