Most music is encoded as (linear)
pulse-code modulation, or formats which decode into LPCM like mp3. That is, the information for each channel is represented by amplitude values that change over time. The resolution of the amplitude values is 24 bits so 2^24 possible values can be represented, and there are 96000 values spaced evenly per second. In other words, you get a value per 1 / 96000 Hz = 10.42 microseconds. This is as opposed to 16-bit resolution at a rate of one per 1 / 44100 Hz = 22.68 microseconds.
With a higher resolution per sample, the quantization error—difference between the actual value and the rounded-off version that you get—is decreased. If properly handled, the error can be handled and processed in such a way that it can be considered just a very small amount of noise, even for 16-bit samples.
With a higher sampling rate, higher frequencies can be represented, as per the
sampling theorem. Due to limits of practical implementations and other issues, you can't quite get up to half of the frequency of the sampling rate for playback.
In practical terms, for further processing in the studio, you want 24-bit samples or higher so errors don't keep accumulating in a significant way. For home playback, it takes a very extreme case where 24 bits has an advantage over 16 bits, mostly involving cranking up the volume to highly unrealistic levels and trying to listen for the noise floor while pretty much nothing is going on. Keep in mind that the microphones and recording process are not noiseless, so in 16-bit tracks, there is usually going to be noise above any of these issues anyway. Getting a 96 kHz sampling rate may mean that there was less sample-rate conversion going on between the master and the version you are playing back, but that's not really an issue. I guess the main thing is losing out on any potential ultrasonic frequencies that you can't hear but your dog can. Most mics aren't designed to really capture too much outside of the range that humans hear, so there might not be a lot outside that range anyway. Take that as you will.
Music that tends to be released as 24 / 96 tends to be mastered better, to sound better on high-fidelity playback systems. That's a significant advantage, not really anything to do with the format itself (though many claim otherwise, for various reasons).