If you're ripping CDs - 16/44.1 is exactly what should be going on (since that's whats on the CD). DVDs can get into the 24/48 realm, and DVD-Audio (where the entire DVD is used for just audio) into the 24/96 realm (DVD-Audio playback can get tricky on computers though). Blu-ray can go somewhat higher, it's big advantage is lossless encoding. We'll ignore that SACD exists because to my knowledge, it isn't available for PCs (and that iBasso can't do DSD anyways).
So, as far as what you can and can't tell apart - 24-bit DACs do and don't exist; devices can process that level of fidelity but in terms of reproducing it "accurately," there isn't anything with minute enough voltage swings (144 dB). It's mostly marketing unless you're dealing with DVD or DVD-A though (and then, it's only because you want to be able to decode the audio, not because it's "better").
Sampling rates are affected by the Nyquist limit, I'll just link the wiki:
http://en.wikipedia.org/wiki/Nyquist_limit
So basically, 96khz allows us to record higher frequencies - the question then becomes: how's your hearing in the 30-40khz range? Can your speakers, amplifier, etc do that?
I am somewhat oversimplifying.
Summary: higher numbers translate into higher theoretical accuracy, because more data points are generated for conversion to analog. Whether or not this accuracy matters is up for debate (but generally I would say that I don't believe it does).
If we're talking about SRC (sample-rate conversion), I'd say: don't. There's no good reason to convert something from 16/44.1 to 24/48 or similar - you can't add data that doesn't exist, and you'll likely introduce more distortion artifacts as the data you already have is stretched and modified to meet the new SR. The only reason you'd want to do SRC (aside from cases where it's forced/not optional, which happens for DSP processing in some cases) is if you're working with a multimedia project and need everything running "the same" for the final output (say if you're making a cartoon and drawing audio from multiple sources).