Look if you have 44.1 khz sample rates you can record frequencies to near 22,050 hz. They brickwall filter between 20khz and 22.05 khz to prevent aliasing.
If you use 192 khz sample rates you can record frequencies up near 96 khz. Normally the 80 khz to 96 khz band will be brickwall filtered to prevent aliasing. And though humans can't hear 80 khz, hirez recordings potentially have that response. They don't stop at 20 khz. And that is all you get from higher sample rates is extended frequency response. You don't get better drawn waves at lower frequencies from interpolation. You get one thing from higher rates and that is more bandwidth.
The reason the extended response draws a better square wave is a squarewave is a sine wave and all of the odd harmonics. More bandwidth allows you to draw in more harmonics. The closer to infinite harmonics you get the more square it will become.
http://xiph.org/video/vid2.shtml
This is a super good video about how digital really works. 24 minutes, and well worth watching. Explains a ton of ideas in a simple easy to understand manner. They use very high quality analog signal generators and analyzers to show what digital can do and how it works. It is most informative.