Sorry, but the lawn mower outside is not anywhere near the approx. 75-80dB I listen at. It still impacts sound quality. People all the time complain about computer fans with open headphones, or look for silent fans. Case fans get at their loudest probably 40dB, unless they're spinning really damn fast. That's 1/16th of 80dB. Yet it's still noticeable and it still is a detriment to sound quality. Remember that even if you listen at 80dB max volume, most of the sounds in the music are much quieter than that. Unless your songs are all compressed dynamically, there will be nuances in the music as quiet as your surroundings allow. A noise at 40dB will blur out all sounds in the music below 40dB. With headphones that reduce that by as little as 10dB, it now only blurs out sounds 30dB and under.
Music often sounds better late at night. There are several hypotheses as to why. One of them is because the ambient volume around you is quieter. Less traffic, fewer annoying birds, less noise in other rooms.