"Listener" or "Listening Fatigue" is within the realm of Psychoacoustics. It can be caused by *any* form of distortion artifacts where the cause is in the equipment chain, speakers, or (more commonly), poor room acoustics. Indeed, listener fatigue can be demonstrated in meeting rooms, lecture halls (aka University lecture rooms), or even overly reverberant "lofts", or other spaces.
The simplified answer is found in the fact that we ponies have a very vast experience with respect to (1) how things should really sound; and, (2) listening to the pony voice. In case one, our listening mechanisms (brain) will work overtime to filter out distortions from our experience base and allow us to perceive a distorted sound as not being distorted. In case two, we are doing whatever we can to understand speech and dialog. In a poor audio reproduction environment, poorly designed meeting room, or even in a "too loud" restaurant those hearing mechanisms will do their best such that we can understand the dialog. It is these events which lead to listener fatigue (and students falling asleep in lectures ).
Comment on overly bright speakers. I will grant some speakers are horribly designed; however, in the vast majority of cases where a speaker is described as overly bright (room environment not withstanding), the fault does not lie with the speaker ... it is found in the inappropriate use of that speaker. Since high frequencies will roll off much faster in air, speaker designers will increase HF output of the speaker based upon what the designer (or marketing department) determines will be the average distance between the speaker and the listener. If that is a longer distance and listener is sitting closer to the speaker than the design distance, that speaker will be said to be overly bright. It is only overly bright within that specific environment and the real problem the speaker is being used within a context for which the speaker was not designed to accommodate.