For the most part I agree with you. You have only to note my fascination with Alma Mahler's love life to understand this, and I am really fascinated by the whole scenario of that Visconti anecdote. But I also believe that understanding the context (historical, cultural) and knowing the psyche (sexual orientation as well) of a composer can help us to appreciate the work only on a purely intellectual level by giving it more associative meaning. What it cannot do is to detract from or add to the actual quality of the work. Great and good men have created very bad art and very bad men have created very great art. That is, ofcourse, one of the great paradoxes of human existance and endeavor. That is why I always have believed that the art does in fact have a separate existance apart from the creator. If I could not make this separation, I could never for example, listen to the voice of Elisabeth Schwartzkopf who was such an enthusiastic member of the nazi party, or marvel at the pyramids of Mexico, which were used for the most abominable rites of sacrifice.
Weak female that I am, Mahler's sexual preferences are to me almost as interesting as his mystical leanings.*
Mahler's mysticism, which existed despite what I perceive to have been his indifference to most forms of organized religion, was probably a lot more important as a contributing factor to his work than his sexuality. On the other hand, I think the question of Mann's sexuality as a factor in his work is more pertinent as Mann did admit to having sexual feelings for young men, incorporated those feelings into his work, and admitted to consciously choosing a hererosexual lifestyle by marrying and have a large family.
In the end, when I consider how very rich the milieu of late 19th century, early 20th century Vienna and Berlin was, the wanton manner in which the assets of that culture were wasted on the first world war becomes so much more important to me when I listen to the music of Mahler and his contemporaries than any speculation about his possible sexual orientation.
*Btw, the more I read, the more I am learning about Mahler's affairs with females during the course of his marriage to Alma -- who herself was promiscuous, the more I doubt the Visconti anecdote. Also, more interestingly, are you aware that when the problems in his marriage became unbearably painful to Mahler, he consulted with Sigmund Freud? Oh to have been a fly on the wall in the good doctor's office during those sessions!