Interesting to see the different perspectives on this. Maybe it's because I'm a writer, but I just can't understand how lyrics could have no effect for some people. We're all different though, so whatever. I personally think it's all closely connected (for music that has lyrics): the music, the lyrics, and the vocal style used to deliver the lyrics. Sometimes there are lyrics that aren't at all impressive on the page, but are given great strength by the music and vocals. The opposite can also be true, where there may be some impressive poetry to lyrics when read, but they fall flat on the delivery, or clash with the music.
If I'm listening to music with lyrics, bad lyrics can just kill it for me, just as much as boring musical elements, or vocals that bore or irritate me. R.E.M., for example, delivers most musical elements I'd normally like, but their insipid lyrics pretty much kill them for me.