You're most definitely not alone in that view ... I read somewhere of Debussy saying he would rather see the "sun rise" than listen to the Pastoral ...
I dare say Beethoven would probably agree given how much he is said to have loved nature. I think the symphony stands out stylistically among the others in that it is love song. No holds barred on sentimentality and devotion, which turns a lot of people off. I will admit that I'm a sucker for that soppiness
I'm mostly drawn to the beautiful melodic phrases, unbridled joy and contentment. In the context LvB's knowledge of his failing health at the time it's an insight to the man beyond his caricature of a grumpy old sod. It is definitely sweet and mellow, but something more profound beyond that too. My favourite symphony of his.