I used to hate them. I went through a bunch of cheaper ones and could never get a reliable fit, and when I did, sound quality wasn't anything compared to good earbuds or cans.
For some reason, in spite of never finding a cheap IEM I liked, I took my birthday money from last year and blew it on SF5s... The OP's 5-10 minutes is nothing, it took me six months before I finally reached a point I could wear them comfortably with something other than Comply tips.
At that point I was a believer. I'm a detail person. Give me a semi balanced sound spectrum with good detail and I'm happy, and those SFs gave me more detail than anything I'd ever heard had. Now I've got triple.fis and they're hands down the best sounding headphones of any kind I've ever heard, and they're comfortable enough that I wear them anytime I don't have to be able hear things going around me.
Comfort/fit issues aside, a lot of it is going to come down to what your personal priorities in sound signature are - being a detail person and not ashamed to admit, I'll gladly trade the open airiness of big cans and the better comfort of earbuds for this immersive ocean of musical detail.