In principle, the fewer connections the better. In actual practice many would say that during a blind A/B comparison few could tell the difference. It also might matter how resolving your system was as to whether you noticed a difference.
The only headphones that I leave that way are my IEMs which are used 95% of the time as airplane/travel headphone. I use the 3.5 mm to 1/4" connector that came with my AKGs on the few occasions I listen to them amped.