I think it's dependent on setup. How your computer is built, etc. There could be batch issues though too, some not having proper grounding. Some people report issues with USB (like me) and others report no noise with USB. I think everybody agrees that if you use coaxial or optical input, there's no noise at all, aside from a high gain hiss (but that was supposed to be fixed many many months ago).
Oh, and you don't HAVE to swap opamps but it's super easy and cheap to do so and it makes the unit sound so much better.