Well I just finished building an amp, thanks to this great guide: http://www.tangentsoft.net/audio/cmoy-tutorial/ . Was simple enough to put everything together but now that I have it all finished when I adjust the volume with a Panny 10K pot I get some weird distortion (only with Audigy 2 source). If it is set to the lowest setting and I gradually increase the volume with the pot I get a weird "brushing" type of noise. No matter where the volume is set if I adjust it either way I get this brushing noise. It isn't really loud, but definately audible. I can barely pick it up if music is playing. No brushing at all if I don't have a source hooked up. I am about to open up my case and do some testing but I wanted to see if this was a common problem with any fixes or more isolated.
By the way I think I am now addicted, can't wait to start my next amp. Also thinking of picking up another pair of Senn's - using HD555's right now.
Sounds like that your Audigy 2 has an output DC offset problem. Use your DMM in DC mV setting and measure the output of the sound card and see what you get. More than a few millivolts will be enough to cause the volume control to sound scratchy.
The workaround is to install an additional input coupling capacitor per channel, between the input jack of your amp and the volume pot. Or, use a better sound card.