Er, this is on the guy's source. Whatever the gain of his amp, is going to muliply that DC offset by.
The simple solution to this is to put either output coupling capacitors on your source or put input capacitors on your amp, thus nullifying DC offset. You can't really get away with amping a source with 12mV of offset, especially if you need to set gain fairly high. Even with a gain of 3 to drive 50ohm cans, that'd still bump offset up to 36mV and that could possibly damage your headphones.
Oops, didn't see the not using an amp part. I wouldn't use good cans on it anyhow. I'm picky about offset, I don't give up until I get any of my DIY amps to have 2mV per channel or lower. I'd amp that source just so I could kill the DC offset with input caps.