When does dc offset become a problem? Well, try putting Ohm's law to work, folks, specifically, P(mW)=(V^2/R)*1000
Measure the DC resistance of your favorite headphones.
Measure the offset voltage from left and right outputs to ground.
Plug those numbers into the equation above to get the amount of power in milliwatts the offset will put through your headphones.
For example, 100mV of offset into 32 ohm headphones equals 0.31mW - too much! This is only 5dB below 1mW, the power level at which the nominal sensitity for headphones is measured.
100mV into 600 ohm headphones is 0.0167mW, which seems insignificant but it's still only ~18dB below 1mW.
added the text below:
Because power is proportional to the square of the applied voltage, even a modest reduction in offset voltage pays off nicely. Reducing the exorbitant offset voltage I used in the above examples down to a much more reasonable, but still not so good, 20mV, would give these results:
20mV into 32R = 0.0125mW/-19dB
20mV into 600R = 0.000667mW/-32dB
But as far as what voltage will start to heat the voice coils up appreciably, well, that's a different story. I doubt that even 100mV of offset would result in noticeable heating, but it will certainly result in distortion from the reduction in driver excursion!