Hi all.
Found this thread while looking the solution for the exact same problem.
My E10 is the old one too (with no Olympus written on it), and the noises are heard on similar situations (while playing/pausing the music player or an online streaming, I can hear it after the system plays short warning sounds too, like the warning messages from Windows).
But mine have one more problem: It only works after I disconnect the cable and connect it again every time I start my computer. Before that, no sound are detected by the system, it's like there's no playback devices installed.
This second problem can be solved by disabling the onboard sound card on BIOS and removing all traces of it's drivers from Windows, but if I do that I have no mic-in, and it sucks.
Some time ago I was using GNU/Linux to do some work, and if I recall, The E10 worked perfectly there, with no issues at all.
So I was thinking this old batches has some communication issue with it's own drivers on Windows, not a hardware problem.
I'll try some more things anyway, like making a better grounding for my computer and messing a little more with drivers and stuff on Windows, but I have a feeling that this problem can't be solved.