The phytoestrogens in soy definitely appear to have some kind of endocrine effect but there is no consensus to support the idea that they are harmful in any real way at normal consumption levels. It's especially evident that soy does not have the same kind of harmful effects that the seriously bad stuff we actually have to worry about - the xenoestrogens we consume from pesticides and other chemicals.
A lot of what we consume actually has an endocrine effect. Milk, for example, seems to have an effect on testosterone levels!
I think the tendency is for people to think of testosterone as the 'male hormone' and estrogen as the 'female hormone' and we want to avoid one or another and that's really a simplistic understanding of how hormones actually work, since men and women have levels of both hormones in the body that regulate proper function of various processes. The body actually does all sorts of neat things on its own, via conversion of testosterone to estrogen and DHT, so by increasing or decreasing the activity of one hormone you tend to effect the activity of the others in sometimes unpredictable ways.
For instance, administration of testosterone in men seem to reduce levels of visceral (belly) fat levels, but increase visceral fat in women.
Sometimes you hear that bald men have more testosterone. From what I understand, DHT is a derivative of testosterone produced by the body that causes body hair growth. Males produce more DHT from puberty onwards and DHT seems to be responsible for a lot of male secondary sexual characteristics. But DHT it also is what seems to cause male pattern baldness! The drugs that they prescribe for baldness by blocking the conversion of testosterone to DHT actually increase levels of free testosterone (since it isn't being converted into DHT) but the body might respond by converting the excess testosterone into estrogen. (Which is why some people suggest these baldness drugs cause things like depression or breast growth).
Alcohol seems to have the effect of blocking DHT activity but not testosterone which is why it seems to reduce the ability to 'perform' despite not supressing the actual sex drive itself. (Tell me when was the last time anyone reported that soy milk had a similar effect).
I find this stuff endlessly fascinating because the endocrine system is the biological basis of binary sexual characteristics, but you can see many different effects in people when the endocrine system has been modified - transgender folk, bodybuilders, heavy drinkers, people on AIDS retrovirals.
You really don't see those kinds of effects in people who drink soy milk in normal levels. If anything the general thrust of evidence is that soy is good for you.