Until there is actual evidence (as in humans, not rats) showing that the amount of hops used for brewing IPA's results in a beer with a significant level of phytoestrogens, making it's consumption capable of altering natural endocrine function, I'm not at all worried about drinking IPA's. Many of the incidences that revealed the activity of phytoestrogens in Hops were involving humans who were around the herb all day (field workers), or who consumed potent preparations for medicinal/therapeutic use. I admit, I used to be scared of consuming any soy products for this same reason, but research has revealed there are no effects on male endocrine systems with normal consumption. Though, I still don't care for soy that much. There definitely is good reason to remain aware of phytoestrogens, as there have been incidences where they can have dramatic effects on human biology.
http://www.nejm.org/doi/full/10.1056/NEJMoa064725
That's a scary example.
Considering many molecules in plants which are classified as phytoestrogens are thought to have very beneficial effects on human health, (resveratrol, those in soy), I'm not going to worry about the IPA's I drink.
As for the xenoestrogen theory, I have to admit I don't have much knowledge regarding xenobiotics found in humans, though I am sure there must be a good number.
Here's a decent research paper done on Hops and beer,
http://jcem.endojournals.org/cgi/reprint/84/6/2249