Why not minimize input from the diet first?
Again, I have no problem with a guy playing it safe.
But, to answer your question, let me ask you a question:
"What bad side effects are there from eating phytoestrogens?"
I know of one potential one, but I doubt you could name any. (You can correct me if I'm wrong.)
So think about that: they have been study phytoestrogens for decades now and people have been consuming them for centuries, so why can't anyone name even one side effect for even a subpopulation of guys?
Now I don't consume soy, but let me play devil's advocate here:
A pro-soy male is going to tell you that soy has no effect on T and will boost nitric oxide and was consumed by the healthiest, most long-lived culture on the planet. So what's not to like?
I mean, seriously, the Okinawans had the highest recorded senior testosterone levels on planet earth. So isn't that an argument to consume the same levels of soy that they did?
Again, I am just playing devil advocate here to point out that the arguments for consuming soy are actually stronger than those against (assuming it is non-GMO'd and hexane-free).