As is so often the case, the exact opposite may very likely be true according to the research. Researchers found that senior men with an average age of 60 without Vitamin E supplementation had testosterone of 573. When they gave them supplemental alpha tocopherol, the testosterone dropped to 539 ng/dl.  This is about a 6% reduction in testosterone. The dosage was fairly low, 50 mg/day, compared to the normal dosage in supplementation, but researchers have long noted Vitamin E’s testosterone lowering powers.
Although an animal study on rams found no statistically significant change with administration of alpha tocopherol,  within just a few years researchers reverified Vitamin E’s anti-testosterone powers.  In fact, that same study even identified exactly how Vitamin E supplementation lowers testosterone and concluded: “Therefore, growth modulation by the antioxidant moiety of vitamin E in androgen-sensitive prostate carcinoma cells is due, at least in part, to its potent antiandrogenic activity.”
Vitamin E’s anti-testosterone powers are so strong that the authors actually recommend its consideration for further study to prevent prostate cancer. Most guys are completely unaware that their testosterone could be affected by a vitamin such as this, especially negatively.
For those who have prostate cancer, perhaps this is something to consider with your doctor. (I mention alternative ways to Prevent Prostate Cancer and General Cancer Prevention.) And there are other possible reasons to take Vitamin E, including its potential blood thinning and brain-protecting powers. However, “eyes wide open” I always say, and I don’t believe most guys want to take anything that will lower their testosterone. Most of us are looking for the opposite!
I should mention, though, that the picture is not quite as clear as we might like. In the above study, testosterone was not the only hormone that decreased with Vitamin E (alpha tocopherol): androstenedione, estrone and SHBG all did as well. Now this brings up an interesting point: the decrease in SHBG will tend to offset the loss in testosterone by raising the percentage of free testosterone that is available to tissues. However, no measures of free testosterone were presented in the study so I cannot comment.
To further muddy the waters, there was also a study that showed that testosterone levels were actually increased with megadoses (438 mg/day) of Vitamin E.  However, this study had no control. Plus,. I would urge caution at megadosing alpha-tocopherol, because yet another study showed a decrease in thyroid hormones with megadosing.  One study even showed a slight risk in lung cancer for those taking Vitamin E. 
As always your safest approach appears to be to stay within normal physiological ranges, i.e. get your tocopherols from a standard diet. This should optimize your hormone levels and protect your Leydig cells. (Testosterone production generates considerable free radicals and Vitamin E can protect your cells from damage.)
NOTE: All the studies on Vitamin E use alpha tocopherol. Both our diet and our bodies actually contain several different tocopherols, including the important gamma tocopherol, and you will find the best supplements include a natural blend of “mixed tocopherols”.
1) The Prostate, Jan 2001, 46(1):33â€“38, “Effects of long-term ?-tocopherol supplementation on serum hormones in older men”
2) Theriogenology, Feb 2000, 53(3):751-760, “Effect of ?-tocopherol on plasma testosterone and plasminogen activator activity or inhibition in ram spermatozoa”
3) Mol Cancer Ther, August 2003 2:797, “Androgen Antagonist Activity by the Antioxidant Moiety of Vitamin E, 2,2,5,7,8-Pentamethyl-6-chromanol in Human Prostate Carcinoma Cells”
4) Endocrinol Jpn, 1982, 29:287-292, “Effect of vitamin E on function of pituitary-gonadal axis in male rats and human studies”
5) Am J Clin Nutr, 1978, 31:831-837, “Study on the effect of mega-Vitamin E supplement in man”