Research has shown that every one of these items can lead to very signficantly lowered testosterone (and libido). I call them the Dirty Dozen.
NOTE: You may also want to read my link on Common Chemicals That Lower Testosterone as well.
1) Overweight. Those extra pounds are strongly correlated with lower testosterone  and a Lot of Other Bad Stuff as well.
2) Metabolic Syndrome (insulin resistance) and diabetes are both equate to decreased testosterone levels. 
3) Chronic and Non-Moderate Alcohol Consumption. Animal studies clearly show heavy drinking for an extended period of time will lead to lower testosterone.  Always play it safe with Alcohol and limit yourself to one or two drinks per day.
4) Stress. We males can intuitively feel how hard stress is on our body and mind and the reason is, among other things, simple chemistry: stress increases cortisol and cortisol has been shown repeatedly to be associated with lowered testosterone levels. I cover this extensively in my link on Stress, Testosterone and Your Brain.
5) Lack of Sleep. This actually follows – and many guys don’t know this – as really a subcategory of #4. Lowering sleep quantiy or quality will leads to increased cortisol and a corresponding testosterone decrease and I cover that extensive in my link on the Nasty Effects of Lack of Sleep. Sleep can also lead to calcification, or hardening, of the arteries and blood sugar management issues as well.
6) Depression. Depression is also actually a subcategory of #4 and leads to increased cortisol and inevitably lowered testosterone levels. Depression also can lead to erectile dysfunction and literal brain damage as well. Read my link on Depression for more details.
7) Vitamin A Deficiency. Being deficient in Vitamin A has nasty results on both sperm and testosterone. In fact, animal studies show that in Vitamin A-deficient animals, lower testosterone is followed by atrophy of the testes.  Don’t forget that you don’t have to directly consume Vitamin A to get your what your body needs: your body can also build it from betacarotene, the orange phytochemical in yams and carrots.
Did you know you can inexpensively do your own testing for most hormones? The industry leader is Discounted Labs..
8) Vitamin D. Vitamin D has been correlated with higher total and free testosterone.  Vitamin D deficiency is incredibly common, as I cover in this link on Vitamin D, and can result from being indoors, skin color, the seasons and so on. The solution, of course, is to take Vitamin D as a supplement in order to avoid lower testosterone. (Read my link on Vitamin D for more information.)
9) Overtraining. If a little exercise is good, then a lot has to be better, right? Unfortunately, that doesn’t apply when it comes to our hormones. Overtraining lowers many of our most precious hormones, such as testosterone, and raises still others that ought not to be raised. You must build up slowly in your training endurance and strength and make sure you have sufficient rest. Read more in my link about Overtraining’s Powerful Testosterone Lowering Abilities.
10) Toxins. There’s a lot of what I call “Socially Acceptable Chemicals” that will give you lower testosterone. In fact, it’s nearly impossible for a modern urban dweller to escape at least some exposure to these nasty hormone killers, which include Pesticides, Excitotoxins and Marijuana. Read these links: no one but you can protect your testosterone.
11) Varicocele. These are nasty distended and/or tangled systems of veins near the testicle that can affect testosterone and fertility. I had one young reader write in who knew exactly when this occurred: during a squat at the gym. (See my link on Fertility for details.)
12) NEWS FLASH: This is hot off the press: rapidly increasing blood sugar levels lead to significantly decreased testosterone.  In fact, blood sugar elevation can whack testosterone by as much as 25%! Over 95% of the men in the above study, both with normal glucose clearance and diabetic, had their testosterone significantly lowered for hours by a glucose drink. Again, you must manage the Glycemic Load of your foods. Read this link on How Sugar Lowers Your Testosterone and Assaults Your Sex Life.
13) NEWS FLASH #2: The Atkins (or Low Carb Diet) received yet another black eye from scholars at the University of North Carolina. Researchers there found that a low carb (30%) diet lowered testosterone by about a third and increased cortisol by about a seventh (15%) when compared to a high carb (60%) diet after some intense training.  The testosterone to cortisol ratio is incredibly important to male long term health and this hammered it from both ends.
14) Fenugreek. Watch out for this herb (at least the seeds) that hail from India, China and other parts of Asia. Several animal studies show a decrease in both testosterone and fertility. 
Mercury will also lower your androgens, but unless you are eating a lot of fish, this shouldn’t be an issue for you. Also, I highly encourage you to read about my link regarding Excitotoxins as, unlike Mercury, this is so prevalent in our foods and supermarkets thanks to the FDA and Food Industry and will take out both your Brain AND your hormones.
NOTE: You should also read my section on Free Testosterone as there I discuss several things that can uniquely lead to lower Free Testosterone as well. Remember that Free Testosterone is the bio-available serum testosterone that can actually “do work” in your tissues.
Remember that this is not a cumulative list. However, it only takes ONE of these to severely reduce your T-levels 30% or more. Remember that low testosterone is associated with heart disease, diabetes, Metabolic Disorder, depression and osteoporosis in males.
NOTE: Please read this link on How Marijuana Can Destroy Your Hormones, Fertility and Lead to Impotence.
1) Intl J of Impotence Res, 2009, 21:89 98, “Obesity, low testosterone levels and erectile dysfunction”
2) Francis Hayes, The results were presented at The Endocrine Society s 91st Annual Meeting in Washington, D.C. This was not published even online at the time of writing but was covered extensively in the health and medical press.
3) Natl Inst on Alcohol Abuse and Alcoholism, “Alcohol and the Male Reproductive System”, Mary Ann Emanuele, M.D., and Nicholas Emanuele, M.D.
4) Reproduction, 2002, 124:173-180, “Regulation and Perturbation of Testicular Functions by Vitamin A” (Review)
5) Clin Endocrinol (Oxf), 2009 Dec 29, [Epub ahead of print], “Association of vitamin D status with serum androgen levels in men”
6) European journal of applied physiology, 2010 Apr, 108:1125-1131, “Influence of dietary carbohydrate intake on the free testosterone: cortisol ratio responses to short-term intensive exercise training.”
7) Contraception, 73(3):301-306, “Evaluation of the potential antifertility effect of fenugreek seeds in male and female rabbits”
8) Phytotherapy Research, Mar/Apr 1993, 7(2):134-138, “Efficacy of the steroidal fraction of fenugreek seed extract on fertility of male albino rats”