Many guys suddenly find themselves hit with a big decrease in testosterone and have absolutely no idea why. Usually they're feeling inexplicably lousy and, after struggling with mulitiple doctor's visits, get tested and find their testosterone has decreased to hypogonadal levels.
That's when most guys will ask, "Why me?" In fact, I will sometimes get a young guy writing in who has lived a relatively healthy life and yet has greatly decreased testosterone. He'll essentially be asking, "How could this testosterone decrease happen to me when I haven't really done anything wrong?" (Read here to find out what are Normal Male Testosterone Levels.)
Post-mortems are always tough, but let me give you a few things that could explain why a big decrease in your testosterone seemingly by "some mysterious force" All of the items below can lead to a decrease decades later or at least appears to occur decades later:
1) Excitotoxins. Excitotoxins, such as MSG (Glutamates) and Aspartates, have been shown in animal studies to cross the placental barrier.  Of course, the fetal brain is much more vulnerable to excitotoxins and there is even evidence that excitotoxins that penetrate the placental barrier can accumulate on the placental side. Animal studies show that this can be a Testosterone Time Bomb, just waiting to explode with a big decrease in testosterone and other hormones when the animal hits adulthood. Animals exposed to excitotoxins during infancy and key developmental stages have altered hormonal outputs, including in some cases to decreased testosterone and shrunken testes after the animal goes through "puberty". In other words, if your mother consumed enough excitotoxins while she was carrying you, it is possible that she inadvertently set up a fuse that was just waiting to go off later in your life with a big decrease in testosterone. In my case, they put MSG in baby food throughout my formative years! (For more information, see this link on Excitotoxins and Testosterone.)
2) Varicocele. Varicoceles are varicose or damaged veins that disrupt proper blood flow to one of the testes, which can effect fertility and lead to a decrease in testosterone. Varicoceles can occur from trauma or heavy lifting and are also a Testosterone Time Bomb, except in their case the decrease in testosterone usually occurs with a few months after the injury, but, of course, sometimes isn't noticed until years later. This can be corrected fairly easily by surgery. (For more information, see this link on Improving Fertility.)
3) Endocrine Disruptors. Many pesticides are endocrine disruptors and animal studies have shown that these also can create a significant testosterone decrease in us males. For example, one study  showed pesticide exposure during development led to decreased testosterone output later in life. In fact, this occurred for four different pesticides and the authors concluded that "present data demonstrates that exposure to EDC [endocrine disruptors] during gonadal organogenesis alter follicular dynamics and steroid levels alter in life". (For more information about these can decrease your testosterone, see this link on Testosterone and Pesticides.)
4) Mycoplasma. "I got the walking pneumonia and the boogy woogy blues". This song is spot on when it comes to pneumonia: researchers have found that mycoplasma infections, a.k.a. walking pneumonia, can leave you with decreased testosterone that you discover years later.
5) Apnea. Apnea is another testosterone decreaser. Many guys do not realize they have apnea - maybe 5-10% of all males in industrialized socieites - and therefore do not realize this sleep disruptor also is disrupting their hormones. Read this linke about How Apnea Produces a Testosterone Decrease.
6) Vitamin D Deficiency. There are surprisingly many lifestyle issues that can lead to a serious Vitamin D deficiency. For example, many guys are trying to avoid skin cancer and stay out of the sun. Still other guys live in climates where there is little sunlight. And many guys just never get outside much. Regardless of the reason, researchers have found that lower Vitamin D levels are associated with decreased testosterone. Read this link about How Low Vitamin D Can Reduce Testosterone Significantly.
7) Lead. Some of us middle-aged and beyond men lived in the days of leaded gas and lead paint. It is still possible to get toxic levels of lead exposure from other sources such as dishes, children's toys and other consumer products. On the job exposure is definitely possible as well. Several studies show that elevated levels of lead can lead to lower testosterone and in the long term possibly lead to hypogonadism as well.  One study documented a significant impact to estrogen, Luteinizing Hormone, Follicle Stimulating Hormone and prolactin as well. 
1) Brain Res, Feb 7, 1997, 747(2):195-206, "Effects of maternal oral administration of monosodium glutamate at a late stage of pregnancy on developing mouse fetal brain"
2) Sheng Li Xue Bao, Feb 1994, 46(1): 44-51, "Transplacental neurotoxic effects of monosodium glutamate on structures and functions of specific brain areas of filial mice.
3) Intern J Neurosci, 1984, 23: 117-126, "Prenatal monosodium glutamate (MSG) treatment given through the mother's diet causes behavioral deficits in rat offspring"
4) Chemosphere, Dec 2006, 65(11):1990-1998, "Broad range analysis of endocrine disruptors and pharmaceuticals using gas chromatography and liqueid chromatography tandem mass spectrometry".
6) Infertility, 1978, 1(1):33-51, "Hypogonadism in chronically lead-poisoned men"
7) Hum Exp Toxicol, Mar 1988, 7(2):2 125-128, "Lead Toxicity on Endocrine Testicular Function in an Occupationally Exposed Population"