Secondary hypogonadism is usually considered to be the result of a defect in the pituitary or hypothalamus. The reasons is that leutinizing hormone is actually secreted from the pituitary gland after receiving signals of gonadotropin-releasing hormone (GnRH) from the hypothalamus. So reductions in either GnRH or LH can result in secondary hypogonadism.
CAUTION: One paper, though, commented that hypothalamic hypogonadism was the most common of this type of secondary loss of testosterone.  I am not in the least bit surprised due to the Abundance of Potentially Testosterone-Lowering Excitotoxins in our food and drinks, which can easily hammer the hypothalamus, which for the most part lies unprotected in front of the blood brain barrier. Diet drinks, fast foods and processed foods all often contain aspartame, MSG, autolyzed yeast extract and/or hydrolyzed proteins, all of which have high levels of excitotoxins. Almost all modern wheys have high levels as well. Read my link Whey of Death for more information.
Tumors on the pituitary or hypothalamus can, of course, cause this as well.
But it should be pointed that there is probably a hidden, often underdiagnosed cause of secondary hypoganadism that often goes undetected:
Free Testosterone-to-Estrogen Ratio Reduction. As men gain weight, they begin to simultaneously experience reduced free tes tosterone levels and increased estrogen levels.  One very interesting (but small) study actually took men with secondary hypothalamic hypogonadism and treated them GnRH.  Because they had true secondary hypogonadism, their testes were reactivated and their T was restored to the upper end of the range. As a second stage to the experiment, the researchers began adding estradiol into the blood streams of these men and, as expected, their testosterone began dropping.
NOTE: Any increase in estradiol can wreck havoc on erectile function. Recent animal studies show that rising estradiol levels lead to delayed orgasm and nerve difficulties.
Researchers are very concerned that xenoestrogens (environmental estrogens) may play a role in this process as well.
How do researchers diagnose secondary hypogonadism? One clue is when a man has lowish LH and FSH readings and also has low testosterone. (High prolactin levels and low testosterone are often an indication of a Prolactinoma on the Pituitary.) Some doctors will often check for secondary hypogonadism by administering something that stimulates the testes, such as HCG. If testosterone rises significantly, then that is a sign the testes are fine and the body’s messaging systems are actually dysfunctional.
This kind of knowledge and experience is where a good endocrinologist (or an anti-aging, sports medicine or fertility physician) can come into play. Some urologists are also becoming very knowledgeable in hormones. Thus it is really up to the patient to find out how much his doctor knows about endocrinological issues. As always, caveat emptor…
1) Medical Hypotheses, April 1998, 50(4):331-333, “The role of estradiol in the maintenance of secondary hypogonadism in males in erectile dysfunction”
2) International Journal of Impotence Research, 2003, 15:38 43, “Oestrogen-mediated hormonal imbalance precipitates erectile dysfunction”
3) Med Hypotheses, 1998 Apr, 50(4):331-3, “The role of estradiol in the maintenance of secondary hypogonadism in males in erectile dysfunction”