Some hormones just don't tend to play well with your testosterone. Cortisol and estrogen are examples. Another hormone that can whack your testosterone and your sex life is prolactin.
Prolactin is a fine hormone and great for sleep, but, unfortunately, it makes a lot of other important things go to sleep, including your testosterone and fertility if levels are high enough. It is, not too surprisingly, often devasting to erections due to decreased testosterone levels and because it antagonizes dopamine and leads to an increased refractory (recovery post-erection) period.  Excessive prolactin is also associated with gynecomastia, i.e. "male boobs".  And probaby most dangerous of all in the long term, prolactin is probably an inflammatory cytokine (TNF alpha, IL-6, etc.) and has been linked to various inflammatory diseases, especially lupus. 
In females, hypothyroidism is a common cause of elevated prolactin levels. In males, hypothyroidism likely raises prolactin
Consider the fact that one study noted that previous research found an "overall prolactin greater than 20 ng./ml. in 1.86% of 1,821 patients, prolactinomas in 7, 0.38%." but found only five out of about a thousand men had prolactin exceeding 20, i.e. about 0.5%.  In other words, high prolactin is generally not a root cause of low T, but it certainly can be and at a rate of somewhere around 1 out of a 100 hypogonadal men. However, the researches still concluded that "prolactin should be determined only in cases of low sexual desire, gynecomastia and/or testosterone less than 4 ng./ml." This is an amazing statement because it recommends that prolactin be pulled for testosterone less than 400 ng/dl, something that is simply NOT done from all I have seen. My impressions from the The Peak Testosterone Forum is that probably one out of every ten men has their prolactin pulled.
NOTE: You can have very high prolactin and yet still not be hypogonadal. The same study found that "testosterone was low in less than 50% of cases with prolactin greater than 35 ng./ml." 
So how do males get very elevated levels of prolactin then? One all-to-common cause is a prolactinoma, a benign tumor of the pituitary that begins secreting, sometimes in great quantities, prolactin. This can have devastating effects. Look at the description of this reader in his mid 40's:
"I started experiencing longer and longer periods between being able to have sex starting about 4 years ago. It is to the point now that I don't know when I can get it up. It has been very hard on my marriage and my wife has since moved out because she felt unloved and unattractive because of my lack of sexual desire. As you can imagine now that she has moved out I am taking this very serious. Here are my issues. I was going to a family doctor and wasn't liking the results I was getting. He told me I had low testosterone and put me on Andogel packets."
His low testosterone was devastating as you can tell. How tragic that a physical malady could lead to the undoing of a marriage. However, he did write me back with some good news:
"Are you sitting down for my Prolactin reading? It was 200. Normal from what the doctor told me is 1 - 12. This is supposed to be curable from what I have heard so I am very exited as you can imagine. Thanks again."
Elevated prolactin is very anti-testosterone and often leads to hypogonadism.  His levels were astronomical and thus it is not surprising that his testosterone was so adversely affected. (He was correct, of course, that normal prolactin levels are less than 12 ng/ml or maybe 15 ng/ml.) In his case, they did not find a prolactinoma or other tumor from an MRI, but put him on the standard medication and he got immediate results.
So, if you are wondering where your testosterone levels went, you may want to ask your doctor about getting a prolactin reading.
OTHER READING: I have another page that covers Causes of Low Testosterone and it covers the frequency of pitutitary tumors in men with low testosterone. Suffice it to say that the frequency is surprisingly high. I also want to point out a must read journal article called "Long-Term Management of Prolactinomas" that is must-read for anyone with a prolactinoma. It looks about long term safety and issues, both positive and negative, with the dopamine agonists that are standard therapy.
CAUTION: Read my link on The Importance of Avoiding Excitotoxins. Animals studies show that prolactin levels can be negatively affected due to excitotoxin damage to the hypothalamus. In addition, minoxidil is suspected of causing prolactinomas and elevated prolactin levels.  Minoxidil (Rogaine) is a common over the counter hair loss treatment.
1) The American Journal of Medicine, May 1978, 64(5):782-787, "Serum prolactin levels in untreated primary hypothyroldism"
2) Psychiatr Serv, 51:983-985, Aug 2000, "Psychopharmacology: Galactorrhea and Gynecomastia in a Hypothyroid Male Being Treated With Risperidone"
3) N Engl J Med 1978; 299:847-852, "Prolactin-Secreting Tumors and Hypogonadism in 22 Men"
4) Rev Urol, 2000 Winter, 2(1):39 42, "Hyperprolactinemia and Erectile Dysfunction"
6) The Journal of Urology, 1997, 158(5):1764-1767, "Endocrine screening in 1,022 men with erectile dysfunction: clinical significance and cost-effective strategy."
7) Rheum Dis Clin North Am, 2000 Nov, 26(4):713-36, "Roles of prolactin and gonadotropin-releasing hormone in rheumatic diseases"