We all joke around about being under stress, but, as you'll see below, it's really not that funny. There are many definitions of stress, but here I am primarily talking about the kind of stress that produces elevated cortisol levels. When you are under stress, your body releases increased levels of adrenaline and other "fight or flight" hormones and chemicals. The adrenaline does a certain amount of damage, raising pulse rates and blood pressure (especially in Type A personalities). But it is the cortisol that comes shortly after that does the most devastating long term damage.
Many people have trouble understanding what really causes stress and elevated cortisol levels. Yes, a death in the family, a divorce - those things can cause stress. But it is really your reaction to these events that produces what I will call "true stress", i.e. elevated cortisol levels.
Let's start with one counterintuitive example: depression. You would expect that depression would actually mute or numb any hormonal response, right? But it's actually quite the opposite. One study found that the cortisol levels of the depressed individuals were 68% higher than those without depression.  That same study found that testosterone, both nighttime and 24-hour, were significantly lower in these individuals and were negatively correlated with cortisol levels. To put it directly: the higher the cortisol, the lower the testosterone. Other studies have verified similar findings. 
So how does stress lower testosterone? Most of the research in this area was done through Population Council endocrinologist Matthew Hardy and his work on rats. He and his teams found out how stress lowers your testosterone: through a tricky little enzyme called 11 HSD-1.  Your body produces most of its testosterone in the Leydig Cells of the testes and this enzyme keeps cortisol, the primary stress hormone, from pushing down your androgens. It literally puts the brakes on cortisol from destroying your testosterone. However, in times of stress, there is simply too much cortisol versus 11 HSD-1 and this leads to a decrease in your testosterone production. Of course, anything that lowers testosterone is bad for fertility.
This means that stress does not just lower testosterone through adverse lifestyle changes, but directly through chemical pathways in the gonads. Of course, stress can also indirectly affect testosterone through "back door" means as well. For example, it causes us to overeat and a high glycemic load can will cause testosterone levels to drop within a few hours of a meal. Likewise, weight gain from the extra "comfort" eating can lead to lowered baseline testosterone levels in the long term.
The bottom line is that, for the sake of your hormone levels, you've got to take seriously the goal of minimizing stress in your life. There are other reasons as well: read my links on Stress and Erectile Dysfunction and Stress and Your Brain for more information.
So how can you combat stress? Are there any practical solutions? Please read my link on Practical Stress Management Solutions for solutions based on the latest research.
CAUTION: It looks like a little stress may be a good thing. Researchers have found that mild stress resulting from an "enriched social, physical and mental environment" ends up decreasing leptin levels.  Leptin was once considered a "good boy" hormone, because it decreases appetite and has other positive effects. However, leptin also causes cancer to thrive and grow and was linked to increased risk of colon and skin cancer.
1) Psychosomatic Med,1999, 61:292-296
2) Psychiatry Res: Neuroimaging,2003,(154)2:191-198
3) Semin Clin Neuropsychiatry,2001 Jan,6(1):27-31
4) J Neurosci,1999,19:5034-5043
5) Int J Sports Med,Oct 2001,22(7):537-43
8) J Androl, 1997, 18:475-4791997, 18:475-479
9) Obesity (Silver Spring), 2009 Aug, 17(8):1513-20. Epub 2009 Mar 26, "Social stress, visceral obesity, and coronary artery atherosclerosis in female primates"
10) BMJ 2002, 325:857, "Work stress and risk of cardiovascular mortality: prospective cohort study of industrial employees"
11) European Heart Journal, Advance Access published online on January 23, 2008, "Work stress and coronary heart disease: what are the mechanisms?", Received 1 August 2007; revised 14 November 2007; accepted 22 November 2007.
12) Obesity, 2008, 17(1):72 77, "Acute Stress-related Changes in Eating in the Absence of Hunger"
13) "Stress of An Enriched Environment Might Curb Cancer Growth", eScienceNews, 7/8/2010
14) CELLULAR AND MOLECULAR LIFE SCIENCES, 1981, 37(12):1296-1297, "The relationship between high and low trait psychological stress, serum testosterone, and serum cortisol"