This is what you need to know: stress raises cortisol and cortisol lowers testosterone. In addition, cortisol will also dissolve your memory and do a host of other nasty things that I document here, but let's keep it simple for now: stress is hard on your hormones and you need your hormones for a decent sex life.
Furthermore, all you middle-aged and beyond guys should realize something critical: a number of studies have shown that cortisol levels, in general, rise with age.  This means all us older guys have less buffer to play with since our cortisol is higher and testosterone is lower. In other words, we have to be more careful.
When you hit middle age, you've got to Choose Wisely. You can't do everything any more. You've got to choose that One Most Important Thing. It's really important that you take some time to figure out what you want to do with your life and what is most important to you. If you are headed toward another career, for example, you must plan that out carefully.
Here's one reason I say that: you need to no longer compromise on your health. In fact, most of you need to repair your health. All of us middle aged and beyond must do what I call the Big Three:
Here are 13 Research-Backed Ways to Lower Stress and Stress Hormones:
1) Overtraining. Guard carefully against overtraining: numerous studies have documented a nasty rise in cortisol with overtraining, especially once performance is affected as well.  Remember that most of these studies have been done on elite athletes. How do you think you're handling that if you have a demanding career and/or a couple of kids?
2) Sleep. A number of studies have correlated lack of sleep with higher cortisol levels.  Lack of sleep is perceived by your body as a stressor and it pumps out extra cortisol accordingly. With enough caffeine you may be able to fool your brain, but you're not fooling your body: the cortisol is still there and wrecking physiological havoc on your system.
3) Choose Wisely. Be careful about having too many hobbies, too many investments, too much stuff, too many friends - you get the idea. Remember this: almost everything is good and too much good will kill you through the stress it creates.
4) Job. Vent about any issues and problems in your job. One study found that guys that "bottled up" negative work experiences were twice as likely to die of a heart attack or heart disease.  Other studies have echoed how hard a bad work environment, or even perceived bad work environment, is on the body.
5) Vitamin C. If you feel trapped temporarily in a bad situation, then there is a proven cortisol-lowering solution available: Vitamin C has its issues, but it has done quite well as a cortisol reducer. Please see this link on Vitamin C for more details (as well as cautions).
6) Fish Oil. Several studies have shown fish oil to reduce not only cortisol but the body's overall stress response, including epinephrine and energy expenditure. 
8) Tai Chi. There's no "theological" endorsement of Tai Chi here, but I want to point out that Tai Chi does have one study behind it showing that it significantly reduced anxiety in patients. 
9) Dark Chocolate. Well, this is still more good news for chocolate lovers. All you type A's out there, listen to this: Nestle's lab found that in high anxiety subjects, a small bar of dark chocolate significantly reduced stress hormones. 
10) Progressive Muscle Relaxation. This tried-and-true technique that involves tensing and relaxing muscle groups is a proven cortisol-buster. In fact, one study showed that subjects lowered cortisol, perceived stress, anxiety and pulse when compared to subjects that just sat quietly for the same amount of time.  (A subsequent study also showed decreased cortisol levels.)  Pretty impressive for a simple of investment of 15-20 minutes per day, eh?
11) Yoga. A lot of guys are scared of Yoga. After all, most of us are sports-minded and that mean lots of kinetic movement. The idea of painfully stretching into a Dali-esque pose is completely foreign to us. However, the fact is that yoga works and works well when it comes to stress management. One 2004 even showed that it lowered salivary cortisol levels. 
12) Meditation. Many studies have shown that meditation lowers cortisol levels and it should be noted that this goes for all the basic types of meditation. For example, both cortisol and blood pressure levels were significantly reduced in young males practicing meditation.  Many men, unfortunately, are intimidated by meditation, because it so often has is sandwiched in the middle of religious jargon and beliefts that antithetical to ones own personal beliefs. The solution is what is called secular meditation, i.e. non-religious meditation. I cover this in my link How to Do Secular Meditation for those interested. Other related links are The Benefits of the Relaxation Response and Mindfulness Meditation for those interested.
13) The Smell of Coffee. One animal study showed that just the smell of coffee turned on key genes that protects neurons from the damaging effects of stress.  This study found that 17 genes were actually regulated by just the smell of coffee and they produced 13 proteins known to reduce the negative effects of elevated stress levels. Time to wake up and smell the coffee, eh? NOTE: This is yet another example of the powerful effects of smell and aromas upon our physiology. Another is lavender, which improves sleep.
Keep the stress levels moderate and you will be surprised what a significant difference it makes to your memory, erectile strength, libido and so on. All of the healthy third world cultures that I talk about on this site, the Kuna, the Tokeluau, the Pukapuka, the Tarahumara, etc., live a much, much more relaxed lifestyle. These cultures are very family and community oriented, tend to sleep when the sun goes down for about 9 hours and so on. They walk everywhere, prepare everything by hand - it's a radically more peaceful lifestyle.
Contrast that with us: we get out of work a little late hungry, drive through the traffic gauntlet to pick up our kids, get them some food as we drive home, take them to practice, help them with homework, fix them a snack, etc., etc. And so then to unwind we stay up late watching TV or reading on the computer.
My advice is always ask yourself, "What is most important?". Or better yet, "Is this really worth my health?"
1) Nov. 24 online edition, Journal of Epidemiology and Community Health, Leineweber, et. al.
2) Psychoneuroendocrinology, 2001, 26:225 240, "Gender differences in age-related changes in HPA axis reactivity"
3) Chronobiology Intl, May 2000, 17(3):391-404, "SLEEP IMPAIRMENTS IN HEALTHY SENIORS: ROLES OF STRESS, CORTISOL, AND INTERLEUKIN-1 BETA"
4) Psychoneuroendocrinology, 1989, 14(4):303-310, "Mood state and salivary cortisol levels following overtraining in female swimmers"
5) Clincial Endocrinology, 66(2):185-191, "Cortisol secretary pattern and glucocorticoid feedback sensitivity in women from a Mediterranean area: relationship with anthropometric characteristics, dietary intake and plasma fatty acid profile"
6) J of Parenteral and Enteral Nutrition, 1995, 19(6):482-491, "Improved Clinical Status and Length of Care With Low-Fat Nutrition Support in Burn Patients"
7) Journal of Analytical Bio-Science, 2006, 29(2):146-150, "Effects of moderate exercise on chronic stress; an analysis from salivary cortisol concentration and subjective mood data"
12) Diabetes Metab, 2003 Jun, 29(3):289-95, "Fish oil prevents the adrenal activation elicited by mental stress in healthy men".
13) Am J Med Sci, 1989 Dec, 298(6):390-6, "Neuroendocrine and stress hormone changes during mirthful laughter"
14) J of Complementary and Integrative Medicine, Vol. 6 : Iss. 1, Article 14, "The Effect of Tai Chi in Reducing Anxiety in an Ambulatory Population,"
15) J of Proteome Res, 2009, Published online ahead of print Oct 2009, "Metabolic Effects of Dark Chocolate Consumption on Energy, Gut Microbiota, and Stress-Related Metabolism in Free-Living Subjects"
16) Biological Psychology, Jul 2002, 60(1):1-16, "The impact of abbreviated progressive muscle relaxation on salivary cortisol"
17) Ann Behav Med, 2004 Oct, 28(2):114-8, "Effects of Hatha yoga and African dance on perceived stress, affect, and salivary cortisol"
18) Applied Psychophysiology and Biofeedback, Dec 2005, 30(4):375-387, "The Impact of Abbreviated Progessive Muscle Relaxation on Salivary Cortisol and Salivary Immunoglobulin A (sIgA)"
19) Physiol Behav, Sep 1991, 50(3):543-8, "Effect of Buddhist meditation on serum cortisol and total protein levels, blood pressure, pulse rate, lung volume and reaction time"
20) J Agric Food Chem, 2008, 56(12):4665 4673,"Effects of Coffee Bean Aroma on the Rat Brain Stressed by Sleep Deprivation: A Selected Transcript- and 2D Gel-Based Proteome Analysis"