One thing I emphasize on this site is that Low Testosterone will kill you over time. But high cortisol levels, which generally result from increased stress levels, do much more rapid damage to something else important to you: your brain cells and neurons. Scientists first noticed this in various animal experiments where animals put under severe stress had not only a decrease in synapses but also an atrophy of their dendrites. Neuronal cell death often occurred as well. In other words, every part of their brain was vulnerable to destruction: both the "cpu" and the "network" if you will.
Then neurologists noticed that a variety of orders that raise resting cortisol levels left thier victims with partially destroyed hippocampuses. (The hippocampus is the key part of the brain associated with memory.) For example, one study of Vietnam Vets with Post Traumatic Stress Disorder found that on average about 10% of the hippocampus had been destroyed.  Other studies have shown up to 26% damage!
You might not guess that depression would be stressful, but it is extremely so. Depression raises cortisol and kills the brain while it's at it. (Around half of all people with major depression have significantly elevated cortisol levels. And it goes both ways: there is some overlap, i.e. stress can cause depression. One study found that "stress is a more important predictor of depression in the elderly than are genetic factors.".)  In 1999 researchers examined 48 women to find exactly how depression effects the brain and found that women who had experienced depression has a 9-13% greater average shrinkage of the hippocampus.  Furthermore, they found that the more depressive episodes the greater the shrinkage of the hippocampus. In other words, the longer the cortisol levels are elevated by depression, the more the damage. And, more importantly, the damage is enduring and long-lasting.
The bottom line is that if you are depressed, get help and get treatment. Your brain depends on it. The longer you ignore the problem, the more neurons are destroyed. Have you lost your zest for life? Do you have the classic signs of depression? If so, then get tested and get help.
This list of stressors could go on and on and include such disorders as PTSD (Post Traumatic Stress Syndrome) and childhood abuse. Some of you may be thinking, "Well I've been fortunate enough never to have faced such a severe mental overload, so none of this applies to me." However, sleep is another counterintuitive stressor of the body.
Lack of sleep can raise morning cortisol levels and that's the last thing your brain and body needs. So remember that stress isn't just the typical "I'm stuck in traffic and late for a meeting" situation. Many counterintuitive things in life can lead to elevated stress hormone levels and may take your memory and cognition with it.
1) Psychosomatic Med,1999, 61:292-296
2) Psychiatry Res: Neuroimaging,2003,(154)2:191-198
3) Semin Clin Neuropsychiatry,2001 Jan,6(1):27-31
4) J Neurosci,1999,19:5034-5043