Testosterone is one of those ubiquitous molecules that affects just about everything in us men. One of the connections that few men realize is that testosterone even affects your adrenals, the "stress glands" of the body. The adrenals are the endocrine glands that sit next to the kidneys and produce such critical stress hormones as cortisol, ephinephrine (adrenaline) and norepinephrine (noradrenaline). In addition, the adrenals also make an important stress modulator, DHEA, that acts in a complimentary fashion to cortisol in particular. The adrenals even make a little testosterone, although you get the lion s share of your T from the testes of course.
So how does testosterone affect the adrenals? Let's look at a few of the comments from men who have gone on HRT (Hormone Replacement Therapy) in the Peak Testosterone Forum and see how their enthusiasm might relate to their adrenals and stress and cortisol levels:
Notice this man's comments before HRT:
"I suffered from extreme fatigue, muscle pains and weakness. (which are not 'common' side effects of low T). The fatigue is remarkably better (so much better I no longer need prescription stimulants to make it through the day!...I suffered from extreme fatigue, muscle pains and weakness." 
If you read through his post, he suffered from all the classic symptoms of what some call "adrenal fatigue". Adrenal fatigue is basically the controversial theory that a number of prominent many alternative practitioners subscribe to and posits that the adrenals can become "exhausted" from chronic overstimulation, leading to decreased output of key hormones such as cortisol. The standard symptoms are very similar to what this man described: fatigue, aches, mental fog, weakness.
Now notice his description as to how he felt after testosterone therapy in the same post:
"After 4 weeks my sex drive is "through the roof"...The fatigue is remarkably better...I no longer need prescription stimulants...The weakness I felt is totally gone! I don't 'feel sick' like I did for the past 5 years!"
Many men who go from low testosterone to higher testosterone on HRT (assuming their estradiol levels are managed properly as well) experience this same remarkable transformation from Mr. Seeming Adrenal Fatigue to, well, the complete opposite.
NOTE: Please see my link on Testosterone and Cortisol for additional information.
Another of our posters had adrenal fatigue-y symptoms: "I had a heavy fog in my head, I was sick of life, I never looked forward to anything and felt like I was worthless."  After HRT, he became a completely different person (which greatly helped his marriage by the way): "I still am feeling AWESOME. I am doing great...I felt incredible. I love the fact that workouts for me does lifting of my spirit and takes a lot of stress and weight off of me. I do not mean a foggy depressed weight in my head but more so an urge to use my testosterone as a man."
I could go on with posts like this, but I'm sure you get the idea. HRT very often makes a huge difference in hypogonadal men and the reasons are many: it lowers insulin; it lowers inflammation; it boosts neurotransmitters and so on. But one question remains: could it also be "boosting" the adrenals?
The answer is that the research shows that there is actually a testosterone-to-adrenal link and one that could possibly help the adrenals to "heal" and "recover". One study looked at young men and basically suppressed their testosterone pharmaceutically and then compared it when testosterone replacement was introduced. So, basically, the researchers were comparing a hypogonadal state versus a high testosterone state.
Now many people would probably expect the higher testosterone state to have higher cortisol levels. The guys should be more "wired" and "restless", right? Wrong! The hypogonadal state had significantly higher cortisol levels, indicating that the adrenals had to work harder without sufficient testosterone. The authors summarized by stating "CRH-stimulated cortisol was significantly decreased during the testosterone-replaced condition compared with the induced hypogonadal (relatively testosterone deficient) condition created by leuprolide with placebo replacement ."  (It is not just this study that shows this link between T and the adrenals: many animal studies have as well.)
How does testosterone do this, i.e. inhibit the overreaction to stress? It does it, actually, by acting on the adrenals themselves, i.e. acting at "ground zero" of cortisol production. Cortisol is actually produced via trigger hormones: CRH is released from the hypothalamus, which in turn triggers ACTH from the pituitary that then tells the adrenals to pump out cortisol. Testosterone works by, according to the above study, inhibiting sensitivity of the adrenals to ACTH. This has many advantages, because under stress conditions, more ACTH may be produced but the adrenals are significantly protected.
There are some who are claiming that one should first fix any adrenal exhaustion. However, I feel this is likely misguided IF the man is low testosterone in the first place. The above study shows that a man likely needs ample testosterone in order to dampen his stress response. Again, the authors pointed this out by saying "in contrast, our data demonstrate that the effect of testosterone in young men is to inhibit rather than augment the cortisol response to CRH stimulation." 
NOTE: For related information, see my Stress and Cortisol Links.
Many men with low or lowish testosterone begin experiencing anxiety and the same can be said with men who believe they have adrenal fatigue. I cannot help but wonder how often the latter do not realize that their testosterone levels have fallen, leaving them vulnerable to overreacting to the stressors that life, especially modern life, seems to naturally bring.
So the bottom line is get tested and start monitoring if at all possible. Find and work with a good physician who will test your cortisol and testosterone levels.
CAUTION: One last note: many men are competitive and high octane if you will. It is very difficult for them to exercise in a relaxed and steady manner. (I'm one of them.) Are you attracted to HIIT, interval training, bodybuilding, triathlons, marathons, etc.? If so, then you need to ask yourself if you are possibly overtraining. It is very easy to do, especially if you are natural and have the right personality. Overtraining can easily lead to disturbed sleep, lowered immunity and - you guessed it - adrenal/cortisol issues as well.
Researchers have noted that a typical pattern is increased ACTH (to the stress of overtraining) which leads to decreased sensitivity to the same and so cortisol levels, initially increased, start to fall. Eventually, ACTH levels actually start to decrease as well, exemplifying a kind of "fatigue" or "exhaustion" response.  Remember: our bodies were built to walk. High octane men do not like to walk and often speak of it as a "waste of time" and in a derogatory manner. But, like it or not, we weren't meant to be Formula One or Indy every day.
So, based on what I've seen in Peak Testosterone Forum, probably 5-10% of the men reading this will have lowish testosterone and "adrenal fatigue-like" symptoms simply because there are one stage or another of overtraining. See my link on How to Overcome Overtraining for some more information.
3) Neuropsychopharmacology, 2005 October, 30(10):1906 1912, "Testosterone Suppression of CRH-stimulated Cortisol in Men"
4) Medicine and Science in Sports and Exercise, 1998, 30(7):1140-1145, "Autonomic imbalance hypothesis and overtraining syndrome"
5) Sports Medicine (Auckland, N.Z.), 1995, 20(4):251-276, "Blood hormones as markers of training stress and overtraining"