Testosterone therapy, or hormone replacement therapy (HRT) as it is sometimes called, does have occasional side effects that should be discussed with your doctor. It is interesting because most of the fears that guys have when considering testosterone therapy are very rare and some of the more common things would never even cross their mind. For example, most guys are concerned that their testosterone production will completely shut off and/or that they will have significant testicular shrinkage. In fact, these are usually not an issue since most men continue to produce some testosterone even when on testosterone therapy. Good doctors test your testosterone levels after you've gone on HRT and will make sure that your testosterone levels are not overly high. Generally, exogenous testosterone can be considered "supplemental" and not in the "replacement" category.
Furthermore, most of us have heard steroid horror stories of man boobs and tiny testicles and we assume that testosterone therapy will do the same. Of course, nothing could be further than the truth. Many steroid users have a "if a little is good, then a lot must be better" mentality and push themselves way beyond the normal physiological hormonal ranges. This often simultaneously pushes their testosterone and estrogen sky high, shrinking their testicles and increasing breast/chest tissue, respectively. (Man boobs are a rare but reported side effect of testosterone therapy, but it is more likely a cause of low testosterone rather than high.  )
NOTE: If you decide to try testosterone therapy, make sure that you get a doctor that will test your liver function, PSA, total testosterone and free every three months. Once you've stabilized, i.e. you've got the right dose, and all tests look okay, then you can possibly go to six months depending on what your doctor says
So are there any legitimate concerns regarding testosterone therapy? Well, certainly, although none of them are common. Let's list some of the ones that should be considered, however, and discussed with your doctor:
1. Fertility. Going on testosterone will very likely affect fertility negatively. I get letters quite often from younger guys asking if they should go on HRT. Of course, I tell them to talk to their doc about fertility, because there somewhat of a rebound effect if you try to go off of testosterone therapy. So adding testosterone can affect LH (leutenizing hormone) and FSH (follicle stimulating hormone) levels. LH governs testosterone production in the testes and FSH is responsible for spermatogenesis, i.e. creation of the sperm. (See my link on Fertility for more information.) But even you older guys that are in a serious relationship with a Younger Woman may need to consider this as well.
2. Apnea. If you have sleep apnea, some studies show low testosterone can worsen that condition.  Again, though, sleep apnea is very correlated to weight. Some dropping some weight, which can boost testosterone directly, may solve your apnea, but work with your doctor in this situation. (By the way, not all experts feel the apnea-HRT connection is valid.)
3. Acne. Some guys get the joy of going through a second puberty when on testosterone. Again, this is uncommon but a possibility in sensitive individuals. See my link on Testosterone and Acne for details and solutions.
4. Hair Loss. Some have reported hair loss as a side effect, although this does not seem to be common and is difficult to prove. Some of the testosterone delivery systems, such as scrotal patches, deliver higher levels of DHT, the primary culprit in typical male pattern baldness.
5. Soy Allergy. Some of the topical preparations are actually manufactured from soy and thus should be avoided by those with a soy allergy.
6. Severe Health Considerations. There was a recent small study on 219 senior men with poor health (diabetes, obesity, high blood pressure, etc.) and poor mobility that showed a significant increase in cardiovascular events when compared with the placebo group. The study was halted and researchers basically said that no conclusion could be drawn due to the small study size. However, you should discuss this study with your doctor, especially if you are an older, sedentary male with a significant health condition.
7. Liver Problems. One last thing: almost all testosterones given in testosterone therapy are very well-tolerated by the liver, but you want a doctor that will test your liver function just to play it ultra safe. Liver issues are associated almost exclusively with the old-line oral testosterones that were difficult to metabolize.
8. Enlarged Prostate. One uncommon side effect occasionally reported with HRT (Hormone Replacement Therapy) is BPH (enlarged prostate).
9. Thickened Blood. A recent meta-analysis found that testosterone therapy significantly increased hemoglobin and hematocrit, which makes blood more thick and likely to clot. 
10. Lowered HDL. The same study above found that that testosterone therapy was associated with a small decrease in HDL as well. 
NEWS FLASH: A recent Newsweek [Nov 9, 2009, p. 56f] article covered how some guys on HRT are either ignorant or being negligent. They are applying the gel to their chest, shoulders and/or arms and then are in direct skin-to-skin contact with their children (or wives). This is very serious indeed: a young child can be developmentally affected by repeated exposure to significant amounts of testosterone, including potentially the brain. In a young boy, for example, the lines on the scrotal sac can be altered and pubic hair thickened and grown.
I was just talking with a pediatric endocrinologist about this and it is imperative that guys wash their hands and keep any areas of application covered. He had only noticed problems in guys who were careless and/or ignorant of the risks involved. There have been enough reports of this problems that the FDA is currently forcing a generic version of Testim to undergo "hand washing studies" to make sure that it is as safe in this regard as the original. Furthermore, even the original products may have to carry a "black box" warning label. Keep your family safe!
If you are interested, you should also read my Overview of HRT and common Methods of Testosterone Delivery (such as gels and injections). These links should give you a lot of good talking points with your doctor.
3) Clin Endocrin (Ox),1988;28:461-470
4) J Clin Endocrinol Metab, Jun 2010; 95:2560-2575, "Adverse Effects of Testosterone Therapy in Adult Men: A Systematic Review and Meta-Analysis"