Calcium is an incredibly important element in the human body. Virtually everyone knows that our bone composition and health is dependent on calcium, but few realize that calcium acts as an "electronic switch" for our neurons and is critically important to the brain.
Furthermore, Vitamin D and calcium (and therefore dairy) have been shown (at least in women) to help the body shed fat metabolically and the protein helps build and maintain muscle as well.  The females in this study lost about 4 pounds of fat and simultaneously gained 4 pounds of muscle, a nice exchange to say the least.
Calcium is used in so many chemical processes in the body that it would impossible to go into them all, but it does, for example, help with periodontal (gum) disease  and I have mentioned elsewhere, in my link on High Blood Pressure and Erections, how it forms a trinity of elements with sodium and potassium to help avoid hypertension.
All of that said, it is definitely possible for men to have "too much of a good thing" and calcium is no exception. Both calcium and dairy can easily lead to very dangerous situations for males, including cancers, and so a few cautions are in order:
1. Prostate Cancer and Calcium. My Prostate Cancer link covers the study that shows that overly high calcium intakes can put males at greater risk for prostate cancer.
2. Generalized Cancer Promotion. Casein, the chief protein in milk and dairy is a known increaser of IGF-1. Many bodybuilders are IGF-1 enthusiasts, because it can add in hypertrophy (muscle building). However, IGF-1 is controversial at best because much research shows that in the right conditions it can promote cancer, especially as we age. I cover this in detail in my link on IGF-1.
3. Liver Cancer. Many people have heard about the bestselling book The China Study and the work of T. Colin Campbell. What is not as well known is that his initial claim to fame in the research labs was examining the remarkable ability of casein to promote liver cancer under certain conditions in laboratory animals. If casein was high enough in the diet, liver cancer rates were dramatically increased.  Again, if you get enough dairy in your diet, there is good evidence it could make you vulnerable to this nasty cancer.
4. Inflammation. Casein is also a known promoter of inflammation in many cases and it is even used in animal studies to do the same.  Another interesting example is how small amounts of dark chocolate have been found to lower inflammation levels but additional amounts yield no net benefit. The reason? It is likely the casein in the chocolate overcoming cacao's excellent benefits. (See my link on The Benefits of Dark Chocolate for more information.)
So, again, moderation is the key when it comes to calcium. I personally try not to consume more than 1-2 times the RDA for calcium. As far as milk and dairy, I avoid them. I will occasionally consume nonfat milk for a protein boost, but keep it to one serving only.
1) Medicine & Science in Sports & Exercise, Jun 2010, 42(6):1122-1130, "Body Composition and Strength Changes in Women with Milk and Resistance Exercise"
2) Mediators Inflamm, 2004 Feb, 13(1): 33 37, "Analysis of leukocyte rolling and migration--using inhibitors in the undisturbed microcirculation of the rat mesentery--on inflammatory stimulation"
3) Journal of Periodontology, Jul 2000, 71(7):1057-1066, "Calcium and the Risk For Periodontal Disease"
4) Journal of the National Cancer Institute, 1987, 78(2):365-369, "Relative contribution of dietary protein level and aflatoxin B1 dose in generation of presumptive preneoplastic foci in rat liver"
5) J Nutr, 1987 Jul, 117(7):1298-302, "Dietary protein level and aflatoxin B1-induced preneoplastic hepatic lesions in the rat"