Some of the body's essential vitamins and minerals seem to have fairly wide band of safety. Magnesium probably falls in this category for example. Vitamin C may be another example, although a couple of recent studies show that megadosing may decrease athletic recovery and performance and possibly pump out free radicals instead of quelling them. Zinc, however, is probably a nutrient that needs to be much more tightly controlled and, on this page, I'll give evidence that even moderately high levels can probably cause a man some issues.
The recent research is particularly disturbing, because correcting low levels of zinc can double a guy's testosterone: see my on Zinc Deficiencies and Low Testosterone. Furthermore, some studies show zinc supplementation can increase the average, healthy male's testosterone a bit, although not all. Because of all of this coverage in the health press, a significant percentage of men are taking rather high doses of zinc - sometimes 3 + times the RDA for example - on a daily basis. Below I will show you why this may not be a good idea.
Before I cover some of the dangers of zinc, I do want to say that it would be very easy for a man to become zinc deficient for many reasons:
a) poor absorption due to GI disorders, which are incredibly common
b) poorly done vegetarian and vegan diets. (I eat an almost entirely plant-based diet, so this is not a criticism of anyone out there.)
c) caffeine, tannins, tea/coffee, iron, etc. can slow zinc absorption
d) a man loses about 5 mg of zinc when he ejaculates.
So, while one should be cautious about zinc, there has to be balance as well, i.e. the other side of the equation can be ugly as well. If you have a zinc deficiency in particular, supplementing with zinc can produce some impressive results and so there are some counterarguments in that case. See my pages on Zinc and ZMA, Controlling Estrogen (Estradiol) and Zinc and DHT for more information.
Here are some of the (potential) risks that have recently been uncovered regarding zinc. (Thanks to the guys on Peak Testosterone Forum who pointed out some of these studies - seppuku!)
1. Possible Increased Prostate Cancer Risk. Some researchers believe that too much zinc could fuel prostate cancer:
"The high concentration of zinc in the prostate suggests that zinc may play a role in prostate health. We examined the association between supplemental zinc intake and prostate cancer risk among 46 974 U.S. men participating in the Health Professionals Follow-Up Study...Although we cannot rule out residual confounding by supplemental calcium intake or some unmeasured correlate of zinc supplement use, our findings, that chronic zinc oversupply may play a role in prostate carcinogenesis, warrant further investigation." 
Men on HRT should particularly take notice, because, if one's PSA rises, the doctor or clinic will likely immediately take you off of HRT. So that will be adding insult to injury...
MEN ON HRT: Some of you are taking lots of zinc in order to control your estradiol levels. For all the reasons on here, that may not be such a good idea.
2. Prediabetes and Metabolic Syndrome. Suppversity has a great article on how an animal study shows that zinc induces Metabolic Syndrome.  For those who don't know, Metabolic Syndrome is the suite of symptoms that is a plaque in Western socieities, characterized by prediabetes (insulin resistance), high triglycerides and high blood pressure. It's very hard on arteries and hormones and leads to a host of chronic diseases. The last thing you want to do is to push the pedal on this one. Also, the article calculated the human equivalent and 30 mg, according to this calculation, could potentially cause trouble.
3. Lowered Libido. This is anecdotal, but one of our forum members posted the following:
"Can you think of any reason why 50mg of zinc absolutely kills my libido i read it should do the opposite. Any theory? Copper deficiency.... low estrogen?" 
4 Dementia. Few guys know that zinc is neurotoxic and probably at pretty low levels. No one knows exactly what that level is, but why take a chance, eh? As an example as to just how this level probably is, consider this statement from one set of researchers:
"Increasing evidence has suggested that zinc (Zn) is central to ischemia-induced neuronal death and, finally, to the pathogenesis of VD [Vascular-Type Dementia]." 
In other words, they believe that even the relatively low levels of zinc in the standard Western Diet may be playing a role in many of the dementias that we see in our seniors. Now imagine a guy taking 3+ times the RDA of zinc.
5. Nasty Lipid Changes. The common supplemental amount that bodybuilders and health nuts take here in the U.S. is 50 mg. I believe that is what is in the standard ZMA supplement that many men take for example. A study out of China showed that giving just 40 mg per day to healthy males showed that "total cholesterol (TC), triglyceride (TG), Low density lipoprotein cholesterol (LDL-C) and apolipoprotein B100 (ApoB100) increased significantly, high density lipoprotein cholesterol (HLD-C) and apolipoprotein A1 (ApoA1) decreased after supplementation.."  This is ugly and could lead to increased arterial plaque and, yes, that can eventually affect you in the bedroom.
6. Lowered Iron Absorption. Too much zinc can slow down iron absorption.  Now this effect may actually help some guys who are eating red meat and getting too much heme iron in their diet. For still other men, though, decreasing iron levels could be an issue. Examples of this are men with undiagnosed hypothyroidism and gut issues, who end up with low iron stores. For example, on the Peak Testosterone Forum, we have had several posters with low ferritin levels.
NOTE: Zinc will also lower copper absorption. Overly high levels of copper from pipes is a probably a huge problem and so this may be an advantage. (Copper has been implicated in Parkinson's Disease for example.)
7. Lowered Magnesium Levels. Some experts say that taking supplemental zinc can lower magnesium levels - never a good idea.  Magnesium plays a huge role in arterial health, testosterone and serotonin production and blood glucose / insulin control (just for starters).
3) Wei Sheng Yan Jiu, 2004 Nov, 33(6):727-31, "Effects of high level Zn intake on metabolism in man"
4) Pediatric Research, 2006, 60:636 636, "Inhibition of Iron Absorption by Zinc: Effect of Physiological and Pharmacological Doses: TL015"
5) Int J Mol Sci, Nov 2013, "The Molecular Mechanisms of Zinc Neurotoxicity and the Pathogenesis of Vascular Type Senile Dementia"
6) J Natl Cancer Ins, 2003 Jul 2, 95(13):1004-7, "Zinc supplement use and risk of prostate cancer"