Let's review a few facts: both a high fat diet  and monounsaturated fats (such as olive oil) are generally pro-testosterone . Furthermore, monounsaturates are more healthy than saturated fats. Throw into this the fact that Olive Oil also improves several key risk factors for cardiovascular disease and Olive Oil seems like the King of Fats. It does, after all, lower inflammation, a key predictor of heart disease.  In fact, about 3.5 T (50g) of Olive Oil has a phytochemical, oleocanthal, in it that is has the same inflammation-lowering strength as a tenth of an NSAID such as ibuprofen. And, as if that wasn't enough, studies have shown that Olive Oil decreases fibrinogen, another key and inflammation-related predictor of heart disease.  One recent study even showed that Olive Oil was associated with a 3% drop in systolic blood pressure. . Plus, Olive Oil is a frequent component of the famed Mediterranean Diet that I often push on this site.
So the obvious solution is that all males should consume olive oil and lots of it, right? In fact, all the stars seem to be aligning toward olive oil being part of any healthy diet, eh? Well, unfortunately it's not that easy: recent research has shown that Olive Oil is not the shining star that everything originally thought it was. In fact, it looks more and more like the Mediterranean Diet's success may have been achieved in spite of Olive Oil not because of it.
One of the first clues actually came decades ago with Dr. Ornish's research. He found that you can actually reverse heart disease, i.e. decrease arterial plaque, with Low Fat (or Ornish Diet). One couldn't help but ask why the Mediterranean Diet, with all it's Olive Oil and heart healthy ingredients, could not also clear out plaque? As time has gone on, the evidence mounted that the key aspects of a Mediterranean Diet are probably its 1) deemphasis of meat and 2) emphasis of grain, vegetables and fruits. Olive Oil, unfortunately, appears to have a number of very negative qualities.
For example, one 2005 study put olive oil and rapeseed (canola) oil head-to-head.  Participants consumed 75g of either olive or canola oil. Canola oil, which is frequently excoriated by the Atkins/Paleolithic crowd, handily beat olive oil and revealed many of its weaknesses. The rapeseed oil tended to decrease cholesterol absorption, increase excretion of cholesterol and decrease overall serum cholesterol significantly (11%) compared to olive oil. In fact, they found out that rapeseed oil actually has significantly more plant sterols as well.
CAUTION: Canola oil is not perfect, though, and sometimes has small amounts of trans fats in it due to manufacturer heating. I personally avoid it.
Recent research, headed in large part by researcher Lawrence Rudel, have shown that the problem with Olive Oil is that it boosts a liver enzyme called ACAT2 leading to increases artherosclerosis.  ACAT2 is at ground zero for atherosclerosis: as levels of ACAT2 increase so do bad cholesterol particles in the blood. In fact, Rudel has shown genetically altered mice that have low ACAT2 production have much lower arteriosclerosis no matter what kind of fat they are fed and scientists hope that one day this knowledge can be used to battle arterial plaque and heart disease. But, in the meantime, we have to
One question you may be asking is if Olive Oil consumption has actually been shown to increase atherosclerosis. The answer is "yes" in numerous animal studies by Rudel on both monkeys and mice. Rudel has found that Olive Oil promotes arterial plaque build up as much as saturated fat. 
CAUTION: The best olive oils have high polyphenol counts and thus may do better in the studies. The problem is that very few oils have the polyphenol counts on the label and there are no studies thatI know of that have examined this in depth. In the meantime, we are left guessing and I think it is better safe than sorry.
1) J of Steroid Biochemistry Vol. 32, No. 6, p. 829-833, 1989; Amer J of Clin Nutr 42:127-134, 1985
2) Ann Intern Medic 2006 Sep 5,145(5):333-41;Eur Journ Clin Nutr,2007 Mar 21
3) Am Heart Jour 2000 Oct,140(4):631-6/Thromb Haemost 1994 Oct,72(4):557-562
4) Jour Nutrition 137:84-87, 2007
5) Eur J of Clin Nutr, Aug 10 2005 (online), 59:1374-1378
6) Rudel, Arteriosclerosis, Thrombosis, and Vascular Biology, not yet published, but press release were in May 2009.
7) Curr Atheroscler Rep, 2010 Nov, 12(6):391 396, "Dietary Monounsaturated Fatty Acids Appear Not to Provide Cardioprotection"