150. That's the magic number. Keep your cholesterol below 150 all your life and you'll never have heart disease. There is a likely corollary to that: keep your cholesterol low to avoid erectile dysfunction.
The reason is simple: your erections are very dependent on a healthy endothelium, the delicate lining of your veins and arteries that pumps out nitric oxide, and a healthy endothelium is dependent on low cholesterol. As it turns out, high cholesterol tends to decrease the ability of the arteries to relax and you need them to relax for blood to flow into the penis.
One study of patients with cardiovascular disease and arteriosclerosis - and most people in modern, industrialized societies have significant arterial plaque buildup - found that Lovastatin (Mevacor) significantly improved endothelial function and arterial dilation.  Of course, the significance of this is that Mevacor is a cholesterol-lowering statin medication.
This was verified with another statin, Lipitor, in a study on erectile dysfunction patients with high cholesterol and low morning erections and hardness factor.  These middle-aged men had significantly improved erectile rigidity, a.k.a. hardness factor, after being on this cholesterol-lowering medication.
Even stronger evidence, however, was a study that found that men with total cholesterol over 240 mg/dl had 1.83 times the risk of erectile dysfunction and impotence. In fact, they observed a dose-dependent rise in erectile dysfunction risk with increasing cholesterol.  This means that the higher the cholesterol levels, in general the lower the erectile strength, adding more solid evidence that the research conclusions were on track.
Other research has shown similar results. One 2003 study found that three factors, total cholesterol > 200, the total cholesterol/HDL ratio and HDL were all related to erectile dysfunction. Again, in keeping with what I wrote in #3, the authors wrote "Erectile dysfunction (ED) is frequently of vascular origin." 
Why is cholesterol so damaging to erections? If you read many of the popular health blogs, cholesterol supposedly has nothing to do with heart disease. Now I would have to agree that total cholesterol has nothing to do directly with heart disease. The reason is that it is really oxidized LDL that damages the endothelial wall. However, there is a decided relationship between total cholesterol and LDL that makes cholesterol a downstream indicator of potential problems. In fact, one study found that oxidized LDL slowed down certain kinds of vasodilation and triglycerides slowed down all kinds.  Remember: you want those arteries, including the ones in the penis, to be able to dilate and allow for increased blood flow.
So what is the best way to lower cholesterol? Of course, the researchers in two of the studies above used pharmaceuticals, statins, to lower cholesterol. However, is that really the best and most safe way? Statins definitely have their issues and side effects, which I document in my link on The Danger of Statins.
What most men and doctors for that matter do not realize is that satured fat is the prime governor of cholesterol and LDL levels in men. This is why reducing saturated (and total) fat can send your cholesterol levels plummeting. I have known many men who could not get their cholesterol levels to singificantly budge and believed that their issue was genetic but were able to dramatically improve when they cut saturated fat levels. The most powerful example of this is a Low Fat Diet with which most men can achieve a total cholesterol level below 150. A Low Fat Diet is so powerful that, if done without cheating, can actually clear out your arteries. For more information, read my link on the Incredible Benefits of a Low Fat Diet and discuss with your doctor. (Don't do anything without discussing with your physician first if you have any medical condition or medications.)
The bottom line is that this is another as to how closely the heart and the penis are interrelated. Heal one and you'll likely heal the other.
1) N Engl J Med, 1995 Feb 23, 332(8):481-7, "Beneficial effects of cholesterol-lowering therapy on the coronary endothelium in patients with coronary artery disease"
2) The Journal of Urology, Jul 2004, 172(1):255-258, "IMPROVEMENT IN ERECTILE FUNCTION IN MEN WITH ORGANIC ERECTILE DYSFUNCTION BY CORRECTION OF ELEVATED CHOLESTEROL LEVELS: A CLINICAL OBSERVATION"
3) Am. J. Epidemiol, 1994, 140(10):930-937, "Total Cholesterol and High Density Lipoprotein Cholesterol as Important Predictors of Erectile Dysfunction"
4) European Urology, 2003, 44(3):355-359, "Erectile dysfunction is associated with a high prevalence of hyperlipidemia and coronary heart disease risk."
5) Asian J Androl, 2000 Sep, 2: 161-166, "Hyperlipidemia and erectile dysfunction"