What is the primary sex organ, the penis or the testes? It's a trick question, because the answer is 'neither.' The answer is that the center of your sex life is the brain. Just ask any high prolactin man and he'll tell you. High prolactin lowers dopamine and that is a huge sex life killer.
So what does that have to do with Propecia (Proscar or finasteride)? As I've discussed on other pages, research has shown that Propecia has an abnormally high rate of long term / permanent sexual side effects. I even have reason to believe that this is what brought my own health spiraling downhill about fifteen years ago, something I discuss in my page Did Propecia Take Out My Health?
For a long time it was assumed that the issue to be hormonal, i.e. decreased testosterone or DHT (dihydrotesterone). However, the research has not really backed this up. So researchers began looking elsewhere and it appears that the answer likely lies in the brain. Look at the bullets points from the research abstract that looked at former Propecia users when compared to controls:
However, before I do that, let me just briefly discuss a 2012 study and shows this is a huge clinical concern. This study showed overwhelming negative and long-lasting psychological side effects in former Propecia (finasteride) users, i.e. men who had not taken Propecia for at least three months: 
"Rates of depressive symptoms were significantly higher in the former finasteride users (75%; 46/61) as compared to the controls (10%; 3/29)".
"Moderate or severe depressive symptoms were present in 64% (39/61) of the finasteride group and 0% of the controls."
" Suicidal thoughts were present in 44% (27/61) of the former finasteride users and in 3% (1/29) of the controls."
These numbers are staggering. Propecia increases rates of depression by 75%! And the lion's share of these results in severe depression. And suicidal tendencies are increase by 2700% and includes almost half of the former Propecia users!
Now I'll ask an obvious question: what do you think this will do to your sex life? Well, of course it will be a sex life killer for many men. Depression is brutally hard on the (penile) arteries and the brain and thus delivers a one-two punch to your bedroom endeavors, something I discuss in my page on Depression and Erectile Dysfunction.
But all of this leads to an obvious question; if finasteride is not changing testosterone or DHT much, just what is it changing? It turns out that blocking 5-alpha-reductase - the enzyme that Propecia acts upon - also lower allopregnenalone and this, in turn, lowers GABA levels. GABA is critical neurotransmiter that, along with sertonin, is your "peace and tranquility" neurotransmitter. If one lowers GABA much, you end up anxious and depressed.
A 2006 study explains the entire pathway that is disrupted by Propecia:
"5alpha-reductase is a critical enzyme in the conversion of several steroids such as testosterone, progesterone, aldosterone and corticosterone in the brain. This enzyme converts testosterone to the most natural potent androgen DHT, and also it acts an important role in conversion of progesterone to dihhydroprogesterone (DHP). DHP is further converted to allopregnanolone (5alpha, 3alpha-tetrahydroprogesterone) by 3alpha-HSD. Allopregnanolone is a modulator of gamma amino butyric acid type A receptor (GABA-A), and increases chloride conductance. This neurosteroid has been found to exert anti-convulsant, anesthetic and anxiolytic effects. Moreover, change in the levels of allopregnanolone is found to be associated with depressive disorders." 
Of course, there is still a lot that we don't know. Is GABA permanently lowered? Is allopregnanalone permanently decreased? Or is there perhaps some other reason not yet discovered? I don't know the answer to those questions, but I can tell you that that is one of the riskiest of the more common medications out there in my opinion.
1) J Clin Psychiatry, 2012 Sep, 73(9):1220-3, "Depressive symptoms and suicidal thoughts among former users of finasteride with persistent sexual side effects"
2) BMC Clin Pharmacol, 2006, "Finasteride induced depression: a prospective study"