Here is another strategy that I have seen work. Go to your doctor and say, "Hey, you've been a great help, but I'd really like to find a doctor that has worked a lot of with men's health issues like this. Do you know anyone whose got a specialization or reputation in dealing with these kind of things?"
Most doctors will not be put off by this. If it's a PCP, in particular, they can easily refer you to a urologist or endocrinologist that they've run across or heard about without any potential harm to their own business.
In the medical profession, it is common, of course, for a doctor to defer to another's specilized expertise and there is no loss of face in doing so. So this will be a non-threatening way for him or her to deal with a subject that he/she quietly admits is a black hole.
Here is kcwebguy's post about a little different approach to the same:
https://www.peaktestosterone.com/forum/index.php?topic=246.0