Clomid probably raises SHBG in two ways, both related to estrogenic effects on the liver. First there's the direct increase in estradiol stimulated by higher testosterone and LH-stimulated intratesticular aromatization. Second, the zucomiphene isomer is estrogenic and probably acts directly on the liver. As an aside, I've been taking enclomiphene, the other isomer of Clomid, and have not seen a change in SHBG.
An aromatase inhibitor has no effect on zuclomiphene, limited effect on intratesticular aromatization, and a significant effect on other systemic aromatization.
Tamoxifen does not have an estrogenic component, but otherwise acts similarly to Clomid.
It wouldn't hurt to look at DHT, IGF-1, also progesterone and thyroid hormones (fT3, fT4, rT3) if you haven't already. I'm not expecting to see anything too out of line. There are way too many guys in this position, with good basic numbers but still having low libido and other problems. Unfortunately I don't think there are widely available tests for kisspeptin levels. Maybe in time more doctors will prescribe kisspeptin-10 as a stimulation test in cases where nothing else is obviously wrong.