I don't fully understand how this all works, but I can tell you that for almost all of my adult life (I'm 46 now) I had many symptoms of low T. Really bad inability to concentrate, bad anxiety and depression, etc. Could not put on muscle mass no matter how hard I tried.
But, until relatively recent to my diagnosis, I had no problem with sexual function. I did not have super-high libido, but still was well within normal, and had no ED at all. Didn't have morning wood, ever, but never had a problem when needed. I eventually started to have some ED and lost libido, which led to my diagnosis. But ... no real problems despite the fact that I am certain that I was low.
I am certain those other symptoms were caused by low T, because they all went away once I started TRT, and have not returned. And, when I first started and was still building up to a full dose, I had some T reads in the high 200's to low 300's, and had no difficulty with any of those symptoms, so I must have been lower than that.
I cannot explain why I did not have the sexual issues until the very end, or why I did not have them for most of my low T time. All I can say is that I don't think testosterone is the only thing involved, and that how everything else plays into it is not well understood. So, while it sounds like you are at an unusual extreme on this, I don't think it is at all implausible that you could have very low T and not have sexual symptoms.