Yes, testosterone that low can definitely be a problem for some guys and so I'm not surprised to hear what you are saying. Also, you are doing the right thing by using as little Cialis as possible imo. Try to solve your problems as much as possible with lifestyle changes (exercise, diet, sleep, weight loss/mgt, etc.), but you do have to have decent baseline T of course as well.
I can't really advise you whether or not to go on Clomid or not of course. I can tell you, of course, that lots of men in the U.S. are now on Clomid. Usually, it's younger guys wanting to preserve fertility or older guys that just don't like the idea of testosterone therapy for whatever reason.
How safe is low dose Clomid long term? Well, no one really knows the answer to that. It's just received more widespread adoption and, as far as I know, no one has studied long term effects.
Some of the big name (or so I have heard) online docs are now using it low dose as a permanent therapy. But is it really safe long term? I have some concerns long term and outline them here:http://www.peaktestosterone.com/Clomid_Risk
I think most guys that go on Clomid see it as something for a few years and then they want to figure out another plan.