Just got a few minutes here - family is keeping me busy - but a few important questions:
1. I don't think you said what time you had the blood draw? Any chance it was in the p.m.?
2. Any chance you are overtraining? This can hit testosterone pretty hard, although I'm not sure it would lower it by 60%!
3. Also, have you been consuming a lot of whey? I have always noticed how many ex-weight lifters/power lifters/athletes write in who have been on whey for years. Although I can't prove this, I can't help but be suspicious that all the free glutamine and aspartates have hammered their neurons/hypothalamus and they are suffering the repercussions. You might try laying off the whey and/or going to undenatured whey if that is the case for you.
4. Any chance you have a varicocele? Supposedly, they are not caused by lifting, but I've had a number of guys swear it was. Anyway, this is the territory of a urologist.
Finally, I would definitely recommend going to an endocrinologist, especially with the reading you have above. Now many endos are like your doc and are not thorough and think that the low 300's are okay for a guy in his early 20's. Imo, your doctor should have said to you, "I'm reluctant to get you on HRT because of your age, but the low 300's will increase your risk for many medical conditions, including heart disease, insulin resistance, hardening of the arteries, erectile dysfunction and diabetes. So we need to dig in and find out what is going on and take this seriously and not just say everything is okay."
I know it's a little stressful and even "scary" when you find out you might have low T, but is definitely worth it even if you never go on HRT. The reason? Imo guys with low T (while they're trying to figure things out) need to take extra good care of their health - abundant exercise (without overtraining), whole foods proven diet (such as the Mediterranean), lots of sleep, stress management, etc. Don't be paranoid or become a hypochondriac, but be serious about your health.
One final thought: you should be very cautious about going by one T reading.