The answer to that is a little more complex and we have to go to animal studies to get some answers and insight. First of all, heat will definitely shrink the testes, something most men are not too excited about. In a study of rams, heat actually shrunk the testes by about a third by weight.  This is particularly remarkable, because the rams were not subject to ultrahigh heat, but rather constant heat in the 28-23 degrees Celcius range (82-90 degrees Fahrenheit) continuously for two weeks.
Of course, shrunken testicles is a very bad sign. But does this mean testosterone was actually lowered? A study on increasing heat using rats showed that that FH and LSH rose. Now testosterone, interestingly enough, did not change, but the testes became less responsive. So it is as if there was a “partial secondary hypogonadism” even though testosterone levels did immediately fall.  In fact, the authors concluded that “it appears that the testosterone produced by Leydig cells from heated testes may not be secreted as effectively as in normal testes.”
So, obviously, something is going on. And, in fact, one study actually looked at using a combination of exogenous (applied from outside the body) testosterone plus heat to create a male contraceptive.  It was so effective that it almost reduced sperm counts to zero.
But does heat affect testosterone levels. One study on boars shows that initially heat may not affect testosterone levels  However, eventually baseline T levels are actually decreased. In other words, it appears there may be a time lag from the application of the heat to the time that testosterone actually decreases. This was verified in the study on rams that showed that testosterone decreased significantly after a two week period.  One study on monkeys verified the same thing: a time lag loss of testosterone after 6 days of 30 minutes heat treatment of the testicle area. 
1) J OF ANIMAL SCIENCE , 1971, 33(4), “EFFECT OF ELEVATED AMB IENT TEMPERATURE ON TESTIS AND BLOOD LEVELS AND IN VITRO BIO – SYNTHESIS OF TESTOSTERONE IN THE RAM”
2) Int J Androl, 1988 Feb, 11(1):61-72, “Effects of local heating of the testes on the concentration of testosterone in jugular and testicular venous blood of rats and on testosterone production in vitro”
3) Endocrinology, Apr 1, 2000, 141(4):1414-1424, “Testicular Heat Exposure Enhances the Suppression of Spermatogenesis by Testosterone in Rats: The Two-Hit Approach to Male Contraceptive Development”
4) Animal Reproduction Science, Aug 1984, 7(5):391 403, “Effects of acute and chronic testicular hyperthermia on levels of testosterone and corticosteroids in plasma of boars”
5) Journal of Andrology, November/December 2002, 23(6), “Mild Testicular Hyperthermia Induces Profound Transitional Spermatogenic Suppression Through Increased Germ Cell Apoptosis in Adult Cynomolgus Monkeys (Macaca fascicularis)”