It's almost certainly lifestyle. For virtually all of human history, we spent a significant portion of our days engaging in rigorous physical activity. For the last few decades, the average level of physical activity per person has declined significantly. Thus we no longer thrive.
If you penned up a wolf in a small pen and fed him a highly processed diet that looked very little like what he was evolved to eat, when he was not thriving and his hormone levels were out of whack, you would not jump to the conclusion that it has to be pesticides on his food or something like that, you would instead recognize that he is being fed a diet he is not evolved for and is not engaging in the physical activity he was evolved to do. So pesticides could be a cause, but its probably not the primary cause.
We are no different. That does not explain every case of low testosterone, for example I have been very active and have ate a whole foods diet my entire adult life (sometimes we have genetics, viruses, or past trauma that can be a cause), but it does explain the vast majority of the general decline in testosterone levels.