I have extensive coverage on my site as to just how much food affects medium and long term baseline testosterone levels from many standpoints. In this link, I want to cover just how rapidly the food in one meal can affect your immediate testosterone levels. This is very important, because it can affect how we feel during the night, which is, for most of us anyway, the primary time for love making. If the food we eat can lower testosterone the same night, that is obviously something to take into consideration.
Now, before I got into post-meal testosterone, it is important to briefly review some of the research that correlates certain foods and dietary regimens with testosterone levels:
1. Fat and Fiber. Some research shows that total fat can raise medium term testosterone levels and fiber does the opposite. These studies are oft-cited around bodybuilding and steroid sites in particular.
2. Mono and Saturated Fats. Some of the same research in #1 showed that monounsaturated fats and saturated fats were testosterone-boosting in the medium term as well. And I actually give an example of some third world peoples that follow an interesting diet along these principles in my link on A Testosterone Diet?
3. Decreased Protein. Again, some studies from the pool of research in #1 showed that lower protein levels were associated with increased testosterone. This flies in the face of almost all the standard bodybuilding advice and popular diets such as the Zone Diet and Body for Life. I do have to mention, though, that one study showed that, while a high carbohydrate diet did raise testosterone levels significantly, it also raised SHBG levels, thus likely washing out any benefits. 
4. A Low Fat Diet. In direct contraction of #1, a Low Fat Diet was found in one study to keep testosterone levels the same yet and yet greatly decrease estrogen levels. (This may be due to #3.) For many men, this could have a profoundly positive effect, because it would boost their testosteorne-to-estrogen ratio. In fact, the study is listed in my link on the Testosterone-to-Estrogen Ratio.
The bottom line is that much of the research on medium term food and testosterone is inconclusive: there are clearly many questions remaining. For example, one of the most important questions is what diet will best preserve testosterone into one's senior years? Most men's forums would unswervingly prescribe a standard Paleo Diet along the lines of #1 above. However, this is debatable advise, because the Okinawans, who do not really follow any of the above rules, have very high testosterone levels well into their senior years compared to Westerners. For more information, see my link on How to Avoid Andropause.
All of these type of questions are difficult to answer definitively. Furthermore, some of the above research is contradictory and confusing. For all of these reasons, on this page I want to focus on a different aspect of testosterone and food: post-prandial (post-meal) testosterone.
This subject is extemely important, especially for men who are middle-aged and beyond. As you may know from my page on The Daily Cycle of Testosterone, testosterone levels dip between 15-35% for men on average from early morning to evening, depending on age and a few other factors
Did you know you can inexpensively do your own testing for most hormones? The industry leader is Discounted Labs..
Many of the younger guys have lots of buffer built in and can afford decreased nightime testosterone. Such is not necessarily the case once you hit about 40. First of all you get a decrease from a natural diurnal cycle and the last thing you need is for the food you eat at dinner to push down your testosterone even further.
NOTE: Men also need to watch how they eat at dinner, because it can so severely limit blood flow, which is never helpful in the bedroom. This is why I warn men so much regarding high total and saturated fat meals. I cannot tell you how many men's sex lives have been helped simply by limiting How and Why Saturated Fat Can Be Bad For Men.
So is there any similar advice for men with regards to testosterone? Can a meal actually affect our testosterone levels significantly? Yes, indeed, and it might not be what you think:
1. High Glycemic Load Meals. As I always say, "Food is clinical." Doctors don't like to admit, but how we eat has a profound effect on the body, sometimes in ways that cannot be anticipated. One of the these ways was recently discovered by researchers who found that men who consumed a significant quantity of glucose dipped their testosterone levels by an average of 25% post-meal. Any high glycemic load meal will likely do something similar and I cover this in my link on Testosterone, Glucose and Sugars. Remember that refined carbs - can you say white bread and white rice - spike your blood sugar and insulin levels about as much as table sugar.
2. Higher Fat Meals. Conventional wisdom says that eating a meal full of fat boosts testosterone and turns you into a gorilla-man. However, what high fat meals actually do is just the opposite: slow down blood flow, leading to "limpification" And, even worse according to one study, eating this way can also lead to significantly reduced post-prandial testosterone levels.  Ouch is right!
Basically the authors in this study compared a lower fat meal with mixed protein and carbs to a high fat meal and found that "the fat-containing meal, but not the nonnutritive or mixed carbohydrate and protein meal, resulted in a significant reduction in total and free testosterone."
As I point out often on The Peak Testosterone Forum, wild game is very lean and almost always low fat. And almost everything edible in nature is low fat with the exception of nuts, seeds and fish. So what?? Well, this means that man has generally not been able to eat the incredible amounts of fat that we find in modern diets from modern oils, butters and high fat livestock. Our bodies are just not tuned for high fat meals and it looks like we pay for it with decreased blood flow and testosterone.
NOTE: In fairness, one study did not show a loss of testosterone after a higher fat meal and instead showed one with a lower fat meal. 
3. Big Meals. An interesting study looked at the effects on cortisol of both a high protein and a high carbohydrate meal on healthy males (avg age = 26). They way they did this was to give a pretty big shake to the participants:
--The high-protein meal was "a salad (iceberg lettuce, cucumber, mushroom, and sunflower oil), Gouda cheese, salami, and a strawberry protein shake."
--The high-carbohydrate meal was "a salad (iceberg lettuce, cucumber, green pepper, and sunflower oil), savory cheese biscuits and TUC bacon biscuits, and a strawberry carbohydrate shake."
Both of these meals were pretty big and were designed to be 30% of daily requirements. That's a pretty big meal and here is the interesting thing: in men both of these meals caused huge rises in cortisol. (Women experienced a much smaller increase in any and all conditions.) As men we appear to be vulnerable to post-mail increases in cortisol for almost any conditions. And it may be these increases that lead to the decreases in testosterone?
4. Soy. One study showed a meal with soy significantly lowered testosterone after a meal.  I caution against soy for other reasons as well in this link on Soy and Men.
Regardless, this points to the fact that the only protection a man may have is to "graze", i.e. do not consume overly high caloric and eat many meals throughout the day. This also is critical for many men for their long term health due to blood glucose and insulin issues. One study showed that type II diabetic men who ate this way, i.e. smaller meals spaced throughout the day, had much smaller insulin and blood glucose responses. And this will be critical not just for men with diabetes, but also men with prediabetes/Metabolic Syndrome as well, i.e. probably half the middle-aged and beyond population in Western societies. Unless you are exercising heavily, odds are that you can protect your beta cells and your testosterone by smaller, more frequent (and low glycemic load) meals.
1) Life Sciences, May 1987, 40(18):1761 1768, "Diet-hormone interactions: Protein/carbohydrate ratio alters reciprocally the plasma levels of testosterone and cortisol and their respective binding globulins in man"
2) Metabolism, Sep 1990, 39(9):943 946, "Effects of a fat-containing meal on sex hormones in men"
3) PLOS One, Received: September 3, 2010; Accepted: January 9, 2011; Published: February 3, 2011, "Influence of Consumption of a High-Protein vs. High-Carbohydrate Meal on the Physiological Cortisol and Psychological Mood Response in Men and Women"
4) Diabetes Care, Jan 1993, 16(1):4-7, "Effect of Meal Frequency on Blood Glucose, Insulin, and Free Fatty Acids in NIDDM Subjects"
5) Metabolism, May 2001, 50(5):505 511, "Postprandial changes in sex hormones after meals of different composition"