Fluctuating population dynamics promotes the evolution of phenotypic plasticity.
Theoretical and empirical studies are showing evidence in support of evolutionary branching and sympatric speciation due to frequency-dependent competition. However, phenotypic diversification due to underlying genetic diversification is only one possible evolutionary response to disruptive selection. Another potentially general response is phenotypic diversification in the form of phenotypic plasticity. It has been suggested that genetic variation is favored in stable environments, whereas phenotypic plasticity is favored in unstable and fluctuating environments. We investigate the "competition" between the processes of evolutionary branching and the evolution of phenotypic plasticity in a predator-prey model that allows both processes to occur. In this model, environmental fluctuations can be caused by complicated population dynamics. We found that the evolution of phenotypic plasticity was generally more likely than evolutionary branching when the ecological dynamics exhibited pronounced predator-prey cycles, whereas the opposite was true when the ecological dynamics was more stable. At intermediate levels of density cycling, trimorphisms with two specialist branches and a phenotypically plastic generalist branch sometimes occurred. Our theoretical results suggest that ecological dynamics and evolutionary dynamics can often be tightly linked and that an explicit consideration of population dynamics may be essential to explain the evolutionary dynamics of diversification in natural populations.