The maintenance of polygenic sex determination depends on the dominance of fitness effects which are predictive of the role of sexual antagonism

Abstract In species with polygenic sex determination, multiple male- and/or female-determining loci on different proto-sex chromosomes segregate as polymorphisms within populations. The extent to which these polymorphisms are at stable equilibria is not yet resolved. Previous work demonstrated that polygenic sex determination is most likely to be maintained as a stable polymorphism when the proto-sex chromosomes have opposite (sexually antagonistic) fitness effects in males and females. However, these models usually consider polygenic sex determination systems with only two proto-sex chromosomes, or they do not broadly consider the dominance of the alleles under selection. To address these shortcomings, I used forward population genetic simulations to identify selection pressures that can maintain polygenic sex determination under different dominance scenarios in a system with more than two proto-sex chromosomes (modeled after the house fly). I found that overdominant fitness effects of male-determining proto-Y chromosomes are more likely to maintain polygenic sex determination than dominant, recessive, or additive fitness effects. Dominance also affects the predicted role of sexual antagonism in maintaining polymorphism. Dominant fitness effects that maintain polygenic sex determiantion, for example, are predicted to have sexually antagonistic multi-chromosomal genotypes. However, the individual proto-sex chromosomes are not expected to have sexually antagonstic effects, demonstrating that sexual antagonism can be an emergent property of the multi-chromosome genotype. Overdominant fitness effects, in contrast, are expected to maintain polygenic sex determination with proto-Y chromosomes that have sexually antagonstic effects even though the multi-chromosome genotypes do not have signatures of sexual antagonism, My results therefore demonstrate that the dominance of fitness effects has consequences for both the likelihood that polygenic sex determination will be maintained as well as the role sexually antagonistic selection is expected to play in its maintenance.