Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes

• Published in: Journal of mathematical biology Vol. 78; no. 1-2; pp. 189 - 224
• Main Authors: Grusea, Simona, Rodríguez, Willy, Pinchon, Didier, Chikhi, Lounès, Boitard, Simon, Mazet, Olivier
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.01.2019; Springer Nature B.V; Springer
• Subjects: Applications of Mathematics; Change detection; Coalescence; Coalescing; General Mathematics; Genes; Genetics; Heredity; Life Sciences; Mathematical and Computational Biology; Mathematics; Mathematics and Statistics; Population; Population genetics; Population number; Population structure; Populations and Evolution; Structured coalescent; Rate matrix; 92D15; Inverse instantaneous coalescence rate (IICR); Population structure; Demographic history; 60J27; 60J35; Population size change; 60E05; 15A18; rate matrix; IICR (inverse instantaneous coalescence rate); population structure; population size change; demographic history; structured coalescent
• ISSN: 0303-6812; 1432-1416
• DOI: 10.1007/s00285-018-1272-4

The increasing amount of genomic data currently available is expanding the horizons of population genetics inference. A wide range of methods have been published allowing to detect and date major changes in population size during the history of species. At the same time, there has been an increasing recognition that population structure can generate genetic data similar to those generated under models of population size change. Recently, Mazet et al. (Heredity 116(4):362–371, 2016 ) introduced the idea that, for any model of population structure, it is always possible to find a panmictic model with a particular function of population size-change having an identical distribution of T 2 (the time of the first coalescence for a sample of size two). This implies that there is an identifiability problem between a panmictic and a structured model when we base our analysis only on T 2 . In this paper, based on an analytical study of the rate matrix of the ancestral lineage process, we obtain new theoretical results about the joint distribution of the coalescence times ( T 3 , T 2 ) for a sample of three haploid genes in a n -island model with constant size. Even if, for any k ≥ 2 , it is always possible to find a size-change scenario for a panmictic population such that the marginal distribution of T k is exactly the same as in a n -island model with constant population size, we show that the joint distribution of the coalescence times ( T 3 , T 2 ) for a sample of three genes contains enough information to distinguish between a panmictic population and a n -island model of constant size.