Pair-based covariance under mixed mating and its consequences for REML variance estimation

Abstract

Abstract This organism-independent theoretical study examines genomic evaluation of open-pollinated progeny from partially selfing species when the fitted genetic covariance contains only an additive genomic relationship matrix. Open-pollinated families contain selfed-selfed, selfed-outcrossed, and outcrossed-outcrossed sibling pairs, so their covariance must be averaged using joint pair probabilities rather than probabilities assigned to individual offspring. An offspring-level shortcut overestimates expected additive covariance whenever selfing and outcrossing coexist. Extending the pair calculation to the covariance between additive effects and dominance deviations in homozygotes gives a balanced-reference projection coefficient equal to twice the selfing rate under mating-system equilibrium and random unrelated outcrossing. When the solution is interior, this coefficient represents the exact restricted maximum-likelihood projection for balanced families with exchangeable covariance blocks. An independent Mendelian simulation with biallelic quantitative trait loci recovered the classical pair-specific coefficients without supplying them to the data-generating model and closely reproduced the theoretical coefficient, with a coefficient of determination of 0.9998 and mean absolute error of 0.0073. The additive genomic relationship matrix was constructed in a separate controlled marker simulation by VanRaden scaling, and additive-only restricted maximum-likelihood estimates closely followed the balanced-reference prediction, with a coefficient of determination of 0.9900 and mean absolute departure of 0.0107. The resulting dimensionless index and scale conversion are therefore sensitivity diagnostics rather than universal corrections for genomic evaluation designs.

The authors' abstract, as published at the source. Genetics, 2026 · DOI ↗