Pairwise comparisons as a scale development tool for composite measures

• Published in: Journal of the Royal Statistical Society. Series A, Statistics in society Vol. 185; no. 2; pp. 519 - 542
• Main Authors: Floridi, Ginevra, Lauderdale, Benjamin E.
• Format: Journal Article
• Language: English
• Published: Oxford Oxford University Press 01.04.2022
• Subjects: Aging; Coefficients; composite index; Concepts; content validity; Experts; Indicators; linear index; Reliability; Reliability analysis; robustness; Weighting; weighting and aggregation; weighting scheme
• ISSN: 0964-1998; 1467-985X
• DOI: 10.1111/rssa.12790

Composite scales are widely used for measuring aggregate social science concepts. These often consist of linear indices obtained as the weighted sum of a set of relevant indicators. However, selecting coefficients (or weights) that reflect the substantive importance of each indicator towards the concept of interest is a difficult task. We propose a method for the generation of linear indices for aggregate concepts based on pairwise comparisons. Specifically, we ask a group of subject‐matter experts to perform a series of pairwise comparisons, with respect to the concept of interest, between profiles displaying different combinations of indicators. This allows us to estimate coefficients for each indicator that provide a linear approximation to how experts make the pairwise evaluations. As we show, the method makes it straightforward to assess intercoder reliability, while being a more accessible task than directly asking experts for coefficients. We demonstrate our method with an application to the concept of ‘productive ageing’, including a cross‐cultural comparison of weighting schemes derived from a group of Italian and a group of South Korean experts on this concept.