Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

• Published in: International journal of selection and assessment Vol. 29; no. 3-4; pp. 467 - 482
• Main Authors: Brown, Matt I., Grossenbacher, Michael A., Martin‐Raugh, Michelle P., Kochert, Jonathan, Prewett, Matthew S.
• Format: Journal Article
• Language: English
• Published: Oxford Blackwell Publishing Ltd 01.12.2021
• Subjects: consensus‐based measurement; Crowdsourcing; implicit trait policies; situational judgment tests; subject matter expertise

It is common practice to rely on a convenience sample of subject matter experts (SMEs) when developing scoring keys for situational judgment tests (SJTs). However, the defining characteristics of what constitutes a SME are often ambiguous and inconsistent. Sampling SMEs can also impose considerable...