Sequential Multi-Class Labeling in Crowdsourcing

• Published in: IEEE transactions on knowledge and data engineering Vol. 31; no. 11; pp. 2190 - 2199
• Main Authors: Kang, Qiyu, Tay, Wee Peng
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.11.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: collaborative computing; Crowdsourcing; Dogs; Encoding; Error correction; Games; Labeling; Labelling; Markov analysis; multi-class labeling; Questions; Reliability; sequential question design; Solvers; Strategy; Task analysis; Ulam-Rényi game
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2018.2874003

We consider a crowdsourcing platform where workers' responses to questions posed by a crowdsourcer are used to determine the hidden state of a multi-class labeling problem. As workers may be unreliable, we propose to perform sequential questioning in which the questions posed to the workers are designed based on previous questions and answers. We propose a Partially-Observable Markov Decision Process (POMDP) framework to determine the best questioning strategy, subject to the crowdsourcer's budget constraint. As this POMDP formulation is in general intractable, we develop a suboptimal approach based on a q-ary Ulam-Renyi game. We also propose a sampling heuristic, which can be used in tandem with standard POMDP solvers, using our Ulam-Renyi strategy. We demonstrate through simulations that our approaches outperform a non-sequential strategy based on error correction coding and which does not utilize workers' previous responses.