Self-paced annotations of crowd workers

• Published in: Knowledge and information systems Vol. 64; no. 12; pp. 3235 - 3263
• Main Authors: Kang, Xiangping, Yu, Guoxian, Domeniconi, Carlotta, Wang, Jun, Guo, Wei, Ren, Yazhou, Zhang, Xiayan, Cui, Lizhen
• Format: Journal Article
• Language: English
• Published: London Springer London 01.12.2022; Springer Nature B.V
• Subjects: Annotations; Budgets; Computer Science; Crowdsourcing; Data Mining and Knowledge Discovery; Database Management; Incentives; Information Storage and Retrieval; Information Systems and Communication Service; Information Systems Applications (incl.Internet); IT in Business; Learning; Quality control; Regular Paper; Workers; Crowdsourcing; Truth inference; Self-paced learning; Task assignment; Quality control
• ISSN: 0219-1377; 0219-3116
• DOI: 10.1007/s10115-022-01759-5

Crowdsourcing can harness human intelligence to handle computer-hard tasks in a relatively economic way. The collected answers from various crowd workers are of different qualities, due to the task difficulty, worker capability, incentives and other factors. To maintain high-quality answers while reducing the cost, various strategies have been developed by modeling tasks, workers, or both. Nevertheless, they typically deem that the capability of workers is static when assigning/completing all the tasks. However, in actual fact, crowd workers can improve their capability by gradually completing easy to hard tasks, alike human beings’ intrinsic self-paced learning ability. In this paper, we study crowdsourcing with self-paced workers, whose capability can be progressively improved as they scrutinize and complete tasks from to easy to hard. We introduce a Self-paced Crowd-worker model (SPCrowder). In SPCrowder, workers firstly do a set of golden tasks with known truths, which serve as feedbacks to assist workers capturing the raw modes of tasks and to stimulate the self-paced learning. This also helps to estimate workers’ quality and tasks’ difficulty. SPCrowder then uses a task difficulty model to dynamically measure the difficulty of tasks and rank them from easy to hard and assign tasks to self-paced workers by maximizing a benefit criterion. By doing so, a normal worker can be capable to handle hard tasks after completing some easier and related tasks. We conducted extensive experiments on semi-simulated and real crowdsourcing datasets, SPCrowder outperforms competitive methods in quality control and budget saving. Crowd workers indeed hold the self-paced learning ability, which boosts the quality and save the budget.