Evaluation of the confusion matrix method in the validation of an automated system for measuring feeding behaviour of cattle

• Published in: Behavioural processes Vol. 148; pp. 56 - 62
• Main Authors: Ruuska, Salla, Hämäläinen, Wilhelmiina, Kajava, Sari, Mughal, Mikaela, Matilainen, Pekka, Mononen, Jaakko
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.03.2018; Elsevier Science Ltd
• Subjects: Animal behavior; Animals; automation; Bovidae; Cattle; Cattle - physiology; Confusion matrix; Confusion probabilities; dairy cows; Error indices; Feeding behavior; Feeding Behavior - physiology; Feeding behaviour; Feeding Methods - instrumentation; Feeding Methods - standards; Female; Information processing; Linear regression; Models, Statistical; Monitoring, Physiologic - instrumentation; Probability; regression analysis; Reproducibility of Results; Validation; Validation studies; Validation Studies as Topic; Validation; Confusion matrix; Confusion probabilities; Linear regression; Feeding behaviour; Error indices
• ISSN: 0376-6357; 1872-8308; 1872-8308
• DOI: 10.1016/j.beproc.2018.01.004

•Device validation with confusion matrices was evaluated empirically.•Assessments by confusion matrices, error indices and linear regression were compared.•Confusion probabilities help to analyse reasons for errors and their importance. The aim of the present study was to evaluate empirically confusion matrices in device validation. We compared the confusion matrix method to linear regression and error indices in the validation of a device measuring feeding behaviour of dairy cattle. In addition, we studied how to extract additional information on classification errors with confusion probabilities. The data consisted of 12 h behaviour measurements from five dairy cows; feeding and other behaviour were detected simultaneously with a device and from video recordings. The resulting 216 000 pairs of classifications were used to construct confusion matrices and calculate performance measures. In addition, hourly durations of each behaviour were calculated and the accuracy of measurements was evaluated with linear regression and error indices. All three validation methods agreed when the behaviour was detected very accurately or inaccurately. Otherwise, in the intermediate cases, the confusion matrix method and error indices produced relatively concordant results, but the linear regression method often disagreed with them. Our study supports the use of confusion matrix analysis in validation since it is robust to any data distribution and type of relationship, it makes a stringent evaluation of validity, and it offers extra information on the type and sources of errors.