Statistical Stratification and Benchmarking of Robotic Grasping Performance

• Published in: IEEE transactions on robotics Vol. 39; no. 6; pp. 4539 - 4551
• Main Authors: Denoun, Brice, Hansard, Miles, Leon, Beatriz, Jamone, Lorenzo
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Benchmark testing; Grasping; Grasping (robotics); Performance evaluation; performance evaluation and benchmarking; Planning; Probability; probability and statistical methods; Protocols; Robot sensing systems; Robots; Statistical analysis; Task analysis
• ISSN: 1552-3098; 1941-0468
• DOI: 10.1109/TRO.2023.3306613

Robotic grasping is fundamental to many real-world applications, and new approaches must be systematically evaluated. However, in most cases, the performance of a specific approach is assessed by simply counting the number of successful attempts in a given task, and this success rate is then compared to those of other solutions, without taking into account the random variability across different experiments (e.g. due to sensor noise or variations in object placement). In order to address this issue, we classify the observed performance into qualitatively ordered outcomes, thereby stratifying the results. We then show how to analyze these results in a statistical framework, which accounts for the variability between experiments. The advantages of our approach are demonstrated in the practical comparison of four grasp planning algorithms. In particular, we show that the proposed approach allows us to carry out several distinct evaluations from a single set of experiments, without having to repeat the data collection process. We demonstrate that differences between the algorithms, which would not be apparent from overall success rates, can be identified and evaluated.