Hand-Object Contact Force Estimation from Markerless Visual Tracking

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 40; no. 12; pp. 2883 - 2896
• Main Authors: Pham, Tu-Hoa, Kyriazis, Nikolaos, Argyros, Antonis A., Kheddar, Abderrahmane
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.12.2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Biological system modeling; Computer Science; Computer Vision and Pattern Recognition; Contact force; Equations of motion; Force measurement; Force sensing from vision; hand-object tracking; Kinematics; Machine Learning; manipulation; Object recognition; Object tracking; Optical tracking; Optimization; pattern analysis; Physical properties; Recurrent neural networks; Robotics; Sensors; tracking; Transducers; pattern analysis; Force sensing from vision; sensors; manipulation; tracking; hand-object tracking
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2017.2759736

We consider the problem of estimating realistic contact forces during manipulation, backed with ground-truth measurements, using vision alone. Interaction forces are usually measured by mounting force transducers onto the manipulated objects or the hands. Those are costly, cumbersome, and alter the objects' physical properties and their perception by the human sense of touch. Our work establishes that interaction forces can be estimated in a cost-effective, reliable, non-intrusive way using vision. This is a complex and challenging problem. Indeed, in multi-contact, a given motion can generally be caused by an infinity of possible force distributions. To alleviate the limitations of traditional models based on inverse optimization, we collect and release the first large-scale dataset on manipulation kinodynamics as 3.2 hours of synchronized force and motion measurements under 193 object-grasp configurations. We learn a mapping between high-level kinematic features based on the equations of motion and the underlying manipulation forces using recurrent neural networks (RNN). The RNN predictions are consistently refined using physics-based optimization through second-order cone programming (SOCP). We show that our method can successfully capture interaction forces compatible with both the observations and the way humans intuitively manipulate objects, using a single RGB-D camera.